TVoice 8.0 Passed – Have Some Notes

So I should have posted these a while ago when I finished by CCNP Voice but better late than never here are some notes on the TVoice 8.0 subject material.

Test blueprint https://learningnetwork.cisco.com/docs/DOC-8990 (CCO Required)

4 Elements of CUCM Environment

  • Call Agent  / CUCM
  • Network Infrastructure
  • Cisco Applications
    • CUPS
    • Unity Connections
    • UCCX
  • Voice Clients

 

Network Infrastructure

  • Routers
  • Switches
  • Firewalls

 

CUCM

  • Primary Call Agent

 

Voice Clients

  • Physical Phone
  • SoftPhone
  • 3rd Party SIP Phone

 

Cisco Applications

  • CUPS
  • CUC
  • UCCX

 

 

Possible Sources of Problems

  • Complex deployments
  • Various protocols
  • Various endpoints and services
  • Network infrastructure

 

Benefits of Systematic troubleshooting method

  • Easier to identify potential problems
  • Learning effect
    • Understanding what is going on where there is no problem:
      • Tracking down internal processes such as call flows
      • Knowing corner case
    • Better ability to spot anomalies
  • Systematic documentation helps to solve future issues.

 

Problem Solving Model

Define Problem

Gather Facts

Consider Possibilities

Create Action Plan

Implement Action Plan

Observe Results

Did it resolve the problem – no go back to start.

Yes

Problem resolved

Document the facts

 

Troubleshooting Tools

  • Cisco Unified Serviceability
    • Alarms
    • Setting Trace
    • CAR
    • Control Center
    • DNA
  • RTMT
    • Alerts
    • Viewing Traces
    • Syslog Viewer
    • Performance Monitoring
  • CLI
  • Show commands
  • Debug commands
  • Packet sniffer
  • Cisco Unified Operations Manager

 

Alarms

Local / External Syslog

SDI Trace

SDL Trace

 

Trace Information

SDI – System Diagnostic Interface

  • Run-time events
  • IP Address
  • TCP Handles
  • Date / Time

SDL – Signaling Distribution Layer

  • Call processing information from CUCM / CTI Manager
  • Used by Cisco TAC

Log4J – Used for Java Applications

 

Two ways to configure traces

Activate Troubleshooting Traces

Configure Traces on a per server / per service level

 

Max number of devices traced (default is 12)

FQCN – fully qualified calling number

 

CCM:StationInit

Sent from phone to CUCM

  • Button Presses

CCM|StationD

From CUCM to Phone

  • Change Phone display
  • Turn on MWI

CCM|Digit analysis

CCM|RouteList

CCM|RouteList

CCM|RouteListCdrc

CCM|RouteListCdrc

 

Dialed Number Analyzer

Used to find out if we could call from phone X to phone Y

 

RTMT Syslog Logs

System

Application

Security

 

Cisco Unified Reporting

GCFM – Generic Call Filter Module

Call filter match-list (up to 16)

Used for filtering the debug output from a router for a specific ANI / DNIS

 

Phone registration process

 

Cisco ILP – in line power

Fast Link Pulse – FLP

802.3af POE

 

Most common problem is mismatch of inline power between endpoint and switch

 

Common DHCP issues

  • DHCP server down
  • DHCP does not have scope for the phone
  • All addresses leased out
  • IP Helper not setup

 

Common TFTP issues

  • Scope does not offer option 150
  • Option 150 points to wrong address
  • TFTP  server on the address is stopped or hung

 

Try to Ping the phones IP address to see if the CM server can reach the phone via OS Admin or via CLI utils network ping

Check TFTP config on phone / DHCP servers

Check Phone status messages

 

MGCP registers to CUCM, GW must be configured to reach out to CUCM and the domain name must be the same as is configured in CUCM.

 

MGCP GW sends Restart in Progress

CUCM ACK

CUCM Audit Endpoint

GW ACK

CUCM Request Notification

GW ACK

 

GW Registered

 

 

MGCP Problems

IP connectivity problems

Wrong TFTP server / CUCM not running TFTP

IOS SW version incompatible

Missing or incorrect CUCM server

Incorrect IOS configuration

 

Show ccm-manager – shows MGCP status

Show mgcp endpoint – shows the endpoints that are controlled by ccm

Show mgcp connection – shows the active connections (calls)

Show mgcp statistics – shows the mgcp message stats

Show mgcp srtp – displays information about SRTP connections

Debug mgcp – enables debugs for the mgcp protocol

Debug ccm-manager events – see the events as the MGCP gateway is attempting to register

 

H.323 / SIP – do not register to CUCM. CUCM will always show unknown.

 

Dial peer monitoring

Show dial-peer voice

Debug voip dialpeer

H.323 GW monitoring

Show gateway

Debug h225 asn1 | events

Debug h225 q931

SIP GW Monitoring

Show sip-ua calls, connections, statistics, status

Show sip service

 

Delayed dial tone / CCM Admin page slow – CUCM server over burdened

  • CPU Usage high
  • Service crash
  • Service Hung
  • Memory Leak

 

Syslog messages can be used to view alerts

RTMT can be used to view CPU / Memory Utilization

 

Phones not registering

Maybe CCM service is not running or TFTP service isn’t running

 

Can’t reach CCM admin page

Tomcat might not be running – can be restarted via CLI

IS DNS resolving the ccm host name

Can we ping the CCM server

ACL / FW change?

 

Utils service list

Utils start Cisco Tomcat

 

Check Duplex / Speed for issues related to “slow network”

 

DB replication issues

Pub has a R/W copy of the DB Subs have R/O copy of the DB

 

Replication issue example: Changed phone setting, but it never shows up

IDS Replication – From PUB to SUBS

Static information phone configuration

ICCS (Intra-cluster communication Service) – Full mesh between all call processors (subscribers)

Dynamic runtime information

 

Why DB goes out of sync

PUB can’t reach subscribers – network issues

QoS not setup to prefer DB replication traffic

DNS issue

Server CPU is taxed

Error occurred during the DB replication process

 

Verify NW connectivity

Verify DNS resolution works

Make sure there is enough BW for replication and the RTT not greater than 80ms between pub and sub

 

RTMT / CLI can be used to verify replication state.

Replication state should be 2 which is good replication

Cisco Unified Reporting – Unified CM Database Status

 

Utils dbreplication status

File view activelog cm/trace/….. To view log file

Utils dbreplication runtimestate – shows realtime replication status across the servers

 

Utils dbreplication repair – if a status of 0 or 4 after 4 or more hours use dbreplication reset to stop and restart database replication.

 

Run the repair command at the publisher only

Wait until completed displayed from runtime state.

 

Go to subscriber, utils dbreplication stop – stops the replication on the subscriber

Go to Publisher, utils dbreplication reset (subscriber name)

 

If that doesn’t work a cluster reset is needed.

 

Stop dbreplication on all nodes

From the pub, utils dbreplication cluster reset

 

Troubleshooting LDAP Integration Issues

Can sync and auth against LDAP

 

Cisco DirSync must be activated / running

Can’t mix LDAP server types

 

When user deleted in LDAP next CUCM sync will tombstone the user object. The user will be deleted after 24 hours.

 

Common issues:

Network connectivity

Utils network ping

Service username and password wrong

check

Service user doesn’t have sufficient permissions

Needs read access to all AD user objects

Search base wrong

Check the DN

DirSync set to manual or too long a period

Check LDAP sync schedule in CUCM LDAP directory setup

DirSync not running

Check that DirSync service is activated and running on publisher

Some end-users missing

Check the search base would encompass where the user is located in LDAP

 

DirSync has Service Parameters that can be set.

 

H.323 GW issue

Call into H323 gateway just gives secondary dial-tone

Direct-inward-dial missing from inbound dial-peer

 

Call Setup issue

Fast busy

Missing / wrong Caller ID

No ring back

Dead air

One-way calling

Inefficient call routing

Unexpected second dial tone

 

Reasons:

Bad CSS / wrong dialed number

Bad digit manipulation

Phone unregistered

Codec mis-match / region misconfiguration

No MTP resources

Bad QoS / no QoS

Configured wrong in CUCM

RSVP configured wrong

CAC blocking call due to no enough BW

WAN having problems

ICT setup wrong between clusters

CCD not configured right

CUBE configured wrong

 

Single site call setup issues

CoS misconfigured

Bad digit manipulation

FW to VM and VM is down

Unregistered phone

 

CUCM call routing

Digit by digit analysis – most specific match logic

 

IP Phone

SCCP – Digit by Digit

SIP – En bloc / KPML / SIP Dial Rules

 

Gateway

MGCP / SIP / H323 – En block / overlap send and receiving

 

Trunk

SIP / H323 – En bloc / overlap sending and receiving

 

Partitions and CSS allow / deny calls

A device can only call those numbers that are in PT that are a part of its CSS

 

<None> PT is always accessible regardless if the calling device has a CSS

Devices that have the <None> CSS can access only other devices that are in the Partition of <None>

CSS is an ordered list of PTs

If no best match is found in a CSS search or they are the same then the partition with a partial match that is listed first wins.

 

Line / Device CSS is concatenated and Line is used first then device.

TOD applied to a PT effectively makes the PT disappear when it’s outside of the schedule.

 

You can use SDI traces to find CSS problems.

TCP handler assigned to the phone when registers identifies specific phone in trace output.

PSS = Partition search string

 

One Way Calling

Phone A can call phone B but phone B cannot call Phone A

Potential CSS issue on Phone B that doesn’t include the PT of Phone A’s DN.

 

Phone A cannot forward calls to another phone.

CSS on the I phone does not include the destination DN

Destination is invalid

Specified Destination is unregistered

 

Can’t forward to voicemail when not answered

CFNA is invalid or not specified

VM profile is wrong

VM ports are not available / busy

 

Can be viewed via RTMT by looking at the Hunt List for the voicemail HL

 

On-net multi-site calling issues

Overlapping dial plan

Site codes can be used to overcome digit overlap

Toll bypass settings

Local end CoS

Remote end CoS

Inter-cluster trunk settings

QoS

CAC Mechanisms

WAN Problems

QoS issues with signaling across WAN

CAC Mechanism prevents the call

Issues with CCD

 

PCMU – G711 codec in SIP SDP

 

Trunks between CUCM

GK could stop calls vi CAC

Trunk may not have registered to GK

Bad digit manipulation

 

CUBE issues:

Misconfiguration

MTP not allocated for H.323 to SIP

CSS and partition issues

Incorrect digit manipulation

CAC failure

 

CCD Issues

Misconfigured SAF trunks

CSS and partition issues

Patterns not advertised

Digit / pattern manipulation issues

Maximum number of learned patterns exceeded

 

GK Functions

ARQ – Can I place the call

ACF – Yes you can place the call and here is the IP you should send call setup to

 

GK Issues

Config errors

Registration issues

Duplicate ID

Terminal excluded

Security denial

Invalid alias

CAC issues

ACF received but get a busy tone

ARJ is null, not enough bandwidth

ARJ received, called party not registered

No tech prefix or no E.164 address for the call

 

Debug h225 asn1

Show gatekeeper endpoints

Debug ras

Debug gatekeeper main 5

Show gatekeeper calls

Show gatekeeper status

Show gatekeeper zone prefix

 

CUCM not registered to GK

Network issues

GK misconfigured

Endpoint misconfigured

Duplicate H.323 ID

Endpoint not authorized to register

 

CUCM registered to gK but phones can’t make calls

Connectivity between cluster issues

Dial plan misconfiguration

Incorrect IP address returned

Called party is not registered

Insufficient bandwidth due to CAC

 

GK calculates BW as double the Payload e.g. G729 8k time 2 = 16K and G711 128K

 

CUBE issues

CUCM misconfigured

CUBE misconfigured

Their configuration does not match

MTP not allocated, DTMF / Codec mismatch

CSS / PT issues

Incorrect digit manipulation on CUCM or CUBE

Call capacity exceeded, CAC failure

 

Show call active voice brief

Show voip rtp connection

 

Allow-connections are unidirectional

Allow-connections h323 to sip

Allow-connections sip to h323

 

Common off-site calling issues

Gateway config errors

Dial plan config errors

Route plan errors

Codec Issues

Location issues

GW issues

IOS config error

COR errors

Problems with QoS

Trunk issues

Invalid number dialed

 

MGCP GW troubleshooting

Go to router and verify mgcp commands

Ccm-manager commands

If PRI is Q931 backhauling configured

 

On CCM is the GW hostname configured properly

Check the endpoint configuration

Show ccm-manager

Show mgcp endpoint

Show mgcp statistic

Show mgcp connection

Debug mgcp error

Debug mgcp packet

Debug mgcp state

 

H323 GW Tshoot

Verify that the dial-peer commands are right

Verify that the voice class h225 timeout for tcp is set to 3 seconds

Use the preference command to determine the CUCM server order

Check that h323 binding is configured

Check RP / RG / RL settings in CUCM

Check the GW IP is correct in CUCM

 

Show dialplan number <number>

Debug vtsp session

Debug vtsp dsp

Debug voice dial

Show isdn status

Debug isdn q931

Debug voice ccapi inout

Debug voip dialpeer inout

Debug cch323 h225

Debug h225 q931

 

SIP GW troubleshooting

Voip dialpeers are configured correctly and have sipv2 enabled as well as a valid session target

SIP UA retry settings are verified

Proper DTMF relay is selected

SIP trunk in CUCM points to GW IP address

Show sip-ua status

Show sip status

Show sip statistics

Debug ccsip calls

 

Overlap receive is digit by digit analysis

 

When digit manipulation is done at RG level is not displayed on phone or in CDR

 

PT-S – Pretransformed Source

PT-D – Pretransformed Destination
T-S – Transformed Source

T-D – Transformed Destination

 

No digits at RP level over rides DDI at RG level

D – Device

G – Route Group

L – Route List

P – Route Pattern

 

Test voice translation-rule

 

No ringback on ip phone

PSTN is not providing ringback

H.323 GW is not cutting through audio

 

Dead air issue

Routing issues

IP ACL

Firewall fixup problems

 

Call drops midway

GW lost communication with CUCM

Remote end accidently hung up

The network had a connectivity event that affected the RTP path

System or Software error occurred

 

Outside dial tone only provided when all patterns left that possibly match have provide outside dial-tone

 

Issues with globalized call routing

Normal call routing issues plus

Unreachable internal number when calling inbound

Call back not possible

Bad transformations

 

Common SAF CCD issues

SAF no enabled

Wrong SAF external client credentials

Interface hellos are deactivated

SAF forwarders are not layer 2 adjacent, or wrong static neighbor configuration in place

Network issues

FW blocking traffic

 

 

External SAF client – Is on a different platform than the SAF forwarder (CUCM)

Internal SAF client – is on the same platform as the SAF forwarder (CUBE, CME, SRST)

 

Is SAF client registered?

Show eirgrp service-family ipv4 clients (detail)

RTMT can show SAF client statistics

 

Debug eirgrp service-family external-client messages

 

Show eigrp service-family ipv4 topology

 

Show eigrp service-family ipv4 events

 

Verify CCD learned patterns via RTMT > CUCM > Learned Patterns

 

Show voice saf dndb all

Patterns learned but marked as unreachable

Means there may be WAN issues

Wrong learned pattern prefix

Patterns placed in a PT not accessible via the CSS of the phones

 

The roaming sensitive settings applied if the physical location is different but the DMG is the same

The device mobility related settings are applied if the physical location is different but DMG is the same

The device mobility related settings do not change if the DMG is different

 

Device mobility CSS only affects the device CSS never the line CSS

 

Device mobility issue

Device mobility set to off on phone or service parameters

Problems with IP subnet configuration

Problems with CAC / codecs

SRST reference issues

Media resource Problems

QoS

CoS issues

 

To troubleshoot Device mobility look at the applied settings on the phone

Use CUCM DNA to check dial plan results

 

Line / device CSS necessary to ensure proper CSS when roaming.

 

Device default profile used when user profile does not match phone

 

Extension Mobility Issues

Various login / logout problems

Phone button problems

Phone service problems

Call routing problems

CoS problems

EMCC issues

 

Phone restarts instead of resetting – usually caused by different User Locale than phone.

Line / Device CSS needed to ensure proper CSS configuration

 

Cisco Unified Mobility

If we’re using MVA the MVA service must be enabled

Must have mobility softkey published to phone

Dusting (*74) feature allows call to be moved from Remote destination to the desk phone without holding the caller.

 

IF you can’t access MVA you probably have an VXML error on the gateway or wrong number assigned to the MVA app.

If you can’t reach destination numbers with MVA it may be an issue with the inbound CSS on the MVA gateway.

Or bad Dial-peer on the gateway.

 

Can’t use enterprise features.

Enterprise features are turned off on the service parameter

DTMF relay wrong on the gateway.

 

Three presence states

Unregistered

On-hook

Off-hook

 

BLF speed dial and presence in call history lists

Type A phones do not support directory

 

Presence groups only affect Directory Lists

Subscribe CSS affects the BLF subscription field

 

For presence to work across SIP trunks

Accept Presence Subscription

Accept Unsolicited notification

Subscript CSS on the trunk

Presence Group on the trunk

 

BLF for Call lists – enable at the enterprise parameter level

 

Unicast MoH doesn’t scale well and is BW intensive.

 

Most common MoH issues

MoH resource not registered

MOH resources are currently in use

Media resource misconfigured

TOH is heard instead of MOH

Call disconnected when placed on hold

MOH audio is poor

Multicast MOH is expected but unicast gets used

 

RTMT can show MOH streams in use

 

Tone on Hold when it should be music

Region issue

CAC blocking the BW for MOH

Audio streams not available on the MOH server

MOH server not active

Media resources misconfigured

 

Debug IP PIM – see multicast advertisements

Show ip mroute – shows multicast routing table

 

MTPs are used for repacketization a-law to u-law

Bridge connections using the same codec but different sample sizes

H.323 supplementary services

H.323 Outbound fast start

MTP needed when endpoints do not have a common method for sending DTMF between them.

 

 

Two non-sip endpoints do not require MTP

SCCP Phone and H323 GW

Two SIP endpoints do not require MTP

All Cisco SIP endpoints support NTEs (RFC2833)

DTMF is sent direction between the endpoints using NTE

No MTP required for G711 Calls

Combination of SIP and non-sip endpoints might require MTP

Depends on the endpoint

CUCM dynamically allocates MTPs on a call by call basis.

 

Issues related to MTP

No supplementary services available on H.323

Call setup fails when MTP required

DTMF issues when NTO not supported by an endpoint

Issues with mixed SIP endpoints

 

General MTP issues

MTP cannot register to CUCM – misconfigured / network problems.

MTP registered but not available to calls because of running out of resources / misconfiguration

 

Show dspfarm profile – can be used to see available HW MTP resources

Show sccp connections – shows mtp / sccp connections on a router

 

Meet-me

Dedicated DN for conference calls

Access controlled by CSS

 

 

Conference originator cannot add participants to Ad Hoc conference

Can’t link conferences together

Can’t setup meet-me conference

Can’t join a meet-me conference

Conference participants drop

 

SCCP media resources are case sensitive

DSPfarm  conference is shutdown by default

 

Conference originator determines the MGRL

 

Ad-hoc conf issues

No conf softkey

No conference bridge message shown on phone no mrgl / no available bridges

Maximum number of participants reached

Conf bridge doesn’t support codec

CAC rejects the conference bridge leg

Conf bridge out of resources

Cisco IP Voice Media Streaming App service not running

Advanced ad hoc not supported

Network issues

 

RTMT has conference performance counters

An end user can not join a meet-me conference via the softkey. Only the originator needs to press the meet-me conf key.

 

Show dspfarm dsp all

 

 

Device that cannot satisfy the end to end requirements (enforced by regions) is used. The device that requires the higher codec will invoke the transcoder.

Transcoder issues

Calls between endpoints with different codecs setup fails

Transcoder is not registered to CUCM

Misconfigured

Network issues

Transcoder registered but is running out of resources / misconfigured

If the Cisco IP Voice media streaming app is not running the transcoder will not register

DSPFarm could be admin down

MRG / MRGL misconfigured

 

RTMT can show current Transcoder resources available

 

MTP is used to represent an RSVP agent

RSVP Agent problems

RSVP CAC blocks calls when BW available

RSVP CAC does not make BW reservations when expected

No reservation

RSVP CAC not enabled

RSVP agents cannot register.

 

MTP SCCP admin state should be up and RSVP should shoe enabled

 

If call cannot be made

Too many reservations

IP phones are not associated to RSVP agents

Current IP paths goes over network segments that don’t have enough BW

RSVP per-flow BW needs to be set to 16kb/s more than actual codec will really require

Network connectivity issues

 

RTMT can be used to show RSVP connections and available BW / out of resources counters

 

Debug ip rsvp resv – can be used to see RSVP messages

Show ip RSVP neighbor – will show known enabled RSVP neighbors

Show ip rsvp installed detail

 

Local RSVP – within cluster only

E2E – means across SIP trunk to another cluster end to end RSVP

Send PRACK Enabled –

 

SIP Preconditions call fails

Not enough BW

Phones aren’t associated with RSVP

RSVP configured wrong

Network connectivity problems

Dial-plan problem

 

Delay below 150ms one way is normally acceptable for voice

Jitter – represents the variation in the delay of received packets

Echo – most commonly caused by impedance mismatch between 2-wire and 4-wire interfaces.

 

Latency < 150ms one way

Jitter < 30ms one way

Loss < 1% one way

17-106 Kb/s guaranteed priority BW

150 b/s + layer 2 overhead guaranteed BW for voice control traffic per call.

 

QoS Policy Implementation

Classification – ACL

Marking – DSCP tag (EF, CS4)

Congestion management – Policy-map

Congestion avoidance – WRED

Policing and shaping –

Link efficiency – LFI / Compression etc.

 

Layer 2 QoS issues

Buffer congestion is an issue

Make sure voice is mapped to the expedited egress queue

Enable the priority queue ingress and egress on the switch

QoS-scheduling

 

 

 

 

 

 

 

 

 

3T represents the drop threshold

SRR – Shaped round robin

Shaped mode every queue get a certain % of the BW and are rate limited

Shared mode minimum guarantee of BW and allowed to expand.

WRED

 

 

 

 

 

 

 

 

 

Ingress / egress queues use weight tail drop

Uses the frame’s assigned QoS label to subject it to different thresholds

ingress-clasficiation

 

 

 

 

 

 

 

 

 

egress-classification

 

 

 

 

 

 

 

queue-map

 

 

 

 

 

 

 

 

01-03 = Queue 1 (priority) Threshold 3 (least droppable)

delay-budge

 

 

 

 

 

 

 

 

You can use the QRT softkey to collect information about a poor quality call.

 Show call active voice – can show if the DSP is filling gaps in the stream of a call

jitter

 

 

 

 

 

 

 

 

 

Echo

Loudness on talker’s device set too high – high input level

Call proceed through tail circuits that do not use echo cancellers

Call path has end to end delay that is not covered by echo cancellers

Attenuation misconfigured on voice gateways

 

Identify types of echo, loud, long or acoustic.

Try first to move the remote phone from acoustic sources

Try replacing speakerphone or handset with better quality and see if it clears

echo

 

 

 

 

 

 

 

 

 

ERL – Echo Return Loss

 

One-way audio

cRTP on one end but not both of the links

ACL blocking return RTP traffic

Early versions of NAT could block RTP

Routing issues

Permanent link to this article: https://tripplehelix.net/tvoice-passed-notes/

Leave a Reply