TVoice 8.0 Passed – Have Some Notes

So I should have posted these a while ago when I finished by CCNP Voice but better late than never here are some notes on the TVoice 8.0 subject material.

Test blueprint (CCO Required)

4 Elements of CUCM Environment

  • Call Agent  / CUCM
  • Network Infrastructure
  • Cisco Applications
    • CUPS
    • Unity Connections
    • UCCX
  • Voice Clients


Network Infrastructure

  • Routers
  • Switches
  • Firewalls



  • Primary Call Agent


Voice Clients

  • Physical Phone
  • SoftPhone
  • 3rd Party SIP Phone


Cisco Applications

  • CUPS
  • CUC
  • UCCX



Possible Sources of Problems

  • Complex deployments
  • Various protocols
  • Various endpoints and services
  • Network infrastructure


Benefits of Systematic troubleshooting method

  • Easier to identify potential problems
  • Learning effect
    • Understanding what is going on where there is no problem:
      • Tracking down internal processes such as call flows
      • Knowing corner case
    • Better ability to spot anomalies
  • Systematic documentation helps to solve future issues.


Problem Solving Model

Define Problem

Gather Facts

Consider Possibilities

Create Action Plan

Implement Action Plan

Observe Results

Did it resolve the problem – no go back to start.


Problem resolved

Document the facts


Troubleshooting Tools

  • Cisco Unified Serviceability
    • Alarms
    • Setting Trace
    • CAR
    • Control Center
    • DNA
  • RTMT
    • Alerts
    • Viewing Traces
    • Syslog Viewer
    • Performance Monitoring
  • CLI
  • Show commands
  • Debug commands
  • Packet sniffer
  • Cisco Unified Operations Manager



Local / External Syslog

SDI Trace

SDL Trace


Trace Information

SDI – System Diagnostic Interface

  • Run-time events
  • IP Address
  • TCP Handles
  • Date / Time

SDL – Signaling Distribution Layer

  • Call processing information from CUCM / CTI Manager
  • Used by Cisco TAC

Log4J – Used for Java Applications


Two ways to configure traces

Activate Troubleshooting Traces

Configure Traces on a per server / per service level


Max number of devices traced (default is 12)

FQCN – fully qualified calling number



Sent from phone to CUCM

  • Button Presses


From CUCM to Phone

  • Change Phone display
  • Turn on MWI

CCM|Digit analysis






Dialed Number Analyzer

Used to find out if we could call from phone X to phone Y


RTMT Syslog Logs





Cisco Unified Reporting

GCFM – Generic Call Filter Module

Call filter match-list (up to 16)

Used for filtering the debug output from a router for a specific ANI / DNIS


Phone registration process


Cisco ILP – in line power

Fast Link Pulse – FLP

802.3af POE


Most common problem is mismatch of inline power between endpoint and switch


Common DHCP issues

  • DHCP server down
  • DHCP does not have scope for the phone
  • All addresses leased out
  • IP Helper not setup


Common TFTP issues

  • Scope does not offer option 150
  • Option 150 points to wrong address
  • TFTP  server on the address is stopped or hung


Try to Ping the phones IP address to see if the CM server can reach the phone via OS Admin or via CLI utils network ping

Check TFTP config on phone / DHCP servers

Check Phone status messages


MGCP registers to CUCM, GW must be configured to reach out to CUCM and the domain name must be the same as is configured in CUCM.


MGCP GW sends Restart in Progress


CUCM Audit Endpoint


CUCM Request Notification



GW Registered



MGCP Problems

IP connectivity problems

Wrong TFTP server / CUCM not running TFTP

IOS SW version incompatible

Missing or incorrect CUCM server

Incorrect IOS configuration


Show ccm-manager – shows MGCP status

Show mgcp endpoint – shows the endpoints that are controlled by ccm

Show mgcp connection – shows the active connections (calls)

Show mgcp statistics – shows the mgcp message stats

Show mgcp srtp – displays information about SRTP connections

Debug mgcp – enables debugs for the mgcp protocol

Debug ccm-manager events – see the events as the MGCP gateway is attempting to register


H.323 / SIP – do not register to CUCM. CUCM will always show unknown.


Dial peer monitoring

Show dial-peer voice

Debug voip dialpeer

H.323 GW monitoring

Show gateway

Debug h225 asn1 | events

Debug h225 q931

SIP GW Monitoring

Show sip-ua calls, connections, statistics, status

Show sip service


Delayed dial tone / CCM Admin page slow – CUCM server over burdened

  • CPU Usage high
  • Service crash
  • Service Hung
  • Memory Leak


Syslog messages can be used to view alerts

RTMT can be used to view CPU / Memory Utilization


Phones not registering

Maybe CCM service is not running or TFTP service isn’t running


Can’t reach CCM admin page

Tomcat might not be running – can be restarted via CLI

IS DNS resolving the ccm host name

Can we ping the CCM server

ACL / FW change?


Utils service list

Utils start Cisco Tomcat


Check Duplex / Speed for issues related to “slow network”


DB replication issues

Pub has a R/W copy of the DB Subs have R/O copy of the DB


Replication issue example: Changed phone setting, but it never shows up

IDS Replication – From PUB to SUBS

Static information phone configuration

ICCS (Intra-cluster communication Service) – Full mesh between all call processors (subscribers)

Dynamic runtime information


Why DB goes out of sync

PUB can’t reach subscribers – network issues

QoS not setup to prefer DB replication traffic

DNS issue

Server CPU is taxed

Error occurred during the DB replication process


Verify NW connectivity

Verify DNS resolution works

Make sure there is enough BW for replication and the RTT not greater than 80ms between pub and sub


RTMT / CLI can be used to verify replication state.

Replication state should be 2 which is good replication

Cisco Unified Reporting – Unified CM Database Status


Utils dbreplication status

File view activelog cm/trace/….. To view log file

Utils dbreplication runtimestate – shows realtime replication status across the servers


Utils dbreplication repair – if a status of 0 or 4 after 4 or more hours use dbreplication reset to stop and restart database replication.


Run the repair command at the publisher only

Wait until completed displayed from runtime state.


Go to subscriber, utils dbreplication stop – stops the replication on the subscriber

Go to Publisher, utils dbreplication reset (subscriber name)


If that doesn’t work a cluster reset is needed.


Stop dbreplication on all nodes

From the pub, utils dbreplication cluster reset


Troubleshooting LDAP Integration Issues

Can sync and auth against LDAP


Cisco DirSync must be activated / running

Can’t mix LDAP server types


When user deleted in LDAP next CUCM sync will tombstone the user object. The user will be deleted after 24 hours.


Common issues:

Network connectivity

Utils network ping

Service username and password wrong


Service user doesn’t have sufficient permissions

Needs read access to all AD user objects

Search base wrong

Check the DN

DirSync set to manual or too long a period

Check LDAP sync schedule in CUCM LDAP directory setup

DirSync not running

Check that DirSync service is activated and running on publisher

Some end-users missing

Check the search base would encompass where the user is located in LDAP


DirSync has Service Parameters that can be set.


H.323 GW issue

Call into H323 gateway just gives secondary dial-tone

Direct-inward-dial missing from inbound dial-peer


Call Setup issue

Fast busy

Missing / wrong Caller ID

No ring back

Dead air

One-way calling

Inefficient call routing

Unexpected second dial tone



Bad CSS / wrong dialed number

Bad digit manipulation

Phone unregistered

Codec mis-match / region misconfiguration

No MTP resources

Bad QoS / no QoS

Configured wrong in CUCM

RSVP configured wrong

CAC blocking call due to no enough BW

WAN having problems

ICT setup wrong between clusters

CCD not configured right

CUBE configured wrong


Single site call setup issues

CoS misconfigured

Bad digit manipulation

FW to VM and VM is down

Unregistered phone


CUCM call routing

Digit by digit analysis – most specific match logic


IP Phone

SCCP – Digit by Digit

SIP – En bloc / KPML / SIP Dial Rules



MGCP / SIP / H323 – En block / overlap send and receiving



SIP / H323 – En bloc / overlap sending and receiving


Partitions and CSS allow / deny calls

A device can only call those numbers that are in PT that are a part of its CSS


<None> PT is always accessible regardless if the calling device has a CSS

Devices that have the <None> CSS can access only other devices that are in the Partition of <None>

CSS is an ordered list of PTs

If no best match is found in a CSS search or they are the same then the partition with a partial match that is listed first wins.


Line / Device CSS is concatenated and Line is used first then device.

TOD applied to a PT effectively makes the PT disappear when it’s outside of the schedule.


You can use SDI traces to find CSS problems.

TCP handler assigned to the phone when registers identifies specific phone in trace output.

PSS = Partition search string


One Way Calling

Phone A can call phone B but phone B cannot call Phone A

Potential CSS issue on Phone B that doesn’t include the PT of Phone A’s DN.


Phone A cannot forward calls to another phone.

CSS on the I phone does not include the destination DN

Destination is invalid

Specified Destination is unregistered


Can’t forward to voicemail when not answered

CFNA is invalid or not specified

VM profile is wrong

VM ports are not available / busy


Can be viewed via RTMT by looking at the Hunt List for the voicemail HL


On-net multi-site calling issues

Overlapping dial plan

Site codes can be used to overcome digit overlap

Toll bypass settings

Local end CoS

Remote end CoS

Inter-cluster trunk settings


CAC Mechanisms

WAN Problems

QoS issues with signaling across WAN

CAC Mechanism prevents the call

Issues with CCD


PCMU – G711 codec in SIP SDP


Trunks between CUCM

GK could stop calls vi CAC

Trunk may not have registered to GK

Bad digit manipulation


CUBE issues:


MTP not allocated for H.323 to SIP

CSS and partition issues

Incorrect digit manipulation

CAC failure


CCD Issues

Misconfigured SAF trunks

CSS and partition issues

Patterns not advertised

Digit / pattern manipulation issues

Maximum number of learned patterns exceeded


GK Functions

ARQ – Can I place the call

ACF – Yes you can place the call and here is the IP you should send call setup to


GK Issues

Config errors

Registration issues

Duplicate ID

Terminal excluded

Security denial

Invalid alias

CAC issues

ACF received but get a busy tone

ARJ is null, not enough bandwidth

ARJ received, called party not registered

No tech prefix or no E.164 address for the call


Debug h225 asn1

Show gatekeeper endpoints

Debug ras

Debug gatekeeper main 5

Show gatekeeper calls

Show gatekeeper status

Show gatekeeper zone prefix


CUCM not registered to GK

Network issues

GK misconfigured

Endpoint misconfigured

Duplicate H.323 ID

Endpoint not authorized to register


CUCM registered to gK but phones can’t make calls

Connectivity between cluster issues

Dial plan misconfiguration

Incorrect IP address returned

Called party is not registered

Insufficient bandwidth due to CAC


GK calculates BW as double the Payload e.g. G729 8k time 2 = 16K and G711 128K


CUBE issues

CUCM misconfigured

CUBE misconfigured

Their configuration does not match

MTP not allocated, DTMF / Codec mismatch

CSS / PT issues

Incorrect digit manipulation on CUCM or CUBE

Call capacity exceeded, CAC failure


Show call active voice brief

Show voip rtp connection


Allow-connections are unidirectional

Allow-connections h323 to sip

Allow-connections sip to h323


Common off-site calling issues

Gateway config errors

Dial plan config errors

Route plan errors

Codec Issues

Location issues

GW issues

IOS config error

COR errors

Problems with QoS

Trunk issues

Invalid number dialed


MGCP GW troubleshooting

Go to router and verify mgcp commands

Ccm-manager commands

If PRI is Q931 backhauling configured


On CCM is the GW hostname configured properly

Check the endpoint configuration

Show ccm-manager

Show mgcp endpoint

Show mgcp statistic

Show mgcp connection

Debug mgcp error

Debug mgcp packet

Debug mgcp state


H323 GW Tshoot

Verify that the dial-peer commands are right

Verify that the voice class h225 timeout for tcp is set to 3 seconds

Use the preference command to determine the CUCM server order

Check that h323 binding is configured

Check RP / RG / RL settings in CUCM

Check the GW IP is correct in CUCM


Show dialplan number <number>

Debug vtsp session

Debug vtsp dsp

Debug voice dial

Show isdn status

Debug isdn q931

Debug voice ccapi inout

Debug voip dialpeer inout

Debug cch323 h225

Debug h225 q931


SIP GW troubleshooting

Voip dialpeers are configured correctly and have sipv2 enabled as well as a valid session target

SIP UA retry settings are verified

Proper DTMF relay is selected

SIP trunk in CUCM points to GW IP address

Show sip-ua status

Show sip status

Show sip statistics

Debug ccsip calls


Overlap receive is digit by digit analysis


When digit manipulation is done at RG level is not displayed on phone or in CDR


PT-S – Pretransformed Source

PT-D – Pretransformed Destination
T-S – Transformed Source

T-D – Transformed Destination


No digits at RP level over rides DDI at RG level

D – Device

G – Route Group

L – Route List

P – Route Pattern


Test voice translation-rule


No ringback on ip phone

PSTN is not providing ringback

H.323 GW is not cutting through audio


Dead air issue

Routing issues


Firewall fixup problems


Call drops midway

GW lost communication with CUCM

Remote end accidently hung up

The network had a connectivity event that affected the RTP path

System or Software error occurred


Outside dial tone only provided when all patterns left that possibly match have provide outside dial-tone


Issues with globalized call routing

Normal call routing issues plus

Unreachable internal number when calling inbound

Call back not possible

Bad transformations


Common SAF CCD issues

SAF no enabled

Wrong SAF external client credentials

Interface hellos are deactivated

SAF forwarders are not layer 2 adjacent, or wrong static neighbor configuration in place

Network issues

FW blocking traffic



External SAF client – Is on a different platform than the SAF forwarder (CUCM)

Internal SAF client – is on the same platform as the SAF forwarder (CUBE, CME, SRST)


Is SAF client registered?

Show eirgrp service-family ipv4 clients (detail)

RTMT can show SAF client statistics


Debug eirgrp service-family external-client messages


Show eigrp service-family ipv4 topology


Show eigrp service-family ipv4 events


Verify CCD learned patterns via RTMT > CUCM > Learned Patterns


Show voice saf dndb all

Patterns learned but marked as unreachable

Means there may be WAN issues

Wrong learned pattern prefix

Patterns placed in a PT not accessible via the CSS of the phones


The roaming sensitive settings applied if the physical location is different but the DMG is the same

The device mobility related settings are applied if the physical location is different but DMG is the same

The device mobility related settings do not change if the DMG is different


Device mobility CSS only affects the device CSS never the line CSS


Device mobility issue

Device mobility set to off on phone or service parameters

Problems with IP subnet configuration

Problems with CAC / codecs

SRST reference issues

Media resource Problems


CoS issues


To troubleshoot Device mobility look at the applied settings on the phone

Use CUCM DNA to check dial plan results


Line / device CSS necessary to ensure proper CSS when roaming.


Device default profile used when user profile does not match phone


Extension Mobility Issues

Various login / logout problems

Phone button problems

Phone service problems

Call routing problems

CoS problems

EMCC issues


Phone restarts instead of resetting – usually caused by different User Locale than phone.

Line / Device CSS needed to ensure proper CSS configuration


Cisco Unified Mobility

If we’re using MVA the MVA service must be enabled

Must have mobility softkey published to phone

Dusting (*74) feature allows call to be moved from Remote destination to the desk phone without holding the caller.


IF you can’t access MVA you probably have an VXML error on the gateway or wrong number assigned to the MVA app.

If you can’t reach destination numbers with MVA it may be an issue with the inbound CSS on the MVA gateway.

Or bad Dial-peer on the gateway.


Can’t use enterprise features.

Enterprise features are turned off on the service parameter

DTMF relay wrong on the gateway.


Three presence states





BLF speed dial and presence in call history lists

Type A phones do not support directory


Presence groups only affect Directory Lists

Subscribe CSS affects the BLF subscription field


For presence to work across SIP trunks

Accept Presence Subscription

Accept Unsolicited notification

Subscript CSS on the trunk

Presence Group on the trunk


BLF for Call lists – enable at the enterprise parameter level


Unicast MoH doesn’t scale well and is BW intensive.


Most common MoH issues

MoH resource not registered

MOH resources are currently in use

Media resource misconfigured

TOH is heard instead of MOH

Call disconnected when placed on hold

MOH audio is poor

Multicast MOH is expected but unicast gets used


RTMT can show MOH streams in use


Tone on Hold when it should be music

Region issue

CAC blocking the BW for MOH

Audio streams not available on the MOH server

MOH server not active

Media resources misconfigured


Debug IP PIM – see multicast advertisements

Show ip mroute – shows multicast routing table


MTPs are used for repacketization a-law to u-law

Bridge connections using the same codec but different sample sizes

H.323 supplementary services

H.323 Outbound fast start

MTP needed when endpoints do not have a common method for sending DTMF between them.



Two non-sip endpoints do not require MTP

SCCP Phone and H323 GW

Two SIP endpoints do not require MTP

All Cisco SIP endpoints support NTEs (RFC2833)

DTMF is sent direction between the endpoints using NTE

No MTP required for G711 Calls

Combination of SIP and non-sip endpoints might require MTP

Depends on the endpoint

CUCM dynamically allocates MTPs on a call by call basis.


Issues related to MTP

No supplementary services available on H.323

Call setup fails when MTP required

DTMF issues when NTO not supported by an endpoint

Issues with mixed SIP endpoints


General MTP issues

MTP cannot register to CUCM – misconfigured / network problems.

MTP registered but not available to calls because of running out of resources / misconfiguration


Show dspfarm profile – can be used to see available HW MTP resources

Show sccp connections – shows mtp / sccp connections on a router



Dedicated DN for conference calls

Access controlled by CSS



Conference originator cannot add participants to Ad Hoc conference

Can’t link conferences together

Can’t setup meet-me conference

Can’t join a meet-me conference

Conference participants drop


SCCP media resources are case sensitive

DSPfarm  conference is shutdown by default


Conference originator determines the MGRL


Ad-hoc conf issues

No conf softkey

No conference bridge message shown on phone no mrgl / no available bridges

Maximum number of participants reached

Conf bridge doesn’t support codec

CAC rejects the conference bridge leg

Conf bridge out of resources

Cisco IP Voice Media Streaming App service not running

Advanced ad hoc not supported

Network issues


RTMT has conference performance counters

An end user can not join a meet-me conference via the softkey. Only the originator needs to press the meet-me conf key.


Show dspfarm dsp all



Device that cannot satisfy the end to end requirements (enforced by regions) is used. The device that requires the higher codec will invoke the transcoder.

Transcoder issues

Calls between endpoints with different codecs setup fails

Transcoder is not registered to CUCM


Network issues

Transcoder registered but is running out of resources / misconfigured

If the Cisco IP Voice media streaming app is not running the transcoder will not register

DSPFarm could be admin down

MRG / MRGL misconfigured


RTMT can show current Transcoder resources available


MTP is used to represent an RSVP agent

RSVP Agent problems

RSVP CAC blocks calls when BW available

RSVP CAC does not make BW reservations when expected

No reservation

RSVP CAC not enabled

RSVP agents cannot register.


MTP SCCP admin state should be up and RSVP should shoe enabled


If call cannot be made

Too many reservations

IP phones are not associated to RSVP agents

Current IP paths goes over network segments that don’t have enough BW

RSVP per-flow BW needs to be set to 16kb/s more than actual codec will really require

Network connectivity issues


RTMT can be used to show RSVP connections and available BW / out of resources counters


Debug ip rsvp resv – can be used to see RSVP messages

Show ip RSVP neighbor – will show known enabled RSVP neighbors

Show ip rsvp installed detail


Local RSVP – within cluster only

E2E – means across SIP trunk to another cluster end to end RSVP

Send PRACK Enabled –


SIP Preconditions call fails

Not enough BW

Phones aren’t associated with RSVP

RSVP configured wrong

Network connectivity problems

Dial-plan problem


Delay below 150ms one way is normally acceptable for voice

Jitter – represents the variation in the delay of received packets

Echo – most commonly caused by impedance mismatch between 2-wire and 4-wire interfaces.


Latency < 150ms one way

Jitter < 30ms one way

Loss < 1% one way

17-106 Kb/s guaranteed priority BW

150 b/s + layer 2 overhead guaranteed BW for voice control traffic per call.


QoS Policy Implementation

Classification – ACL

Marking – DSCP tag (EF, CS4)

Congestion management – Policy-map

Congestion avoidance – WRED

Policing and shaping –

Link efficiency – LFI / Compression etc.


Layer 2 QoS issues

Buffer congestion is an issue

Make sure voice is mapped to the expedited egress queue

Enable the priority queue ingress and egress on the switch











3T represents the drop threshold

SRR – Shaped round robin

Shaped mode every queue get a certain % of the BW and are rate limited

Shared mode minimum guarantee of BW and allowed to expand.











Ingress / egress queues use weight tail drop

Uses the frame’s assigned QoS label to subject it to different thresholds




























01-03 = Queue 1 (priority) Threshold 3 (least droppable)










You can use the QRT softkey to collect information about a poor quality call.

 Show call active voice – can show if the DSP is filling gaps in the stream of a call












Loudness on talker’s device set too high – high input level

Call proceed through tail circuits that do not use echo cancellers

Call path has end to end delay that is not covered by echo cancellers

Attenuation misconfigured on voice gateways


Identify types of echo, loud, long or acoustic.

Try first to move the remote phone from acoustic sources

Try replacing speakerphone or handset with better quality and see if it clears











ERL – Echo Return Loss


One-way audio

cRTP on one end but not both of the links

ACL blocking return RTP traffic

Early versions of NAT could block RTP

Routing issues

Permanent link to this article:

Leave a Reply