CUCM Traces

Understanding CUCM Traces

AKHIL BEHL NCE, GDC UC CCIE (Voice, Security)

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

1

Session Objective  CUCM Traces – Introduction  SDI ttraces  SDL traces  Explain p types yp of traces  Trace configuration  Collecting trace files from CUCM  Common trace file locations and collection  Trace file reading and analysis  Best practices for collecting and configuring traces

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

2

CUCM Traces – Introduction

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

3

CUCM Trace Introduction 

CUCM has its own logging system. These are in form of txt files



These txt files are called Traces. They help in troubleshooting and are a very effective way to solve an issue



The trace Th t files fil can provide id in i depth d th d details t il off various i events t as th they occur with time stamp, event id, trace handles (TCP handle) etc

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

4

System Diagnostic Interface (SDI) Traces 

CUCM (also known as System Diagnostic Interface or SDI) traces are the user friendly call processing trace files we have available



Traces can be set at different levels to capture different activities

 

Detailed D t il d : Provides P id d detailed t il d d debug b iinformation f ti and d hi highly hl repetitive messages that are primarily used for debugging, including Keepalives and responses Arbitrary : Provides low-level debug traces. This level is best suited for debugging difficult problems. This level includes nearly everything that is included in Detailed with the exception of KeepAlives

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

5

 

 

Error : Provides traces generated in abnormal conditions, such as coding errors or other errors that normally should not occur Special S i l : Provides P id ttraces ffor allll iinformational, f ti l non-repetitive titi messages such as process startup messages, registration messages, and so on. All system and device initialization traces are at this level State Transition : Provides traces for call processing events or normal events traced for the subsystem (signaling layers) Significant : Provides trace for Media Layer events SCCP Signaling RTP Voice Stream

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

6

System Distribution Layer : SDL Traces  An SDL Trace log file contains call-processing information from services such as Cisco CallManager, Cisco CTIManager, and Cisco TFTP  Its important to understand how the SDL files are named – SDL trace filenames will be in the format: SDLxxx_yyy_zzzzzz.txt x is the node id , y is the application id, z is the trace file number  The node id is unique to each node in the cluster and is assigned at the time the server is added to the System->Communications Manager page in CCMAdmin  100 - CallManager (ccm), 200 - CTI Manager (ctimanager) etc SDL

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

7

SDL Trace Contents SDL trace contents Line Number | date + timestamp | node id | type of signal | signal name | state of receiving process | receiving process | sending process | tag field | signal queue depth | additional info Line Number starts at 000000000 when the service starts starts. It keeps incrementing until the service is restarted and the value is preserved across trace files The date/timestamp is the date and time when the trace file was written Node id is simply a reference within each SDL trace line indicating the id of the node that wrote the trace line The type of signal has multiple values Signal - the name of the signal

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

8

SDL Contents (Contd.) SDL trace file contents Line Number | date + timestamp | node id | type of signal | signal name | state of receiving process | receiving process | sending process | tag field | signal queue depth | additional info The state of the receiving process is the state before the signal is processed The receiving process is the PID of the process recieving the SDL signal The sending process is the PID of the process sending the SDL signal The tag field is miscellaneous information that is not commonly used in troubleshooting The signal queue depth outlines the number of signals waiting to be processed in each queue The additional info field is typically the data associated with the signal being sent to the receiving process. For an error it will contain details about the error. It is valid for this field to be empty AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

9

T Types off SDI Traces T

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

10

Callmanager : Types of Traces and their g meaning Cisco CallManager Trace Fields  Enable H245 Message Trace : This is used for H H.245 245 signaling for H.323 calls, including the media processing messages.  Enable PRI Trace : This activates a trace of Primary Rate Interface (PRI) devices.  Enable ISDN Translation Trace : This is used for a Layer 3 trace of Q.931 (ISDN messages).  Enable H225 & Gatekeeper Trace : This is used to trace the H.225 signaling messaging for H H.323 323 calls calls.  Enable Miscellaneous Trace : Activates a trace of miscellaneous devices.  E Enable bl C Conference f B Bridge id T Trace : Used U d tto perform f ttrace off the th conference bridges.  Enable CM Real-Time Information Server Trace : This is used for CallManager RIS traces - the real-time real time information server.  Enable CDR Trace : This enables tracing of call detail record (CDR) processing. AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

11

Callmanager : Types of Traces and their g (Contd.) ( ) meaning  Enable Analog Trunk Trace : This is used for a trace of all MGCP-based devices using an analog interface.  Enable All Phone Device Trace : To get a trace of phone devices, including Cisco IP SoftPhones, and shows events such as on-hook, off-hook, key presses, and so on.  Enable MTP Trace : This trace if for media termination point devices and transcoders.  Enable All Gateway Trace : This activates a trace of all analog and digital gateways.  Enable Forward and Miscellaneous Trace : Used for trace of call forwarding and all subsystems not covered by another checkbox checkbox.  Enable MGCP Trace : For MGCP messages for MGCP-based devices.  Enable Media Resource Manager g Trace : This is trace for media resource manager. AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

12

Other Traces  There are other service traces as well in addition to CUCM traces  CTI traces – to debug and troubleshoot CTI

related issues 

CDR – To troubleshoot CDR related issues

 DB and Admin – To troubleshoot DB sync issues, MLA logging etc  Performance and Monitoring – Same as windows Perfmon  Security services – CAPF related traces  Directory services – DirSync (LDAP trace) traces  Backup and Restore – Related to DRS traces  System Services – CCMWeb service, common interface etc traces  SOAP Services – AXL SOAP related traces  Platform services – OS Admin interface traces AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

13

CUCM Trace Configuration

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

14

CUCM Trace Configuration g  CUCM Trace – Configuration -> Ci Cisco Unified U ifi d serviceability i bilit -> Trace T -> Trace T configuration fi ti

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

15

CUCM Trace Configuration (Contd.)

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

16

Trace Collection

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

17

CUCM Trace Collection  CUCM Traces from CUCM 5.x onwards are collected using RTMT  RTMT is a plugin available in CUCM plugins  Download RTMT for windows and install it on your pc  Login with CUCM admin credentials with Pub’s IP  Login with desired profile (default is CM_Default)  Go G to t -> > Tools T l -> > trace t and d log l central t l

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

18

CUCM Trace Collection Procedure (Step by Step)  Step 1: Turn on Traces in CallManager Open up CallManager Admin page and navigate to Serviceability Serviceability. Then go to Trace Configuration, select publisher and choose the CallManager Service. Make sure Trace is on, and that debug trace level is set to Detailed. We will also see a lot of options with checkboxes. Make sure that the “apply to all nodes” d ” iis also l checked h k d and d click li k save  Step 2: Download and Install RTMT ((Real Time Monitoring g Tool)) This tool is accessed from the normal CallManager Administration page. It can be found under Application Plugins. Download and install this tool. Make sure we can run the RTMT tool and log in using the admin credentials you use to t log l iinto t th the web b admin d i page. Ch Choose T Tools l ->Trace T ->Trace T & Log L Central (ensure that is there’s a firewall between management PC and CUCM, the ports for RTMT are opened – 1090 and 1099 TCP )

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

19

CUCM Trace Collection Procedure (Step by Step) Contd.  Step 3. Place Your Test Call Once we have RTMT loaded up up, we need to place your test call or create the condition we are trying to capture. Try to repeat the problem re creation twice so that, we can be sure that we can find it  Step 4. Collect Trace Files Now go to RTMT window that we opened in step 2, and double click “Collect Files”. Find CallManager in the list Files list, and click the checkbox for “All All Servers” Servers since we may have a situation where a phone is registered to one server and the gateway is registered to another. This is good info to have beforehand so the search for the trace is a bit easier. Now for the file options, choose relative range and d go b back k 5 or 10 minutes. i t M Make k sure whatever h t we choose h iis sufficient to capture the test call or error condition

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

20

CUCM Trace Collection  Click Collect files

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

21

CUCM Trace Collection (Contd.)  Choose CUCM traces (and any other traces desired)

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

22

CUCM Trace Collection (Contd.)  Accept the defaults for save (except delete collected log files from server and also select Zip Files)  Select S l t th the d date t and d ti time ffrom which hi h ttraces are tto b be analyzed l d

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

23

CUCM Trace Collection (Contd.)  The RTMT tool will collect the files for the selected time period  The files will be saved in selected location (default is C:\Program Fil \Ci Files\Cisco\CallManager \C llM Serviceability\JRtmt\ S i bilit \JRt t\

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

24

Common Trace locations and collection

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

25

CUCM frequently used traces and file location  SDI traces - /var/log/active/cm/trace/ccm/sdi  SDL traces - /var/log/active/cm/trace/ccm/sdl  Perfmon logs - /var/log/active/cm/log/ris/csv and /var/log/active/cm/log/amc/PerfMonLog  Callmanager app log - /var/log/active/syslog/CallManager*.* /var/log/active/syslog/CallManager* * Note – to get the files use the syntax - file get activelog cm/trace/ccm/sd*/*  Equivalent to CLI trace collection, in RTMT this can be done as follows –  For UCM sdl and sdi traces, check the "Cisco CallManager" box for each server  For RIS Perfon logs, check the box "Cisco RIS Data Collector PerfMonLog" for each server you are interested in  For the System Event Logs and Application logs, check the box for "Event Event Viewer System Log and Application log" for each server you are interested in AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

26

Reading CUCM Traces

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

27

Reading g CallManager g SDI Traces How to Read a Trace ?  Stationinit is a message from the end point to the CallManager.  StationD is the message from CallManager to endpoint.  Start finding dd=“xxx, where dd is Dialed Digits and xxx is number dialled  When you get the Dialed String, make sure to note the TCP Handle because it doesn’t changes for a Phone unless its reset.

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

28

Reading CallManager Traces (Contd.)  Now, Now follow the tcp handle to go with flow of call  Alternatively y (less ( preferred p method)) yyou can follow the Device ID (Mac address) or the IP of the endpoint  If a conference f is i involved, i l d a new ttcp handle h dl iis generated. t d S So, one now has to follow this new tcp handle till call lasts or conference lasts  The dialed string space also gives a clear picture of CSS, PSS, translation pattern hit, Route Pattern hit etc

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

29

Example p Trace File Trace File

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

30

An Example of CFB Trace  In this trace we can see how the CFB Resources are configured on the CallManager CallManager. Also we can see the error of resource unavailable unavailable. CallManager|UnicastBridgeControl::StationRegister MaxStream=0, ActiveStream=0| 04/25/2002 16:41:58.948 Cisco CallManager|UnicastBridgeControl::StationRegister CurrentAvail= 0 MaxAvail=0| 04/25/2002 16:41:58.838 Cisco IP Voice Media Streaming App|kRequestedCFBStreamsFailed - CFB requested streams failure. Requested streams:48 Allocated streams:0 App ID:Cisco IP Voice Media Streaming App Cluster ID:CALLMANAGER01-Cluster Node ID:10.7.0.25| 04/25/2002 16:41:34.306 Cisco IP Voice Media Streaming App|CIpVMSMgr::ReadConfiguration CFB DeviceName = CFB_CALLMANAGER| 04/25/2002 16:41:34.322 Cisco IP Voice Media Streaming App|CIpVMSMgr::ReadConfiguration CFB ServerName[0] = 10.7.0.25, Port = 2000| 04/25/2002 16:41:34.338 Cisco IP Voice Media Streaming App|CIpVMSMgr::ReadConfiguration CFB Number of Parties = 48| 04/25/2002 16:41:34.338 Cisco IP Voice Media Streaming App|CIpVMSMgr::ReadConfiguration CFB Run Flag 1| 7 0 25> =1|CTftpConnect::tftp[3976:132.143.64.20:25000] | 02/20/2008 17:34:15.144 TFTP|CTftpConnect::checkAndHandleDiagReq[0268F1F8],[3976:132.143.64.20:25000], File[SEP00506003606C.cnf.xml]| 02/20/2008 17:34:15.144 TFTP| CTftpConnect::tftp[3976:132.143.64.20:25000] [SEP00506003606C.cnf.xml] not found in bin/config cache Looking at disk|

 This issue was resolved by rebuilding the CNF files from TFTP service parameters AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

33

An Example p of PRI Trace  PRI Trace showing issue with an incoming call on MGCP gateway 03/09/2010 11:26:46.940 CCM|MGCPpn9d - S0/SU1/DS1-1@FFX-BW-7-VoIP-GW - Collision Detected in PriNi2 Interface, callphase= 9| 03/09/2010 11:26:46.940 CCM|Entering | g default,restarting , g channel| 03/09/2010 11:26:46.941 CCM|MGCPpn9d::conflictingChannelHandler - Don't Restart this channel since it is marked OOS_NE under service parameter:S0/SU1/DS1-1@FFX-BW-7-VoIP-GW= or due to MGCP gateway hardware failure|

 This was a Hardware issue as all channels were in service  This issue was resolved by upgrading DSP firmware

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

34

Best est Practices act ces for o Trace reading

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

35

Best Practices for Trace Reading g  Always, have an Idea about the time period when issue occurs. It becomes very very difficult to go through hundreds of Trace files  Always y Understand the Call Flow before troubleshooting g  Information about network setup and topology always helps understand d t d th the fl flow off signaling i li and d media di  Always try to save older trace files and begin afresh if debugging a new issue

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

36

Q&A?

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

37

Th k You Thank Y

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

38

AKHIL BEHL CUCM Traces © 2010 Cisco Systems, Inc. All rights reserved. Cisco Confidential

39