Troubleshooting Mcrnc
Short Description
Troubleshooting Mcrnc...
Description
Nokia Networks
WCDMA RAN, Rel. RU50 and RU50 EP1, Operating Documentation, Issue 05 Troubleshooting Multicontroller RNC DN0976768 Issue 02B Approval Date 2015-02-23
Troubleshooting Multicontroller RNC
The information in this document applies solely to the hardware/software product (“Product”) specified herein, and only as specified herein. This document is intended for use by Nokia Solutions and Networks' customers (“You”) only, and it may not be used except for the purposes defined in the agreement between You and Nokia Solutions and Networks (“Agreement”) under which this document is distributed. No part of this document may be used, copied, reproduced, modified or transmitted in any form or means without the prior written permission of Nokia Solutions and Networks. If you have not entered into an Agreement applicable to the Product, or if that Agreement has expired or has been terminated, You may not use this document in any manner and You are obliged to return it to Nokia Solutions and Networks and destroy or delete any copies thereof. The document has been prepared to be used by professional and properly trained personnel, and You assume full responsibility when using it. Nokia Solutions and Networks welcome Your comments as part of the process of continuous development and improvement of the documentation. This document and its contents are provided as a convenience to You. Any information or statements concerning the suitability, capacity, fitness for purpose or performance of the Product are given solely on an “as is” and “as available” basis in this document, and Nokia Solutions and Networks reserves the right to change any such information and statements without notice. Nokia Solutions and Networks has made all reasonable efforts to ensure that the content of this document is adequate and free of material errors and omissions, and Nokia Solutions and Networks will correct errors that You identify in this document. But, Nokia Solutions and Networks' total liability for any errors in the document is strictly limited to the correction of such error(s). Nokia Solutions and Networks does not warrant that the use of the software in the Product will be uninterrupted or error-free. NO WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY OF AVAILABILITY, ACCURACY, RELIABILITY, TITLE, NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, IS MADE IN RELATION TO THE CONTENT OF THIS DOCUMENT. IN NO EVENT WILL NOKIA SOLUTIONS AND NETWORKS BE LIABLE FOR ANY DAMAGES, INCLUDING BUT NOT LIMITED TO SPECIAL, DIRECT, INDIRECT, INCIDENTAL OR CONSEQUENTIAL OR ANY LOSSES, SUCH AS BUT NOT LIMITED TO LOSS OF PROFIT, REVENUE, BUSINESS INTERRUPTION, BUSINESS OPPORTUNITY OR DATA THAT MAY ARISE FROM THE USE OF THIS DOCUMENT OR THE INFORMATION IN IT, EVEN IN THE CASE OF ERRORS IN OR OMISSIONS FROM THIS DOCUMENT OR ITS CONTENT. This document is Nokia Solutions and Networks’ proprietary and confidential information, which may not be distributed or disclosed to any third parties without the prior written consent of Nokia Solutions and Networks. Nokia is a registered trademark of Nokia Corporation. Other product names mentioned in this document may be trademarks of their respective owners, and they are mentioned for identification purposes only. Copyright © 2015 Nokia Solutions and Networks. All rights reserved.
f
Important Notice on Product Safety This product may present safety risks due to laser, electricity, heat, and other sources of danger. Only trained and qualified personnel may install, operate, maintain or otherwise handle this product and only after having carefully read the safety information applicable to this product. The safety information is provided in the Safety Information section in the “Legal, Safety and Environmental Information” part of this document or documentation set.
Nokia Solutions and Networks is continually striving to reduce the adverse environmental effects of its products and services. We would like to encourage you as our customers and users to join us in working towards a cleaner, safer environment. Please recycle product packaging and follow the recommendations for power use and proper disposal of our products and their components. If you should have questions regarding our Environmental Policy or any of the environmental services we offer, please contact us at Nokia Solutions and Networks for any additional information.
2
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
Table of Contents This document has 71 pages
Summary of changes..................................................................... 7
Issue: 02B
1 1.1 1.2 1.3 1.4
Overview of Multicontroller RNC Troubleshooting......................... 8 Troubleshooting Multicontroller RNC recommendations................8 Information sources in fault situations............................................9 Problem types.............................................................................. 10 Generic troubleshooting procedure.............................................. 11
2
Reporting problem to NSN........................................................... 14
3 3.1 3.2 3.3 3.4 3.5
Symptoms data collection............................................................ 19 Standard symptoms report...........................................................19 Collecting standard symptom reports...........................................23 Listing the standard symptom reports.......................................... 28 Copying standard symptoms reports to remote machine.............29 Deleting the standard symptom reports....................................... 30
4 4.1 4.2 4.3
Generic software troubleshooting instructions............................. 32 Displaying all blackbox files......................................................... 32 Viewing active alarms and alarms history.................................... 33 Collecting core dump file information........................................... 34
5 5.1 5.1.1 5.1.2 5.1.3
Configuration management troubleshooting................................ 36 Saving configuration snapshot fails..............................................36 Description................................................................................... 36 Symptoms.................................................................................... 36 Recovery procedures................................................................... 36
6 6.1 6.1.1 6.1.2 6.1.3 6.2 6.2.1 6.2.2 6.2.3 6.3 6.3.1 6.3.2 6.3.3 6.3.4 6.3.5
Software management troubleshooting....................................... 37 Installation of a new software delivery fails.................................. 37 Description................................................................................... 37 Symptoms.................................................................................... 37 Recovery procedures................................................................... 37 Activation of a new software delivery fails....................................37 Description................................................................................... 37 Symptoms.................................................................................... 38 Recovery procedures................................................................... 38 Troubleshooting software configuration management................. 38 Executing pre-download script fails..............................................38 Parsing targetBD.xml fails............................................................39 Executing pre-activation script fails..............................................39 Upgrading eSW fails.................................................................... 40 Alarm 2518 - NO VALID FALLBACK COPY FOR DEFAULT PACKAGE is triggered................................................................. 41
DN0976768
3
Troubleshooting Multicontroller RNC
4
7 7.1 7.1.1 7.1.2 7.1.3 7.2 7.2.1 7.2.2 7.2.3 7.3 7.3.1 7.3.2 7.3.3 7.4 7.5
Networking troubleshooting..........................................................43 Packet loss in certain traffic flows unexpectedly high.................. 43 Description................................................................................... 43 Symptoms.................................................................................... 43 Recovery procedures................................................................... 43 Multihop BFD sessions are not established ................................ 46 Description................................................................................... 46 Symptoms.................................................................................... 46 Recovery procedures................................................................... 46 OSPF is not working properly...................................................... 48 Description................................................................................... 48 Symptoms.................................................................................... 48 Recovery procedures................................................................... 49 IP signaling link activation fails.....................................................52 The state of all subsystems in the remote network element is unavailable (UA)...........................................................................53
8 8.1 8.1.1 8.1.2 8.1.3 8.2 8.3
Hardware troubleshooting............................................................ 54 No traffic detected........................................................................ 54 Description................................................................................... 54 Symptoms.................................................................................... 54 Recovery procedure.....................................................................54 SFP ports added for USPUs and CSPUs do not show correctly their operational state nor alarms are raised................................55 Troubleshooting with mcJANE tool.............................................. 56
9 9.1 9.2 9.2.1 9.2.2
Performance management troubleshooting................................. 69 Threshold monitoring alarm is not sent to NetAct........................ 69 Transport and HW measurement management in mcRNC..........70 McRNC measurement management concepts............................ 70 McRNC measurement management commands......................... 71
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
List of Figures
Issue: 02B
Figure 1
Forcing change of the SPF post state................................................ 64
Figure 2
Example usage of the cp command................................................... 65
Figure 3
Checking the port monitoring..............................................................66
Figure 4
Removing port monitoring.................................................................. 67
DN0976768
5
Troubleshooting Multicontroller RNC
List of Tables
6
Table 1
NSN Case Type..................................................................................15
Table 2
NSN Problem Priority......................................................................... 16
Table 3
Supported plugins...............................................................................19
Table 4
Parameters related to save symptom-report command.............. 24
Table 5
Mapping between pip-mark and Queue ID......................................44
Table 6
Priority values for different DSCP codes............................................ 45
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
Summary of changes
Summary of changes Changes between document issues are cumulative. Therefore, the latest document issue contains all changes made to previous issues.
Changes between issues 02A(2014-05-23, RU40) and 02B(2015-02-23, RU40) New subchapter has been added. Troubleshooting software configuration management
Changes between issues 02(2013-09-15, RU40) and 02A(2014-05-23, RU40) OSPF is not working properly has been updated. The feature management name has been changed to OSPFForRedundancy.
Changes between issues 01D (2012-11-23, RU30) and 02(2013-09-15, RU40) Activation of a new software delivery fails •
Commands have been updated.
Installation of a new software delivery fails •
Commands have been updated.
Displaying all blackbox files •
Information on permissions required for executing the command in this section is added to Before you start section.
Collecting core dump file information •
Issue: 02B
Commands have been updated.
DN0976768
7
Overview of Multicontroller RNC Troubleshooting
Troubleshooting Multicontroller RNC
1 Overview of Multicontroller RNC Troubleshooting 1.1 Troubleshooting Multicontroller RNC recommendations If you have a contract with Nokia for the operation and maintenance of the network (or some other agreement), the actions you need to take in a fault situation may be different from the ones suggested in the troubleshooting instructions. If the general principles are in conflict with the operation and maintenance contract or any other contract, carry out actions as agreed in the contract. The operation and maintenance personnel that carry out troubleshooting should be familiar with the hardware and software of Nokia network elements.
Electrostatic precautions When handling plug-in units, it is important to use Electrostatic Precautions (ESP). This means that you must be earthed to equipment racks using an approved wrist strap and connecting lead. Approved ESP equipment makes a resistive connection to ensure the safety of the personnel and to prevent a sudden static discharge during connection to the earthing point.
Security procedures You are recommended to establish security procedures at your site to ensure appropriate staff and terminal access to the personnel.
Disaster recovery plan Establish a disaster recovery plan to help the personnel to deal with emergency situations. Remember that emergency situations can be best avoided by detecting abnormal conditions early. A disaster recovery plan should cover various disaster scenarios and disaster recovery procedures for personnel. The operation and maintenance personnel should also be able to contact the persons who are capable of dealing with the problem in question. Therefore, each site should have an escalation plan available with appropriate contact information.
Escalation plan An escalation plan offers contact lists of internal and external support personnel and services available to tackle problems. It should contain information on who to contact and in what kind of situations, for example, air conditioning, power back-up system and Nokia Emergency/Help Desk numbers.
Preventive maintenance Perform preventive maintenance routines on a regular basis. For example, carry out regular alarm and unit state surveillance.
8
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
Overview of Multicontroller RNC Troubleshooting
Safecopying Taking regular safecopies of the software and databases of the network element ensures that you have a functional copy of the software which you can use if there are problems with the software or hardware of the network element. How often you need to take safecopies depends on the size and type of the network element.
Performance monitoring The purpose of performance monitoring is to measure the overall quality of the system. Performance monitoring can help you to detect very low rate or intermittent problems and possible degradation of some part of the system. The performance monitoring parameters for network elements of different kinds and sizes vary.
Documentation Establish a procedure for keeping the documentation up-to-date and make sure that the operation and maintenance personnel have access to all relevant external and internal documents.
Network element diary It is recommended that you maintain a network element diary. The diary should be network element -specific, but you can store it in the Operation and Maintenance Centre if the network element is not usually manned. Start filling in the network element diary already when the network element is being set up and installed. You are recommended to record the following events in the network element diary: • • • • • •
Hardware changes Software and hardware updates (for example, change notes and correction deliveries) Essential modifications to the configuration or routing in the network element Safecopying Operational failures Any other relevant information
A network element diary can provide useful information on the system's performance in the past and hints on what might cause the current problems.
1.2 Information sources in fault situations Use at least the following information sources when carrying out troubleshooting.
Alarms Alarms are the primary source of information in most situations where troubleshooting is needed. Alarms are printed out on the alarm printer and/or other devices that you have specified for the network element.
Issue: 02B
DN0976768
9
Overview of Multicontroller RNC Troubleshooting
Troubleshooting Multicontroller RNC
Diagnosis reports Diagnosis reports contain data on the plug-in units that the system suspects to be faulty. Diagnosis reports are usually printed out on the report printer and/or other devices that you have specified for the network element.
Error messages General error messages of the system tell why the system cannot carry out a task. They can appear in the supplementary information fields of alarms and diagnostic printouts, in the printouts of the starting phases monitored through a service terminal, and in SCLI and Element Manager command outputs.
Statistical reports Different statistical reports contain useful data, for example, on traffic on speech and signalling circuits, use of services and load and availability measurements. Monitor and assess statistical data regularly as this data can indicate forthcoming problems before they affect the traffic. For more information, see performance management documentation.
Logs and other relevant statistical information Different logs (for example, computer and operating system logs and SCLI session reports) contain useful data that can be attached to the fault report when you need Nokia' help to solve a problem. Take the logs from the unit sending the alarm and the unit that is the object of the alarm, and if the alarm is I/O related, from the OMU.
Unit states Check also the unit states.
1.3 Problem types Here are some problem types you may encounter.
Reproducible problems You can reproduce the symptoms using a set of actions. Reproducible problems can be solved by narrowing down the possible causes of the problem to a single cause or to a number of causes and applying corrective actions. This requires knowledge of how the system works and tests to eliminate wrong conclusions.
Intermittent problems You cannot reproduce the symptoms consistently using any set of actions. However, an intermittent problem can reproduce itself randomly. In such a case, some kind of tracing or monitoring of the system may lead you to the origin of the trouble. If an intermittent problem occurs very seldom and it has no serious consequences, it may be best to just ignore the problem. You can also perform general maintenance and see if the problem disappears. If the problem occurs occasionally, try to conclude which factors seem to affect or contribute to the appearance of the problem.
10
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
Overview of Multicontroller RNC Troubleshooting
Several related or isolated troubles active at the same time Study whether the symptoms relate to each other or not and try to isolate the problems if possible.
1.4 Generic troubleshooting procedure Description Depending on whether you have a contract with Nokia on the operation and maintenance of the network (or some other agreement), the actions that you need to take in a fault situation may be different from the ones presented here. When you suspect that a Nokia network element is not performing as it should, carry out at least the following checks.
Symptoms A Nokia network element is not performing as it should.
Recovery procedures Troubleshooting process before calling Nokia Solutions and Networks help desk Steps 1
Evaluate how serious the consequences of the trouble are If the problem has very serious consequences, you may have to call for expert help or apply an emergency plan immediately.
2
Analyse the situation where the problem or failure first appeared Consider the following before you carry out any corrective actions. • • • • • •
3
Try to eliminate the possibility of human error •
Issue: 02B
What is the problem? Where is the problem? When did the problem occur? What were the circumstances that led to the problem? What is the impact of the problem (for example, to what extent does the fault affect the end customers)? Who is responsible for taking care of the problem?
For example, recent changes in the software or hardware configuration of the system (for example, circuits added or equipping changed) are possible sources of problems. The changes may have been carried out incorrectly. Check the configuration parameters, physical connections, strappings, plug-in units and so on, very carefully.
DN0976768
11
Overview of Multicontroller RNC Troubleshooting
show functional-unit unit-info Human error is a very common cause of problems – therefore, check and double-check every possible problem source. Type history. A failure can also occur spontaneously (for example, the remote end system may have problems or a service breakdown, or a plug-in unit may fail due to ageing). Check the alarms, clear codes, unit and link states and logs as described below.
•
•
4
Troubleshooting Multicontroller RNC
Make an accurate description of the symptoms You may not be able to solve the problem yourself. A detailed description of the situation where the symptoms occurred can help an expert solve the problem. Gather also data on the failure event. A symptom description should contain all the basic facts, such as: Date, name of the person who detected the trouble, phone number and e-mail address Details of the system; for example, what equipment and software is in use Description of the symptoms (alarms, error messages, clear codes, faulty states of the units and links and so on) Any other relevant information, for example, log and message monitoring files. All data may be valuable even if they seem irrelevant at the time. Store this information preferably in electronic format.
• • • •
5
Check and analyse the alarm situation Check the alarms that are currently on. You are recommended to also study the alarm history. Display the alarm history so that it shows all alarm events from the time period which starts one hour before the occurrence of the problem situation, and ends one hour after the problem situation was over. You should display the alarm history. # show alarm active The system may set an alarm and cancel it immediately. You can find these alarms in the alarm history. Alarms behaving in this way may indicate that some part of the system is about to break down or its functionality has been reduced. Check also the diagnostic reports.
•
•
6
Option
Description
If
the fault can be located based on the alarm situation
Then
Carry out the appropriate maintenance actions •
•
12
If you have ended up with more than one probable and possible cause for the trouble, change only one thing at a time – otherwise you cannot be sure of which change corrected the failure or problem. Remember that random actions can make problems worse. Generally, you should not take any radical corrective actions if you are not sure what the problem is and what the consequences of the corrective actions are. Losing traffic because of incorrect actions is not what you want.
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
7
Overview of Multicontroller RNC Troubleshooting
Option
Description
If
you cannot locate the fault based solely on the alarm situation
Then
Try to narrow down the possible problem source •
•
•
8
Analyse and categorise symptoms and list possible causes for the symptoms. Sometimes there can be several related or isolated troubles active at the same time. Study whether the symptoms relate to each other or not. Prioritise symptoms and collect further facts if needed. Based on tests and your knowledge of the system, eliminate symptoms that are not relevant to the trouble you are trying to solve. This way you can focus on symptoms and causes that are more likely to produce a solution to the problem. Examine what works and what does not. However, even though you may not be able to analyse what the cause of the trouble is, you can carry out general maintenance to eliminate some trivial causes of troubles, such as loose cables and bad connections.
Use measurements to trace any abnormal trends For more information, see performance management documentation.
9
Check the states of units, links and circuits Check unit states. show functional-unit unit-info
10 Fill in a problem report if needed Describe the problem in detail in the problem report. Include all relevant information that you have available from the problem situation and describe also the corrective measures that you have carried out after the problem occured.
Issue: 02B
DN0976768
13
Reporting problem to NSN
Troubleshooting Multicontroller RNC
2 Reporting problem to NSN Problem reports are used to communicate problems and failures to service personnel. Report only one fault in one problem report. Include the attachments in the compressed format. To make the investigation of a problem faster, include the following information in the problem report: • • •
a title that gives a brief description of the problem a clear and exact description of the problem itself problem background information: – – – –
– – •
symptom data reports Most of the Multicontroller RNC configuration data and logs essential for investigation of the problem can be collected with the standard symptom plugin report group-RNC and subreport-MessageMonitor.This data is always collected if Problem Report is submitted to NSN. To collect the Multicontroller RNC standard symptom data, execute the following SCLI commands: –
–
•
14
software and hardware releases situation in the beginning, for example, the first symptoms of the problem situation after the problem occured operations you made which possibly caused the failure, for example: hardware, software, parameter, feature or configuration changes, including opertions in transport network and third party equipment describe if the problem can be reproduced and what actions are required to reproduce this problem troubleshooting and recovery actions that you made
symptom data and message monitoring: save symptom-report name include group-RNC include subreport-MessageMonitor basic symptom data: save symptom-report name include group-RNC
For more instructions on the standard symptoms data collection framework, see Standard symptoms data collection. The data must be collected as soon as possible after an abnormal situation has taken place and before any recovery action is performed, such as Multicontroller RNC restarts or replacing hardware. This is important because the information stored about the problem (for example, blackbox of a certain unit) may get overwritten in the process of time or be lost because of recovery actions. The standard symptom report group-RNC is expected to be completed in around 15 minutes depending on Multicontroller RNC configuration. Recovery actions can be started, if needed, as soon as symptom report generation is completed. A copy of the symptom report can be transferred to local machine later at a convenient time. When you send out a problem report, make sure that all the possible attachments are included in the problem report, to avoid unnecessary information requests. For OMS symptom data collection, see Troubleshooting OMS document. If possible, include additional problem specific symptom data. For example, problem specific message monitoring logs, network analyzer interface traces.
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
• • • •
Reporting problem to NSN
NetAct KPI/counter report in case the problem concerns certain KPI/counter In a multivendor environment, include detailed information on the other third party products. NSN Case Type; Emergency, Trouble Resolution or Technical Query. For details, see #c105225791/table_bjv_g12_rp NSN Problem Priority: Major, Medium or Minor. For details, see #c105225791/table_ndz_pd2_rp.
Table 1
NSN Case Type
NSN Case Type
NSN Definition
Examples
Emergency
This case type is for Total and Partial outages.
Total Outage:
Available priority - Critical. This case type is for cases under Emergency Support service. Total Outage Total loss of voice and data traffic capability. An unscheduled event must be longer than 15 seconds.
•
All Iub links are down.
•
All WCELs are in incorrect state.
•
RNC spontaneous restart (unplanned).
•
RNC system restart (planned), all links are down.
Partial Outage: • •
Partial Outage Loss of greater than 10% of the provisioned capacity for origination and/or termination of voice and/or data traffic. Total loss of one or more critical services. An unscheduled event must be longer than 15 seconds.
• •
• •
Trouble Resolution
This case type is for problems • which caused by a suspected or an identified defect. Available problem priorities Major, Medium, and Minor.
•
•
Technical Query
Issue: 02B
This case type is for technical • questions regarding daily network operations and maintenance issues.
DN0976768
Loss of 10% or more of voice or data traffic capacity for at least 15 seconds. Spontaneous restarts of active computer unit(s) when there is no redundant unit available to take over the services provided by the restarted one. One or more network interfaces are down for over 15 seconds. Total loss of subscriber related RNC functionality (ISHO, HSDPA, HSUPA, Internet browsing, and so on) RNC operation and maintenance from NetAct is totally lost. Emergency calls not possible (for example, 112 or 991). SW defect is identified or suspected. A correction will be required if the defect is confirmed. HW design defect is suspected. A correction (HW retrofit or SW change) will be required if the defect is confirmed. An error or omission in the product technical literature is suspected. A documentation correction will be required if the defect is confirmed. Technical questions on procedures or features that are covered in the documentation shipped with the product.
15
Reporting problem to NSN
Table 1
Troubleshooting Multicontroller RNC
NSN Case Type (Cont.)
NSN Case Type
Table 2
NSN Definition
Examples
Available problem priorities – Major and Medium.
•
Information requests on NSN product that will be used to help interface this product with a third party product.
NSN Problem Priority
NSN Problem Priority
NSN Definition
Examples
Critical
Problems under Emergency Support service.
N/A
Major
Only Total or Partial outages which are not avoidable with a workaround solution.
N/A
Medium
Loss of less than 10% of the provisioned capacity for origination and/or termination of voice and/or data traffic.
•
Total or Partial outages avoidable • • with a workaround solution. Partial loss of one or more critical • services. •
•
•
• •
•
Minor
Minor fault not affecting operation • or service quality • •
16
DN0976768
total or more than 10% loss of voice or data traffic capacity for at least 15 seconds, avoidable with workaround single restart of computer units configuration changes (RNW, HW and SW) are not working activation of a feature fails single performance measurement is not working completely partial loss of subscriber related RNC functionality (ISHO, HSDPA, HSUPA, Internet browsing, and so on) partial loss of alarm management of objects (BTS, functional units) problems with back-up major errors in documentation, for example, an alarm or parameter description is missing from documentation vital documents are missing from the documentation library failures not seriously affecting traffic errors in command line syntax or output minor errors in documentation
Issue: 02B
Troubleshooting Multicontroller RNC
Reporting problem to NSN
Most of the Multicontroller RNC configuration data and logs essential for problem investigation can be collected with the standard symptom plugin report group-RNC and subreport-MessageMonitor.This data is always collected if Problem Report is submitted to Nokia Solutions and Networks. To collect the Multicontroller RNC standard symptom data, execute the following SCLI commands: •
•
symptom data and message monitoring: save symptom-report name include group-RNC include subreport-MessageMonitor basic symptom data: save symptom-report name include group-RNC
For more instructions on the standard symptoms data collection framework, see Standard symptoms data collection. The data must be collected as soon as possible after an abnormal situation has taken place and before any recovery action is performed such as Multicontroller RNC restarts or replacing hardware. This is important because the information stored about the problem may get overwritten in the process of time or lost due to recovery actions. To save or collect the basic symptom report group-RNC, enter the following command: save symptom-report name include group-RNC include subreport-MessageMonitor _nokadmin@CFPU-0 [RNC-37] > save symptom-report group-RNC include subreport-MessageMonitor CFPU-0@RNC-37 Mode : Normal mode Max. execution time limit : 30 minutes Max. size of each part : 100 MB Max. size of full report : 350 MB ReportName FileName Store ---------------------walle20141016 /stdsymp/walle20141016_0.tar CFPU-0 walle20141016 /stdsymp/walle20141016_0.tar MessageMonitor CFPU-0 walle20141016 /stdsymp/walle20141016_0.tar CFPU-0 walle20141016 /stdsymp/walle20141016_0.tar CFPU-0 walle20141016 /stdsymp/walle20141016_0.tar CFPU-0 walle20141016 /stdsymp/walle20141016_0.tar CFPU-0 walle20141016 /stdsymp/walle20141016_0.tar CFPU-0 walle20141016 /stdsymp/walle20141016_0.tar CFPU-0 walle20141016 /stdsymp/walle20141016_0.tar CFPU-0 walle20141016 /stdsymp/walle20141016_0.tar CFPU-0
Issue: 02B
DN0976768
name walle20141016 include [2014-10-16 13:48:57 +0200]
Chunk
Subreport
------
--------
Part 1
ipmgmt
Part 1 Part 1
rncipconfig
Part 1
rncrnw
Part 1
rncsignaling
Part 1
rnchw
Part 1
rncinfo
Part 1
rncmon
Part 1
rncuplane
Part 1
rnchas
17
Reporting problem to NSN
walle20141016 CFPU-0
Troubleshooting Multicontroller RNC
/stdsymp/walle20141016_0.tar
Part 1
rncalarm
Successfully collected symptom report data for 11 out of 11 subreports
The standard symptom report group-RNC is expected to be completed in around 15 minutes depending on Multicontroller RNC configuration.
g
When using the "to" and "from" options with the standard symptom report group-RNC, the syslog, syslog.1 files are not filtered according to the time range provided. The time range is not applicable to syslog and syslog.1 files. Recovery actions can be started, if needed, as soon as symptom report generation is completed. Copy of the symptom report can be transferred to local machine later at a convenient time. When you send out a problem report, make sure that all the possible attachments are included in the problem report, to avoid unnecessary information requests.
18
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
Symptoms data collection
3 Symptoms data collection 3.1 Standard symptoms report Standard symptom report is a framework used to collect symptom data from Multicontroller RNC to support troubleshooting of problems. The framework collects standard symptoms data by running individual or multiple plugins. Some of the plugins are put into groups, thus it is possible to collect the symptom data on the group level too. The framework allows easy enhancement with additional plugins, if the available standard plugins are not sufficient. The collected symptoms data is stored in /mnt/backup/stdsymp directory. The following table provides details about standard plugins and their functionality: Table 3
Supported plugins
Plugin Name
Description
group-RNC
It collects Multicontroller RNC configuration data and logs essential for investigation of the problem.
w
NOTICE: This data must always be collected if Problem Report is submitted to Nokia.
This is the report group containing the following sub-reports: subreport-rnchw subreport-rncinfo subreportrncipconfig subreport-rncsignaling subreportrncrnw subreport-rnchas subreport-rncalarm subreport-rncuplanesubreport-rncmon groupBackupandrestore
It collects the following backup and restore-related logs:
group-Networking
It collects the following general statistics of the network:
subreport-clusterinfo subreport-syslog subreport-corefiles subreport-processinfo subreport-clusterstatus
subreport-clusterinfo subreport-osinfo subreport-processinfo subreport-clusterstatus subreport-syslog subreport-corefiles subreport-alarmsinfo subreport-debug subreport-routing subreport-ipmgmt groupSignalingProtocol
It collects the following reports related to the signaling protocols and the signaling network manager: subreport-signalingSS7 subreport-signalingSCCP subreport-signalingNetManager subreportsignalingPacketCaptureSctp subreportsignalingPacketCaptureSccpUser subreportsyslog subreport-corefiles subreportalarmsinfo subreport-debug
Issue: 02B
DN0976768
19
Symptoms data collection
Table 3
Troubleshooting Multicontroller RNC
Supported plugins (Cont.)
Plugin Name
Description
group-IpalPlatform
This is the report group containing the following sub-reports: subreport-IpalSignalling subreportIpalCallManagement subreport-IpalCommon subreport-IpalTransportsubreportIpalBasicServices
subreport-alarmsinfo
It collects alarm data for a selected period of time.
subreport-clusterinfo
It collects cluster and node information.
subreportclusterstatus
It collects information about the status of cluster, nodes, recovery groups or units.
subreport-corefiles
It collects core data available within the /crash directory of the active node.
subreportdatabaseinfo
It collects the database information such as Database Deployed, Disk Space, Postgres Configuration files, Process lists, and Enterprise Database details.
subreport-debug
It collects debug data collected from master debug log.
subreport-fastpath
It collects information from nodes which have 6windip stack. For example, ngctl.dat, fpdebug, fpstat.
subreport-ipmgmt
It collects the IP networking configuration data.
subreport-licinfo
It collects license management related information.
subreport-licdebug
It collects license management related information.
subreport-ipsec
It collects information about IKE template, IP sec rules, VPNs, and configuration files.
subreportMessageMonitor
It collects two message logs for the following traffic: •
call scenarios
•
operation and maintenace (O&M) scenarios
Message monitoring parameters are stored in the msgmon.ini file under /opt/nsn/SS_SysReport/stdsymp/plugins/ path. By default, the buffers size is set to 8 MB (0x8FFFFF), and duration of the monitoring is 30 seconds. The message monitoring is performed in two runs, the first is for call scenarios, and the second is for O&M scenarios. To modify the settings that are used to capture message logs with subreport-MessageMonitor, proceed as follows:
20
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
Table 3
Symptoms data collection
Supported plugins (Cont.)
Plugin Name
Description
1. Switch to root user and bash shell (type exit later on to return to fsclish). set user username root Provide the password for root (default is root). 2. Copy the msgmon.ini file to /mnt/backup/share and modify it. cp /opt/nsn/SS_SysReport/ stdsymp/plugins/msgmon.ini /mnt/backup/ share/msgmon.ini To restore default settings, delete the msgmon.ini file from /mnt/backup/share directory. subreport-osinfo
It collects information about operating system version, cluster uptime, disk usage, shared memory, build label, image variants, current snapshot name, and complete RPM information.
subreport-processinfo
It collects process-related information.
subreport-routing
It collects the current routing information, IP address, and routing instances.
subreportsignalingNetManager
It collects the ring-based buffer trace files created for netmanager in the following location: /trace
Issue: 02B
subreportsignalingPacketCaptur eSccpUser
It collects the tcp dump pcap files created for capturing the flow of packets between sccp and sccp-user.
subreportsignalingPacketCaptur eSctp
It collects the tcp dump pcap files created for capturing the flow of packets at the SCTP level.
subreportsignalingSCCP
It collects the trace files for SCCP subsystem, ldap configuration, and output of SCLI show command at the starting and stopping time of signaling diagnostics.
subreportsignalingSS7
It collects the trace files for SS7 subsystem, ldap configuration, and output of SCLI show command at the starting and stopping time of signaling diagnostics.
subreport-syslog
It collects relevant data from syslog.
subreport-techservice
It collects the output of some software commands like fsswcli, uname and so on, and also some scripts required by tech service people.
DN0976768
21
Symptoms data collection
Table 3
Troubleshooting Multicontroller RNC
Supported plugins (Cont.)
Plugin Name
Description
subreport-tracemgmt
It collects tracing-related information useful for debugging purposes such as: •
build version
•
tracing-related rpm versions
•
configuration files
•
LDAP fragments under tracing
•
features related to tracing LDAP fragment
•
contents of fptl_admin shared library
•
list of trace files
•
list of plugin libraries
• •
process states of NodeTraceManager (NTM) and ClusterTraceManager (CTM) list of buffers/admin buffers
•
disk usage of tracing-related filesystems
•
consistency of tracing configuration
subreporttracesnapshot
It collects the snapshot for default_platform buffer in the CLA0 node.
subreportIpalBasicServices
It collects data on distributed computing services, such as functional unit states, name services information, and feature management data.
subreportIpalCallManagement
It collects information about call management and user plane management, such as call details, connection information, and user plane service allocation.
subreport-IpalCommon
It collects general information for troubleshooting. This includes software build version, hardware configuration information, recovery units/recovery groups/node configuration, and basic counters.
subreportIpalSignalling
It collects information about signaling connections, including all NBAP/SCTP link information and all SIGTRAN link information.
subreportIpalTransport
It collects information about traffic transport that is handled by EIPU, such as transport forwarding table, GTP tunnel ID information, IPBR configuration.
subreport-rnchw
It collects information on Multicontroller RNC functional units.
subreport-rncinfo
It collects information on the software builds, disk space usage, available snapshots, recovery groups states, PRFILE parameters, and so on. This includes also relevant information from the syslog.
22
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
Table 3
Symptoms data collection
Supported plugins (Cont.)
Plugin Name
Description
subreport-rncipconfig
It collects information on the IP-related configuration, that is, IP interfaces, IP addresses, routing, user plane resources (ipbr and ipro objects), and so on.
subreportrncsignaling
It collects information on SIGTRAN-related configuration, that is, status of local and remote subsystems, local application server(s), SCTP associations, SCTP profiles, M3UA limits, SSP filter timers, and so on.
subreport-rncrnw
It collects information on the RNW status, that is BTSOM connections status, NBAP connection status, WBTS and WCEL objects status, IPNB object status, IUCS and IUCSIP objects statuses IUPS and IUPSIP object statuses.
subreport-rnchas
It collects information on the node, recovery groups, and recovery units. This includes also HAS related information from the syslog.
subreport-rncalarm
It collects information about active alarms, alarms history, and so on.
subreport-rncuplane
It collects information on the user plane configuration and status.
subreport-rncmon
It collects information oncontrol plane UE specific monitoring for abnormal calls as part of HPL logging functionality.
3.2 Collecting standard symptom reports Purpose Follow this procedure to save (collect) the standard symptom reports.
Steps 1
Save or collect the standard symptom reports. To save (collect) the standard symptom report, enter the following command: save symptom-report name
The full syntax of the command is: save symptom-report name {[exclude [include ] [quick-mode ] [timeout ] [report-max-size ] [single-file-max-size ]} from date-time to date-time
Issue: 02B
DN0976768
23
Symptoms data collection
Troubleshooting Multicontroller RNC
w
NOTICE: It is recommended to use normal-mode for the symptoms data collection for group-RNC. It is because, for group-RNC, the data collection in normal-mode completes in the time allocated for quick-mode. Simultaneous multiple sessions of symptom reports data collection with the same report name is not allowed. If a report file already exists, collecting standard symptom report cannot be initiated with the same report name. When using the "from date-time" and "to date-time" options with the standard symptom report group-RNC, the syslog and syslog.1 files are not filtered according to the time range provided. The time range is not applicable to syslog and syslog.1 files. Symptom reports that are older than 10 days are removed from the file system automatically. The parameters in the curly brackets are not allowed in the command syntax after the "from date-time" and "to date-time" parameters. When retrieving symptom reports from the previous year, the value of the from date-time and to date-time parameters do not go back further than 333 days from the current date. The following messages are displayed according to the triggering scenarios encountered during command execution: •
Use case 1: The date specified in the from date-time parameter is from the previous year. Message displayed: From/To dates cannot be older than 333 days from current date Date has to be given in specified format, refer usage
•
Use case 2: The date specified in the from date-time parameter is from the previous year while the date specified in the to date-time parameter is in the current year. Message displayed: From/To dates cannot be older than 333 days from current date Date has to be given in specified format, refer usage
If the user only specified the from date-time parameter, or if the user only specified the to date-time parameter, then the one that was not specified is calculated by a default offset of 10 days. For the description of the parameters of the save symptom-report command, see Table 4: Parameters related to save symptom-report command. Table 4
Parameters related to save symptom-report command
Parameter
Description
name
This mandatory parameter creates a single or multiple (it depends on subsequent options) tar file(s) with the provided reportname. The report(s) are stored under the directory /mnt/backup/stdsymp of the Multicontroller RNC. The report file(s) generated contain subreport(s) that are compressed archive file(s) (with .tar.gz extension) containing one or more file(s).
24
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
Table 4
Symptoms data collection
Parameters related to save symptom-report command (Cont.)
Parameter
Description In the report file(s), there is also one text file that summarizes executed commands, their execution status and time. Its format is: __summary.txt Special characters for report-name are not allowed. Numbers are not allowed as first character. The report name length must not exceed 25 characters.
[exclude ]
If this optional parameter is used, the standard symptom report of the selected plugin and group are not collected.
[include ]
If this optional parameter is used, the standard symptom report can be collected from a specified plugin or group.
[quick-mode ]
This optional parameter is used in emergencies to gather important and relevant information rather than having all information available. When quick mode is enabled, the symptom subreports automatically include data that is critical and quickly retrieved. If "yes" is selected, each subreport includes only data that is quick to collect. If "no" is selected, each subreport includes all data that is needed in a support case. The default value for this option is "no".
[single-file-max-size ]
This optional parameter creates the final report in chunks or pieces of tar file, each with a maximum size specified as the argument. Each chunk is a tar file which contains reports that are independently analyzable. Report produced by single subreport is not split even if the report size exceeds the specified maximum size. This implies that size of a single file may sometimes be bigger than the specified maximum single file size. The default value for this option is 10 MB. A size of 0 means no limit.
Issue: 02B
[report-max-size ]
This optional parameter accepts the maximum allowed size limit (in MB) for the data generated. When the size of the data generated reaches the specified limit, the framework stops the data collection after the
DN0976768
25
Symptoms data collection
Troubleshooting Multicontroller RNC
Table 4
Parameters related to save symptom-report command (Cont.)
Parameter
Description currently executing subreport completes its execution. Since the data generation is not abruptly stopped upon reaching the size limit, the resulting size of the symptom data collected may still exceed the specified limit. The default value for report-max-size is 30 MB. A size of 0 means no limit.
[timeout ]
This optional parameter specifies the timeout value for symptom data collection. When timeout occurs, the plugin that is currently being processed fails to be collected, and all the queued plugins (if any) are skipped. Timeout option is strictly followed except when plugins go into kernel mode. In kernel mode, delays occur because the signals are not processed until it logs out of the kernel mode. Permitted range is 0-60. A value of 0 means there is no timeout. Timeout option allows you to stop the symptom data collection after the specified timeout value expires. The default timeout is 30 minutes.
from date-time
This optional parameter saves symptom report up to a particular date. Accepted date formats are YYYY.MM.DD-HH:MM:SS or DD.MM.YYYY-HH:MM:SS. When trying to collect symptom reports from the previous year, make sure the parameter does not go back further than 333 days from the current date.
to date-time This optional parameter specifies the date till which the standard symptom report must be collected. The accepted date formats are YYYY.MM.DD-HH:MM:SS or DD.MM.YYYYHH:MM:SS. When trying to collect symptom reports from the previous year, make sure the parameter does not go back further than 333 days from the current date. Examples a) To collect the basic symptom report with a specific name (RNC311), and with a subreport group (group-RNC), execute the following command: save symptom-report name RNC311 include group-RNC
26
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
Symptoms data collection
b) To collect the standard symptom report with a specific name (RNC311), with a subreport group (group-RNC), and with a message monitoring (subreportMessageMonitor), execute the following command: save symptom-report name RNC311 include group-RNC include subreport-MessageMonitor
c) To collect the standard symptom report with a specific name (RNC311), with a subreport group (group-RNC), and from the given date (2013.08.0509:00:00), execute the following command: save symptom-report name RNC311 include group-RNC from date-time 2013.08.05-09:00:00
d) To collect the standard symptom report with a specific name (RNC311), with a subreport group (group-RNC), from the given date (2013.08.05-09:00:00) to a particular date (2013.08.06-09:00:00), execute the following command: save symptom-report name RNC311 include group-RNC from date-time 2013.08.05-09:00:00 to date-time 2013.08.06-09:00:00
e) To collect the standard symptom report with a specific name (RNC311), with a subreport group (group-RNC), from the time the Multicontroller RNC was commissioned to a particular date (2013.08.06-09:00:00), execute the following command: save symptom-report name RNC311 include group-RNC to date-time 2013.08.06-09:00:00
f)
To collect all the standard symptom reports with a specific name (RNC311), execute the following command: save symptom-report name RNC311
g) To collect all the standard symptom reports with a specific name (RNC311ipconifg), and with the specific subreport (rncipconifg), execute the following command: save symptom-report name RNC311ipconfig include subreportrncipconfig
h) To collect the standard symptom report with a specific name (RNC311), with a subreport group (group-RNC), and with the limited size of a single report file (5 MB), execute the following command: save symptom-report name RNC311 include group-RNC single-file-maxsize 5
i)
To collect the standard symptom report with a specific name (RNC311), with a subreport group (group-RNC), and with the maximum size of the report (10 MB), execute the following command: save symptom-report name RNC311 include group-RNC report-max-size 10
j)
To collect the standard symptom report with a specific name (RNC311), with a subreport group (group-RNC), and with timeout for data collection (10 minutes), execute the following command: save symptom-report name RNC311 include group-RNC timeout 10
k) To collect the standard symptom report with a specific name (RNC311), with a subreport group (group-RNC), and with the single subreport excluded from the group (subreport-rnchw), execute the following command: save symptom-report name RNC311 include group-RNC exclude subreport-rnchw
Expected outcome: The files are stored as in directory /mnt/backup/stdsymp, with the name used in the save symptom-report command.
Issue: 02B
DN0976768
27
Symptoms data collection
Troubleshooting Multicontroller RNC
3.3 Listing the standard symptom reports Purpose Follow this procedure to list the standard symptom report of files, plugins or groups available on the Multicontroller RNC. Before you start Default system username and password are "_nokadmin" / "system".
Steps 1
List the symptom reports. To list all the standard symptom reports, enter the following command: show symptom-report all Sample Output CFPU-0@RNC-311 07:52:43 +0200] ReportName Store -------------AllRNC-311 CFPU-0 AllRNC-311 CFPU-0 MessageMonitor CFPU-0 RNC-311 CFPU-0 rncmon CFPU-0 RoutingFails CFPU-0 history CFPU-0
w
[2013-08-06 FileName
Chunk/Total
--------
--------
/stdsymp/AllRNC-311_0.tar
part 1/2
/stdsymp/AllRNC-311_1.tar
part 2/2
/stdsymp/MessageMonitor_0.tar
part 1/1
/stdsymp/RNC-311_0.tar
part 1/1
/stdsymp/rncmon_0.tar
part 1/1
/stdsymp/RoutingFails_0.tar
part 1/1
/stdsymp/history_0.tar
part 1/1
NOTICE: The Chunk/Total field displays the number of chunks present for the particular report. This assists the user in learning whether the files of the report have been listed properly.
2
List the detailed contents of a particular symptom report. show symptom-report name Example To list the particular standard symptom report (RNC-311), enter the following command: show symptom-report name RNC-311
28
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
Symptoms data collection
Sample Output CFPU-0@RNC-311 +0200] ReportName Store -------------RNC-311 CFPU-0 RNC-311 CFPU-0 RNC-311 CFPU-0 RNC-311 CFPU-0 RNC-311 CFPU-0 RNC-311 CFPU-0 RNC-311 CFPU-0 RNC-311 CFPU-0
3
[2013-08-06 07:57:14 FileName
Chunk/Total
Subreport
---------
---------
-----
/stdsymp/RNC-311_0.tar
part 1/1
rnchw
/stdsymp/RNC-311_0.tar
part 1/1
rncinfo
/stdsymp/RNC-311_0.tar
part 1/1
rncipconfig
/stdsymp/RNC-311_0.tar
part 1/1
rncsignaling
/stdsymp/RNC-311_0.tar
part 1/1
rncrnw
/stdsymp/RNC-311_0.tar
part 1/1
rnchas
/stdsymp/RNC-311_0.tar
part 1/1
rncalarm
/stdsymp/RNC-311_0.tar
part 1/1
rncuplane
Display the list of available plugins. Example To display the list of available plugins, enter the following command: show symptom-report plugin-list
4
List the group of plugins based on the group names. Example To list the group of plugins based on the group names (group-RNC), enter the following command: show symptom-report group group-RNC
3.4 Copying standard symptoms reports to remote machine Purpose Follow this procedure to copy standard symptoms reports from Multicontroller RNC local storage to remote machine. Before you start Default system username and password are "_nokadmin" / "system". Expected outcome
Issue: 02B
DN0976768
29
Symptoms data collection
Troubleshooting Multicontroller RNC
The standard symptoms reports are successfully copied to external storage.
Steps 1
Switch to bash shell (type exit later on to return to fsclish). shell bash full Confirm the command by pressing "y".
2
Use SCP to copy standard symptoms report to remote machine. Note that the report files are stored in /mnt/backup/stdsymp directory. scp /mnt/backup/stdsymp/.tar @:/
Note that you can use wildcard symbol "*" in the to copy all the report files from /mnt/backup/stdsymp directory. If the same remote machine is used to store the reports from different Multicontroller RNCs, it is recommended to include specific Multicontroller RNC name and identifier in the folder name to differentiate the reports.
w
NOTICE: The report files can also be copied directly from the Multicontroller RNC to external machine using SFTP or SCP protocol.There is a default account _nokadmin which provides the access to Multicontroller RNC file system. The default password for _nokadmin user is "system”. The reports are stored under /stdsymp directory.
3.5 Deleting the standard symptom reports Purpose Follow this procedure to delete the standard symptom reports. Before you start Default system username and password are "_nokadmin" / "system".
Steps 1
Delete the particular standard symptom report. To delete a particular standard symptom report, enter the following command: delete symptom-report name Example To delete a particular standard symptom report with name RNC311, enter the following command: delete symptom-report name RNC311 Sample Output: CFPU-0@RNC-311 08:29:10 +0200] ReportName
30
[2013-08-06 FileName
DN0976768
Store
Issue: 02B
Troubleshooting Multicontroller RNC
Symptoms data collection
---------RNC311
2
-------/stdsymp/RNC311_0.tar
----CFPU-0
Delete all the standard symptoms reports. To delete all the standard symptom reports, execute the following command: delete symptom-report all Sample Output: CFPU-0@RNC-311 08:32:00 +0200] ReportName ---------AllRNC311 AllRNC311 MessageMonitor 0 RoutingFails 0 history 0 rncmon 0
Issue: 02B
[2013-08-06 FileName Store -----------/stdsymp/AllRNC311_0.tar CFPU-0 /stdsymp/AllRNC311_1.tar CFPU-0 /stdsymp/MessageMonitor_0.tar CFPU-
DN0976768
/stdsymp/RoutingFails_0.tar
CFPU-
/stdsymp/history_0.tar
CFPU-
/stdsymp/rncmon_0.tar
CFPU-
31
Generic software troubleshooting instructions
Troubleshooting Multicontroller RNC
4 Generic software troubleshooting instructions 4.1 Displaying all blackbox files Purpose This section provides instructions on how to display all the blackbox files in the system. These blackbox files are normally located in /srv/Log/crash folder. Before you start Make sure you have the authority to the secondary group_nokfsuicrashlog and the permission fsASView. To avoid adding the user account to a long list of groups, you are recommended to add the permissionfsASView to group _nokfsuicrashlog using the command add user-management group-to-permission gid _nokfsuicrashlog permid fsASView and then assign the group _nokfsuicrashlog to the target user account.
1
Show all blackbox file groups for all crashed processes. Enter the following command: show troubleshooting blackbox list
Expected outcome All the blackbox names are listed. Each name indicates there is a process crash. An example of the output is as follows. Apr 10 16:34 CLA-1-11945-534657b0-snmpmdserver-ABRT Apr 21 09:27 CLA-0-11435-53547401-snmpmdserver-ABRT
Steps 1
Show all blackbox file groups for all crashed processes. Enter the following command: show troubleshooting blackbox list
Expected outcome All the blackbox names are listed. Each name indicates there is a process crash. An example of the output is as follows. Jun Jun Jun Jul Jul
32
23 23 24 18 18
14:34 15:30 14:24 13:15 13:15
CFPU-0-21089-4e02db77-sokeri-ABRT CFPU-0-21441-4e02eb8e-ilalarm-ABRT CFPU-0-7052-4e042aaa-slapd-ABRT CFPU-0-7905-4e23c149-lastproc-ABRT CFPU-0-7871-4e23c188-starter-ABRT
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
Generic software troubleshooting instructions
4.2 Viewing active alarms and alarms history The alarm system indicates potential faults in the system as well as faults that require corrective actions. After an alarm is raised, the fault causing the alarm must be solved. The solution can be an automatic recovery or a manual corrective action. Alarms are typically used in situations where it is possible to give instructions for corrective actions in the alarm description, such as replacing a hardware unit. The alarms can also be raised through the alarm system to indicate that the system is not working normally, for example, when the hard disk is full and the system cannot write to it. Such alarms are cleared automatically when the system returns to its normal state. You cannot clear these alarms manually.
Multicontroller RNC alarm handling •
Multicontroller RNC OMS provides Fault Management application that enables user to perform operations such as: – –
managing Multicontroller RNC, Multicontroller RNC OMS, and Flexi BTS alarms viewing Multicontroller RNC, Multicontroller RNC OMS, and Flexi BTS alarms history
For information on how to check and manage the alarms, see Managing Faults with OMS.
g
To check the active alarms and alarms history directly from Multicontroller RNC command line interface, enter the following commands: # zahoUsage: zaho [ -h | --help | -a | --alarms | -p | --alarmhistory ] [ -x | --alarmx | -s | --summary ] Obligatory parameters: -h | --help : this help -a | -alarms : active alarms -x | --alarmx : shows alarms with certain alarm number -s | --summary : summary of the alarmsExample: zaho -a : prints all active alarms zaho -ax 12345 : prints all 12345 alarms from active alarms zaho -ps : prints summary about alarm history •
For information on the Multicontroller RNC specific alarms, check the following reference documentation: Multicontroller RNC Notices (0-999) Multicontroller RNC Disturbances (1000-1999) Multicontroller RNC Failure Printouts (2000-3999) Multicontroller RNC, IPA-RNC and I-HSPA Adapter Base Station Alarms (7000-7900)
Flexi BTS alarm handling •
•
Issue: 02B
Flexi BTS alarms are managed using OMS Fault Management application. As an alternative, they can be also checked by using BTS Site Manager. For more information, seeChecking BTS alarms ofTroubleshooting Flexi Multiradio BTS WCDMA. For information on the Flexi BTS specific alarms, see Flexi Multiradio BTS WCDMA Faults.
DN0976768
33
Generic software troubleshooting instructions
Troubleshooting Multicontroller RNC
Multicontroller RNC OMS alarm handling • •
Multicontroller RNC OMS alarms are managed using OMS Fault Management application. For information on the specific Multicontroller RNC OMS alarms, see: OMS Alarms.
4.3 Collecting core dump file information Purpose When an application crashes, it writes the contents of its execution environment (stacks, variables and so on) into a file. This file is known as a core dump. A core dump is useful for problem solving, since it provides the following details: • •
The application's task during failure The application's execution environment
In addition to the core dump, the system also gathers information such as recent syslog entries and information about the node. This information is also saved to files and labeled the same way as the core files. Before you start Default system username and password are "_nokadmin" / "system".
Steps 1
Open an SSH connection to Multicontroller RNC IP address.
2
List all the blackbox files in the system. show troubleshooting blackbox list These blackbox files are normally located in /srv/Log/crash folder.
3
Switch to root user and the bash shell. set user username root Provide the password for root user (default is root).
4
Collect the log and the core files. To check the files that can be accessed from the node, where the log recovery group is running, enter the following command: ls /srv/Log/crash The filenames will be in the following format: ---..gz Where: •
34
node is the name of node where the crash occurred.
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
• • • •
Generic software troubleshooting instructions
pid is the process id of the crashed process. timestamp is the time of crash in seconds since 1970-01-01 00:00:00 UTC in hexadecimal notation. name is the name of the crashed process. signal is the signal that caused the process to dump the core.
type can be one of the following: • • • •
5
core - the core dump file syslog - a snapshot of syslog entries from the time of crash debug - a snapshot of debug entries from the time of crash blackbox.tar - information about the node
Copy the files from the Multicontroller RNC to external machine. scp /srv/Log/crash/ @://
6
Issue: 02B
Send the files to your Nokia Solutions and Networks representative.
DN0976768
35
Configuration management troubleshooting
Troubleshooting Multicontroller RNC
5 Configuration management troubleshooting 5.1 Saving configuration snapshot fails 5.1.1 Description If there is no free space on the disk, saving the configuration snapshot is not possible.
5.1.2 Symptoms The save snapshot command fails with an error message.
5.1.3 Recovery procedures If saving configuration snapshot fails, follow either of these steps:
1
Check the existing configuration snapshots and delete the old configuration snapshots that are not needed. To display the existing configuration snapshots, enter the following command: show snapshot list To delete all configuration snapshots that are no longer needed, enter the following command: delete snapshot config-name
2
Check the existing software volumes and delete the old software deliveries that are not needed. To display the existing software volumes, enter the following command: show sw-manage list To delete the old software deliveries that are no longer needed, enter the following command: delete sw-manage delivery
36
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
Software management troubleshooting
6 Software management troubleshooting 6.1 Installation of a new software delivery fails 6.1.1 Description The installation of a new software delivery fails if the software image is corrupted or already installed.
6.1.2 Symptoms It is not possible to install a new software delivery.
6.1.3 Recovery procedures Default system username and password are "_nokadmin" / "system".
1
Check if the software image is corrupted. If the software image is corrupted, copy it again from the provided transfer medium.
w
NOTICE: The integrity of the software delivery image can be verified by using the md5sum tool.
2
Check if the image to be installed is already existing. If the image to be installed already existing, then the new software image cannot be installed on the system.
3
Check the log files related to software management. a) Switch to bash shell (type exit later on to return to fsclish). Confirm the command by pressing "y". b) Log files are stored to the following locations: /var/log/ilsw_upgrade.log /var/log/swm.log
6.2 Activation of a new software delivery fails 6.2.1 Description In the mcRNC hardware, when activating a new software delivery, there is a fallback mechanism to activate the previous delivery if the new delivery does not boot up. The system switches to the previous delivery after five failed boot attempts.
Issue: 02B
DN0976768
37
Software management troubleshooting
Troubleshooting Multicontroller RNC
6.2.2 Symptoms After activation of a new software delivery, the system takes a long time (about five times longer than normally) to boot up. When the system is operational again, the previous delivery is activated instead of the new software delivery.
6.2.3 Recovery procedures Default system username and password are "_nokadmin" / "system".
1
Contact Nokia Solutions and Networks representative and attach the following log files to carry out the recovery actions. a) Switch to bash shell (type exit later on to return to fsclish). Confirm the command by pressing "y". b) Log files are stored to the following locations: /var/log/ilsw_upgrade.log /var/log/swm.log /var/log/fsconfigure.log
6.3 Troubleshooting software configuration management 6.3.1 Executing pre-download script fails Symptoms The execution of Download task on NetAct interface fails. The following error logs (as an example) are found in the ilsw_upgrade.log file (that locates in /var/log folder on mcRNC): 2011-09-14 17:56:58 ILSWMAN: Execute pre_downloading_02_fail.sh error! Status: 1 2011-09-14 17:56:58 SOKERI: Download pre-checking error! Status: 255, Additional Status: 25
Reason analysis Before downloading the software delivery, the system executes the pre_downloading_xx.sh script to do pre-download inspection if such script exists. An error in the execution of the script causes the failure of the Download task.
Recovery procedures Steps
1
38
Make a copy of syslog file and ilsw_upgrade.log file (both under /var/log directory) respectively.
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
2
Software management troubleshooting
Send the copied files to Nokia Customer Service Center.
6.3.2 Parsing targetBD.xml fails Symptoms The execution of Download task on NetAct fails. The following error logs (as an example) are found in the ilsw_upgrade.log file (that locates in /var/log folder on mcRNC): 2011-09-14 16:35:58 SOKERI: Parse target BD SWPackages/default/TargetBD.xml start. 2011-09-14 16:35:59 SOKERI: Parse target BD error! Status: 17466, Additional Status: 18
Reason analysis When the operator starts a Download task on NetAct, mcRNC downloads targetBD.xml file before downloading the actual software delivery. The reason for the failed parsing of the targetBD.xml file can be: The format of the targetBD.xml file is incorrect. Key information is missing in the file.
• •
g
In this example, the target BD file is named as targetBD.xml. This name could be different in the real environment.
Recovery procedures 1
Make a copy of the needed files The files that need to be copied are: • • •
2
targetBD.xml located in /var/opt/nsn/SS_ILOMU/SS_ILSWMAN/build/ folder syslog located in /var/log folder ilsw_upgrade.log located in /var/log folder
Send the copied files to Nokia Customer Service Center.
6.3.3 Executing pre-activation script fails Symptoms The execution of Download and Activate or Activate task on NetAct fails. The following error logs (as an example) are found in the ilsw_upgrade.log file (that locates in /var/log folder on mcRNC): 2011-09-16 09:35:56 ILSWMAN: Execute pre_activation_02.sh start 2011-09-16 09:35:56 ILSWMAN: Execute pre_activation_02.sh error! Status: 255 2011-09-16 09:35:57 SOKERI: Pre-Activate error! Status: 255, Additional Status: 25
Issue: 02B
DN0976768
39
Software management troubleshooting
Troubleshooting Multicontroller RNC
Reason analysis Before activating the software delivery, the system executes the pre_activation_xx.sh script to do pre-activating inspection if such script exists. An error in the execution of the script causes the failure of the Download and Activate or Activate task. The reason for the failed execution of the script varies depending on what are specified in the script.
Recovery procedures 1
Make a copy of syslog file and ilsw_upgrade.log file (both under /var/log directory) respectively.
2
Send the copied files to Nokia Customer Service Center.
6.3.4 Upgrading eSW fails Symptoms The eSW upgrade fails during the mcRNC upgrade. The status INSTALLATION_FAILED is displayed on the ACTIVE/MAIN BANK or the PENDING BANK field when you execute the SCLI command show sw-managed embedded-sw status.
Reason analysis A successful eSW upgrade has certain requirements on the disk space. When the free disk space in the /tmp folder is below 7 MB, or the usage of the /dev/mtdblock1 file system exceeds 83%, the eSW upgrade fails.
Recovery procedures 1
Enter the bash shell as a root user. To enter the bash shell as a root user, execute the following command: shell bash full
2
Check the available space of each LMP. To check the available space of LMP, execute the following command for each LMP: ssh LMP-1-X-1 "df –h"
Example output for LMP-1-1-1 # ssh LMP-1-1-1 "df -h" Filesystem Size /dev/mtdblock1 117M tmpfs 506M /dev/mtdblock0 10M none 32M none 64M none 64M none 64M /dev/mtdblock5 117M
40
Used 103M 2.6M 2.6M 31M 0 22M 22M 96M
Avail 14M 504M 7.5M 1M 64M 43M 43M 21M
DN0976768
Use% 88% 1% 26% 97% 0% 34% 34% 83%
Mounted on / /dev/shm /boot /tmp /mnt/var /opt/fastpath/tmpfs /opt/fastpath2/tmpfs /mnt/backupfs
Issue: 02B
Troubleshooting Multicontroller RNC
3
Software management troubleshooting
Delete some unnecessary files to free more space. Delete some unnecessary files in the /tmp directory or root / file system of each LMP until: The free disk space in the /tmp directory is above 7 MB. The usage of the file system mounted on root / (/dev/mtdblock1 in the output of step 2) is below 83%.
• •
g
Do not delete any files that are important for the system functionality. For example, it is safe to delete the files in the /tmp/sysrep directory.
6.3.5 Alarm 2518 - NO VALID FALLBACK COPY FOR DEFAULT PACKAGE is triggered Symptoms Alarm 2518 is seen both from the NetAct Alarm Monitor graphical user interface (GUI), and the mcRNC command line interface when executing show alarm active SCLI command. An example of the command output is as follows: Alarm ID : 550 Specific problem : 2518 - NO VALID FALLBACK COPY FOR DEFAULT PACKAGE Managed object : fshaRecoveryUnitName=OMUTestServer0,fsipHostName=CLA-0,fsFragmentId=Nodes,fsFra gmentId=HA,fsClusterId=ClusterRoot Severity : 3 (major) Cleared : no Clearing : automatic Acknowledged : no Ack. user ID : N/A Ack. time : N/A Alarm time : 2012-01-04 11:17:47:633 CST Event type : x2 (processing error) Application : fshaProcessInstanceName=IL_Sokeri,fshaRecoveryUnitName=OMUTestServer0,fsipHostN ame=CLA0,fsFragmentId=Nodes,fsFragmentId=HA,fsClusterId=ClusterRoot IAppl Addl. Info : 4 Appl. Addl. Info :
Reason analysis The fault is caused by outdated fallback (FB) build. It is required that the FB build must be saved at a frequency of every seven days.
Recovery procedures Steps
1
Enter SCLI command interface on mcRNC To get into SCLI command interface, enter the following command: fsclish
Issue: 02B
DN0976768
41
Software management troubleshooting
2
Troubleshooting Multicontroller RNC
Create an FB build Enter the command as follows: save sw-manage app-sw-mgmt fb-build
Expected outcome The following output is displayed: FB build has been saved successfully.
Unexpected outcome The following output is displayed when the active build is not a backup (BU) build. Current build is not BU.
For more information on creating an FB build, see Creating fallback build from active backup build. 3
Check if Alarm 2518 has been cancelled After the new FB build is created, the alarm is cancelled automatically. You can see the result by either of the following ways: •
Use the show alarm active SCLI command or the show alarm active filter-by specific-problem 2518 command. Alarm 2518 still exists in the output, but the “Cleared” field is now “yes” and “Severity” has changed to “6 (cleared)”. For example: Alarm ID : 550 Specific problem : 2518 - NO VALID FALLBACK COPY FOR DEFAULT PACKAGE Managed object : fshaRecoveryUnitName=OMUTestServer0,fsipHostName=CLA-0,fsFragmentId=Nodes,fsFra gmentId=HA,fsClusterId=ClusterRoot Severity : 6 (cleared) Cleared : yes Clearing : automatic Acknowledged : no Ack. user ID : N/A Ack. time : N/A Alarm time : 2011-12-19 16:21:00:749 CST Event type : x2 (processing error) Application : fshaProcessInstanceName=IL_Sokeri,fshaRecoveryUnitName=OMUTestServ er-0,fsipHostN ame=CLA0,fsFragmentId=Nodes,fsFragmentId=HA,fsClusterId=ClusterRoot IAppl Addl. Info : 4 Appl. Addl. Info :
•
42
Check the active alarms in the Alarm Monitor of the NetAct GUI. Alarm 2518 has disappeared.
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
Networking troubleshooting
7 Networking troubleshooting 7.1 Packet loss in certain traffic flows unexpectedly high 7.1.1 Description The network element keeps the packet drop from different traffic flows under control due to the Quality of Service (QoS) system feature implemented on the ingress (incoming) and egress (outgoing) sides. The QoS on ingress side manages cases when NPU is overloaded and QoS on egress side manages cases when outgoing interface capacity is overloaded. In addition, the internal traffic is managed. The packet loss in certain traffic flow may be unexpectedly high because: •
•
The egress port rate-limit threshold is less than the converged traffic flows volume offered and ACL rules to set the proper pip mark for any outgoing streams might not be created Incoming packets might have wrong DSCP mark.
7.1.2 Symptoms The packet loss in certain traffic flows unexpectedly high.
7.1.3 Recovery procedures 1
Analyze the traffic flows coming out of the interface. To verify whether the packet loss happens in the egress interface, collect the egress interface QOS statistics by using the performance management measurement jobs, while the traffic is ongoing. While analyzing the report file, look for any discarded packet count for the specific egress interface. If there are no significant discarded packets, then skip step 1 and step 2 which handles egress side and proceed to step 3 which handles the ingress side.
2
Check the rate-limit value for the egress interface. To check the rate-limit value for the egress interface, enter the following command: show networking ether owner iface\ Check the rate-limit value in the output. If the rate-limit value is equal to the traffic passed, the lowest priority traffic may be dropped. To prevent packet loss, the rate-limit threshold may be increased. If this action is not desired, then proceed to step 2. To set the new rate-limit threshold, enter the following command: set networking ether iface rate-limit
Issue: 02B
DN0976768
43
Networking troubleshooting
g
Troubleshooting Multicontroller RNC
If the egress interface is a link aggregation group interface, then make sure that the rate limit value does not exceed the summation of all the ports that was under the link aggregation group interface. 3
Check if the traffic flow is appropriately handled by egress interface QoS. Each queue from a queue set attached to the interface implement particular PHB. The queue is selected for each packet according to the Packet Internal Priority. PIP is marked by classification function of the ACL. To check if the pip-mark for the flow under consideration is configured properly, enter the following command: show networking aclrule The following table provides the mapping between the queue ID and pip-mark. Table 5
Mapping between pip-mark and Queue ID
pip-mark
queue ID
0x00
0
0x01
1
0x02
2
0x03
3
0x04
4
0x05
5
0x06
5
0x07
5
To map the selected traffic to the right queue as listed in the table, see the section, Configuring ACL settings. The priority of the queue is determined by the weight assigned to the queue in the attached queue set. Increasing the length of the queue can also reduce the packet loss during short traffic bursts. However, system resources and traffic latency has to be taken into account. For more information refer to Configuring queue sets section. Also, the other traffic flow(s) may get higher priority than the flow under consideration, thus occupying the system resources. To check whether the attached queue set has appropriate weight and length, enter the following command: show networking qset name To modify the weight and length of the queue, enter the following command: set networking qset test1 queue weight \ length queue weight \ length
44
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
4
Networking troubleshooting
Verify that the incoming packet has expected DSCP value. a) Check the DSCP value from the Multicontroller RNC. •
Start tcpdump on Multicontroller RNC side to monitor traffic on the desired interface. –
–
•
Switch to root user and the bash shell. (Type exit later on to return to fsclish.) set user username root Provide the password for root user. Enable tcpdump for selected interface. #tcpdump -i -w /mnt/backup/tcp_dump.pcap
Open the collected trace with protocol analyzer and verify the DSCP value of the desired traffic flow.
Or a) Check the DSCP value from the external workstation. • •
Run protocol analyzer and capture the traffic of the desired interface. Verify the DSCP value of the collected traffic flow.
The following table lists the expected DSCP codes for external traffic. The DSCP codes not mentioned in the table are classified as best effort class and are handled with lowest priority. Table 6
Priority values for different DSCP codes
PHB Class
DSCP Code
Priority
nc
0x30
High priority
ef
0x2E
af41
0x22
af42
0x24
af43
0x26
af31
0x1A
af32
0x1C
af33
0x1E
af21
0x12
af22
0x14
af23
0x16
Issue: 02B
Medium priority
Low priority
DN0976768
45
Networking troubleshooting
Table 6
Troubleshooting Multicontroller RNC
Priority values for different DSCP codes (Cont.)
PHB Class
DSCP Code
af11
0x0A
af12
0x0C
af13
0x0E
be
0x00
Priority
7.2 Multihop BFD sessions are not established 7.2.1 Description The multihop BFD sessions may not be established if there are problems in BFD service, configuration, or network connectivity.
7.2.2 Symptoms The multihop BFD sessions are not established or sessions do not reach UP state.
7.2.3 Recovery procedures 1
Check the BFD service. To check the BFD service, follow these steps: a) Verify the state of the FSBFDServer recovery unit in the node where BFD is configured. Enter the following command: show has state managedobject //FSBFDserver The following output is displayed: root@CFPU-0 [RNC-89] > show has state managed-object /CFPU0/FSBFDserverOBJECT ADMINISTRATIVE OPERATIONAL ROLE PROCEDURAL DYNAMIC/CFPU-0/FSBFDServer UNLOCKED ACTIVE ACTIVE -
USAGE ENABLED
If the recovery unit is locked, unlock it using the following command: set has unlock managed-object //FSBFDServer
2
Check BFD alarms. Verify that there are active alarms for BFD (specific problem 70348). Enter the following command: show alarm active filter-by specific-problem 70348 If the active BFD alarms are found, verify BFD configuration (see step 4) and network connectivity (see step 5).
46
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
3
Networking troubleshooting
Check the BFD runtime information. Verify that the BFD sessions are found in the runtime environment and that the session state is correct. Enter the following command: show networking instance bfd runtime node /
The Init state means that the remote system is communicating and the local system brings the session up, but the remote system does not realize it. If the sessions are not found, verify the BFD service state provided by HAS. If the session state is down, verify the basic IP connectivity.
g
Diagnostic field shows a diagnostic code specifying the reason for change in session state.
4
Check BFD configuration. To check the BFD configuration, follow these steps: a) Verify that BFD is enabled on the correct node with correct parameters. Enter the following commands: show networking monitoring bfd config all If the BFD is not enable on the correct node with correct parameter, see the section, Configuring IP Connection for Multicontroller RNC for more information. b) Verify that local IP address for BFD session is found from the node and is assigned to the correct VRF instance. Enter the following command: show networking address
g
BFD sessions can be configured before IP address is added to the interface. However, the BFD session control packet are sent only after the IP address is added. If the local IP address for the BFD session is not found and assigned to the correct VRF instance, see the section, Configuring the BFD multihop session section for more information. c) Verify that there are no ACL rules that block BFD packets. Enter the following command: show networking aclrule If required add rule for allowing BFD packets in both the directions, see the section, Configuring IP Connection for Multicontroller RNCfor more information.
g 5
Multihop BFD control packets are transmitted in UDP with destination port 4784.
Check the IP connectivity between routers. To check the IP connectivity between routers, follow these steps: a) Verify basic IP connectivity between network devices using ping command. Ping from the correct VRF in node you have configured BFD session. If there is no response to ping verify the interface state. To execute the ping command from the SCLI, start the bash shell (in full mode) and then use ssh to connect to the correct node. Enter the following commands: shell bash full ssh ping -V or ping exit
Issue: 02B
DN0976768
47
Networking troubleshooting
Troubleshooting Multicontroller RNC
If you want to make sure that the node where BFD is configured, accepts ICMP (protocol 1) packets in input and output chain, enter the following command: show networking aclrule owner / If the node does not accept ICMP (protocol 1) packets, then add accept rules by entering the following commands: add networking aclrule / index 20000 chain\ input protocol 1 accept add networking aclrule / index 20001 chain\ output protocol 1 accept
g
The neighboring router may be configured to dropping requests and may cause ping not to respond. b) Verify that the state for the interface where BFD session is configured has admin state up and has running flag set. Enter the following command: show networking interface runtime If running flag is not present, verify that cable is properly connected to the interface. If the admin state is not up, see Configuring IP Connection for Multicontroller RNCfor more information. c) Verify that the BFD control packets are sent and received on the interface where BFD session is configured. To execute this command from SCLI, start the bash shell (in full mode) and then use ssh to connect to the correct node. Enter the following commands: shell bash full ssh tcpdump -i port 4784 exit This command does not require root access. If only outgoing BFD packets are seen, verify the configuration at the remote endpoint.
w
NOTICE: Nokia advises that only advanced users, deeply familiar with the behavior of the network element, must use commands of the full bash in general and the tcpdump command in particular. Even if the root account or dedicated non root account access is used, the commands from the full bash shell can be easily exploited, For example, to change security-relevant configuration. Also the tcpdump command in particular when accidentally or on-purpose misused can cause significant performance penalties (a drop of 0-100%) in networking (throughput and latencies), disk I/O or processing capacity. In the worst case this might, for example, cause automatic recovery actions. Also some parts of the tcpdump command might not work as expected due to the peculiarities of the network element architecture.
7.3 OSPF is not working properly 7.3.1 Description If there are problems with OSPF configurations, interface state, or network connectivity the OSPF neighbor adjacencies are not formed and therefore OSPF route advertisement does not work.
7.3.2 Symptoms OSPF adjacencies are not formed and the routing table does not include OSPF routes. There is no network connectivity to networks advertised by OSPF. The OSPF neighbors are not seen or OSPF not configured error message is displayed when monitoring the OSPF neighbor adjacency state from the SCLI command.
48
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
Networking troubleshooting
7.3.3 Recovery procedures 1
Check the routing service. Check the state of FSRoutingDaemonServer recovery unit in the node where OSPF is configured. Enter the following command: show has state managed-object //FSRoutingDaemonServer The following output is displayed: OBJECT ADMINISTRATIVE OPERATIONAL PROCEDURAL DYNAMIC//FSRoutingDaemonServer UNLOCKED ACTIVE ACTIVE -
USAGE
ROLE ENABLED
If the FSRoutingDaemonServer recovery unit is locked, the OSPF neighbor query to the corresponding node will result in an OSPF not configured error. If FSRoutingDaemonServer recovery unit is locked, unlock it using the following command: set has unlock managed-object //FSRoutingDaemonServer
2
Check the OSPF runtime information. To check the OSPF runtime information, follow these steps: a) Verify that the neighbor adjacency state has reached 2-way or full state. Enter the following command: show networking instance node ospf neighbors
g
The full adjacency is formed only with designated router and backup designated router. The time to reach 2-way or full state depends on the configured OSPF parameters. If the command fails with the OSPF not configured error, verify routing service state (see step 1) and OSPF configuration (see step 3). If OSPF neighbor adjacency are not formed, check the configuration (see step 3). b) Verify that the basic IP connectivity between routers is working (see step 4). c) Verify that the OSPF hello packets are transmitted and received. Enter the following command: show routing instance node \ ospf packets If the OSPF hello packets are not transmitted and received, check that OSPF is configured properly (see step 3). d) Verify that OSPF routes are found in routing table. Enter the following command: show routing instance node \ route ospf OSPF internal routes are marked with O in front of the route and external routes are marked with OE. If some of the routes are missing check the configuration (see step 3).
3
Check OSPF configuration. To check the OSPF configuration, follow these steps: a) Verify that the OSPF license has been installed and the feature admin state is ON. Enter the following command: show license feature-mgmt name OSPFForRedundancy
Issue: 02B
DN0976768
49
Networking troubleshooting
g
Troubleshooting Multicontroller RNC
OSPF does not start if the license is not installed or if the license is installed but is inactive or has expired. Also if the feature admin state is off or config. If the feature admin state is off or config, enter the following command to set it to on: set license feature-mgmt name OSPFForRedundancy feature-adminstate on If the OSPF license is not installed, then enter the following command to install the license: add license file Where, file specifies the location of OSPF license. b) Verify that OSPF is enabled on the correct interface and OSPF area. Enter the following commands: show routing instance node \ ospf config If the OSPF is not enabled on the correct interface and area, see Configuring IP Connection for Multicontroller RNCfor more information. c) Verify that an IPv4 address is set for the interface where OSPFv2 is configured. Enter the following command: show networking address Example If the OSPFv2 is configured for the eth_sfp1 interface, enter the following command: show networking instance default address owner /IPNIU1-0\ iface eth_sfp1 The following output is displayed: address instance default interfaces : eth_sfp1 type : dedicated address : 200.0.0.1/24 owner : /IPNIU1-0
If the IPv4 address is missing, add it to the corresponding interface, see Configuring IP Connection for Multicontroller RNCfor more information. d) Verify that the router-id is set. Enter the following command: show routing instance node \ router-id The system automatically assigns router ID based on the IPv4 addresses found. Note that router-id must be unique for the OSPF area. If the router-id is not set for the OSPF area, enter the following command to set the router-id: set routing instance node router-id\ e) Verify that there are no OSPF protocol errors. OSPF errors happen usually due to configuration mismatch between the routers where OSPF is configured. Enter the following command: show routing instance node \ ospf errors If some errors are occurring continuously, check the configuration related to the error. For example, if hello timer mismatch error counter is increasing, verify that the hello interval parameters are same in both endpoints where OSPF is configured. f) Verify that there are ACL rules that explicitly accept OSPF traffic or there are no ACL rules that explicitly drop OSPF traffic. Enter the following command:
50
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
Networking troubleshooting
show networking aclrule If required add rule for allowing OSPF packets in both the directions, see Configuring IP Connection for Multicontroller RNCfor more information. g) Verify that there are no unwanted inbound filtering rules. Enter the following commands: show routing node inbound-filter config\ proto ospf2ase all
g
Inbound filtering rules apply only to the OSPF external routes advertised as type 5 LSAs. To disable the unwanted inbound filtering rules, enter the following commands: set routing node inbound-filter proto \ ospf2ase off
4
Check the IP connectivity between routers. To check the IP connectivity between routers, follow these steps: a) Verify the basic IP connectivity between routers using the ping command. Ping from the correct VRF (if supported) in node where OSPF is configured. If the neighbor does not respond to ping verify the interface state. To execute the ping command from the SCLI, start the bash shell (in full mode) and then use ssh to connect to the correct node. Enter the following commands: shell bash full ssh ping -V or ping exit If you want to make sure that the node where IPsec traffic under investigation is configured accepts ICMP (protocol 1) packets in input and output chain, enter the following command: show networking aclrule owner / If the node does not accept ICMP (protocol 1) packets, then add accept rules by entering the following commands: add networking aclrule / index 20000 chain input\ protocol 1 accept add networking aclrule / index 20001 chain \ output protocol 1 accept
g
The neighboring router may be configured to drop ping requests and may cause ping not to respond. b) Verify that the state for the interface where OSPF is configured has admin state up and has running flag set. Enter the following command: show networking interface runtime Example If the OSPF is configured for ethsfp15 interface, enter the following command to verify the state of the interface: show networking interface runtime iface ethsfp15
The following output is displayed: ethsfp15 index : 11 node : EIPU-2 type : VLAN flags : UP BROADCAST RUNNING PROMISC MULTICAST FP_OUTPUT speed : 1G MAC : 00:d0:c9:ba:65:fb VRF name(ID) : default (0) vid : 15 realiface : xaui1 MTU : 1500 admin state : up oper state : up Rx packets : total : 67202 bytes : 6968122 error : 0 Tx packets : total : 117762 bytes : 11572092 error : 0
Issue: 02B
DN0976768
51
Networking troubleshooting
Troubleshooting Multicontroller RNC
If running flag is not present, verify that the cable is properly connected to the interface. If the admin state is not up, see Configuring IP Connection for Multicontroller RNCfor more information.
7.4 IP signaling link activation fails Description The procedure for checking IP type signaling links is different from checking ordinary signaling links. Follow this procedure if you have not been able to activate the IP signaling link you have created, and the signaling link stays in the state inactive.
Symptoms You have not been able to activate the IP signaling link you have created, and the signaling link stays in state inactive.
Recovery procedures Checking why IP signaling link activation fails Steps 1
Check the states and parameters of the association sets. show signaling ss7 association all
2
Check the SCTP level parameters by checking SCTP profile. show signaling sctp-profile all Checking is especially important in case of retransmission and also if the SCTP association goes down, but there is no LAN failure.
3
Check the service access point configuration and local/remote AS. Checking SAP configuration by commands: show signaling service-access-point all show signaling ss7 local-as all show signaling ss7 remote-as all
4
Check that the IP addresses of the QNUP and SS7SGU users are correct. show networking address Based on the command output, check if the following conditions are fulfilled: • •
52
QNUP is set as a user for Iu-CS and Iu-PS U-plane IP address SS7SGU is set as a user for Iu-CS and Iu-PS C-plane IP address
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
Networking troubleshooting
Compare the IP addresses to the IP addresses assigned for the associations at the far end. 5
Check the availability of the peer IP address with PING command.
6
Check the IP network configuration.
7.5 The state of all subsystems in the remote network element is unavailable (UA) Description The state of all subsystems in the remote network element is unavailable (UA).
Symptoms The state of all subsystems in the remote network element is unavailable (UA).
Recovery procedures Checking why the state of all subsystems in the remote network element is unavailable (UA) Steps 1
Check the state of the remote subsystem. show signaling sccp subsystem all
2
Check mcRNC IP reachability by ping tool. If the subsystem of the remote node is in state ”not allowed” , try to re-establish the signaling link by disable and enable related association: set signaling ss7 association admin-state disabled id set signaling ss7 association admin-state enabled id
Issue: 02B
DN0976768
53
Hardware troubleshooting
Troubleshooting Multicontroller RNC
8 Hardware troubleshooting 8.1 No traffic detected 8.1.1 Description Traffic could be cut if ports administration has not been configured or switch ports are damaged. Also cabling could be damaged or unpluged.
8.1.2 Symptoms The following symptoms may indicate a problem with the traffic: • •
There is a traffic break detected in the RNC. There is no traffic in the RNC after maintenance activity.
8.1.3 Recovery procedure 1
Check ports configurationr with map e m using the mcJANE tool.
2
Check ports status with the command ZLAI using the mcJANE tool. Administrator mode should be Enable and link status should be up. If admninistrator mode is Disable, there is configuration problem or switch ports might be broken. If only link status is down and administrator mode is Enable, the cable is broken or unplugged. Example: # zjane ZLAI Fri Apr 27 10:31:09 EEST 2012 : BCM: show SFP ports: Admin Physical Physical Link Link Intf Type Mode Mode Status Status Trap ------ ------ ------- ---------- ---------- ------ ------SFP1 Enable 10G Full 10G Full Up Enable SFP2 Enable 10G Full 10G Full Up Enable SFP3 Enable Down Enable SFP4 Enable Down Enable SFP5 Enable Down Enable SFP6 Enable Auto 1000 Full Up Enable SFP7 Enable Auto 1000 Full Up Enable SFP8 Enable Auto 1000 Full Up Enable SFP9 Disable Auto Down Enable SFP10 Disable Auto Down Enable SFP11 Disable Auto Down Enable SFP12 Disable Auto Down Enable SFP13 Disable Auto Down Enable SFP14 Disable Auto Down Enable SFP15 Enable Auto 1000 Full Up Enable SFP16 Enable Auto 1000 Full Up Enable SFP17 Disable Auto Down Enable
54
DN0976768
LACP Mode -----Enable Enable Enable Enable Enable Enable Enable Enable Enable Enable Enable Enable Enable Enable Enable Enable Enable
Actor Timeout -------long long long long long long long long long long long long long long long long long
Issue: 02B
Troubleshooting Multicontroller RNC
SFP18 SFP19 SFP20 SFP21 SFP22
3
Hardware troubleshooting
Disable Disable Disable Disable Disable
Auto Auto Auto Auto Auto
Down Down Down Down Down
Enable Enable Enable Enable Enable
Enable Enable Enable Enable Enable
long long long long long
Check link data layer If physical connection is ok, use ss command to check if any packets have been received. In case of positive statistics result, a link between switches is not correct. This might indicate wrong switch configuration. Use cp command to analyze a traffic and packets destination.
4
Check cabling Check whether the cables are broken or unpluged.
8.2 SFP ports added for USPUs and CSPUs do not show correctly their operational state nor alarms are raised Description The SFP ports are assigned to USPU/CSPU nodes directly but only for monitoring and log collection purposes (for example, Megamon). The actual network connectivity is always provided by the CFPU (O&M) and EIPU nodes. If SFP ports are added to USPUs or CSPUs, then the operational state (RUNNING flag) of those interfaces does not change and alarms are never raised. This is a limitation in the Linux kernel because there is no way to change the RUNNING flag in those interfaces without changing the USPUs or CSPUs.
Symptoms If SFP ports are added to USPUs or CSPUs as Ethernet interfaces, then the operation state (oper state) for these interfaces are always ‘up’ regardless of the state of the physical port. And no alarms are raised for these interfaces even if the physical port goes ‘down’.
Recovery procedures Not available
Issue: 02B
DN0976768
55
Hardware troubleshooting
Troubleshooting Multicontroller RNC
8.3 Troubleshooting with mcJANE tool The mcJANE is the debugging tool for the mcRNC hardware and software. The debugging area of mcJANE concerns Octeon processor, BCM chip and External Interface Processing Unit (EIPU). The mcJANE tool collects data from the mcRNC ports. Analyzed data are used to find faults in the mcRNC hardware.
Logging to the mcJANE tool Expected outcome After logging to the unit you want to debug use the zjane. The command syntax for mcJANE tool. show troubleshooting z-commands zjane box
Steps
1
Log in to one of the mcRNC units with ssh.
2
Use show troubleshooting z-commands zjane box and press TAB to view mcJane commands available. _nokadmin@CFPU-0 [RNC-38] > show troubleshooting z-commands zjane box 10 [X] - press to execute the command [X] zi - Show interface configuration from running-config. [X] zlai - Show SFP modules port status (SFP1-22). [X] zm - Show port monitor (mirroring) configuration from main and ext switch. [X] zmac - Show Main and Extension switch mac-addr-table. [X] zmap - Show Main and/or Extension switch port mapping. [X] zr - Show running-config from BCM. Give [e|m] for getting Ext or Main running conf only [X] zreg - Show Octeon PIP registers [ ipd | pip | pko | pow | sso]. [X] zs - Get L2 unicast Rx/Tx packet count statistics from BCM switch port in loop mode. [X] zss - Get detailed L2 statistics from BCM switch port or clear statistics. [X] zt - Get traplogs from main and ext switch. [X] zu - Show uptimes.
3
Start debugging with one of the mcJANE tool commands.
Debugging mcRNC switch hardware
56
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
Hardware troubleshooting
Steps
1
Enter the map command. The map command shows SPF port module status.
g
Note To see both main and extension switch port mapping enter: map e m. # zjane map ZJANE @RNC-37BOX: 1Motherboard:BCNMB-ACPU:Octeon+Thu Oct 16 14:15:29 CEST 2014 Thu Oct 16 14:15:29 CEST 2014 : Show port map ------------| BCM56512 | | extension | | | HG0e:-----|0/25 | HG1e:-----|0/26 | | | SFP7:-----|0/1 | SFP8:-----|0/2 | SFP9:-----|0/3 | SFP10:----|0/4 | SFP11:----|0/5 | SFP12:----|0/6 | SFP13:----|0/7 | SFP14:----|0/8 | SFP15:----|0/9 | SFP16:----|0/10 | SFP17:----|0/11 | SFP18:----|0/12 | SFP19:----|0/13 | SFP20:----|0/14 | SFP21:----|0/15 | SFP22:----|0/16 | -------------
------------| BCM56820 | | main | SFP1:-----|0/21 | SFP2:-----|0/22 | SFP3:-----|0/23 | SFP4:-----|0/24 | SFP5:-----|0/1 | SFP6:-----|0/2 | | | NP1x0:----|0/19 int | NP1x1:----|0/20 ext | NP2x0:----|0/17 int | NP2x1:----|0/18 ext | NP3x0:----|0/15 int |
Issue: 02B
DN0976768
57
Hardware troubleshooting
Troubleshooting Multicontroller RNC
NP3x1:----|0/16 ext | NP4x0:----|0/13 int | NP4x1:----|0/14 ext | NP5x0:----|0/11 int | NP5x1:----|0/12 ext | NP6x0:----|0/9 int | NP6x1:----|0/10 ext | NP7x0:----|0/7 int | NP7x1:----|0/8 ext | NP8x0:----|0/5 int | NP8x1:----|0/6 ext | | | HG0m:-----|0/3 | HG1m:-----|0/4 | -------------
2
Check the SFP port module status by entering the ZLAI command. Check if all SPF ports have the appropriate status. The Disabled status for ports which are not used and Enabled status for used ports. For example: # zjane ZLAI ZJANE @RNC-37BOX: 1Motherboard:BCNMB-ACPU:Octeon+Thu Oct 16 14:15:48 CEST 2014 Thu Oct 16 14:15:48 CEST 2014 : BCM: show SFP ports: Admin Physical Physical Link Link LACP Actor Intf Type Mode Mode Status Status Trap Mode Timeout ------ ------ ------- ---------- ---------- ------ ------- ------ ------SFP1 Enable 10G Full 10G Full Up Enable Enable long SFP2 Enable 10G Full 10G Full Up Enable Enable long SFP3 Disable Down Enable Enable long SFP4 Disable Down Enable Enable long SFP5 Disable Down Enable Enable long SFP6 Disable Down Enable Enable long SFP7 Disable 1000 Full Down Enable Enable long SFP8 Disable 1000 Full Down Enable Enable long SFP9 Disable 1000 Full Down Enable Enable long SFP10 Disable 1000 Full Down Enable Enable long SFP11 Disable 1000 Full Down Enable Enable long SFP12 Disable 1000 Full Down Enable Enable
58
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
long SFP13 long SFP14 long SFP15 long SFP16 long SFP17 long SFP18 long SFP19 long SFP20 long SFP21 long SFP22 long
3
Hardware troubleshooting
Enable
1000 Full
1000 Full
Up
Enable
Enable
Enable
1000 Full
1000 Full
Up
Enable
Enable
Disable 1000 Full
Down
Enable
Enable
Disable 1000 Full
Down
Enable
Enable
Enable
1000 Full
1000 Full
Up
Enable
Enable
Enable
1000 Full
1000 Full
Up
Enable
Enable
Disable 1000 Full
Down
Enable
Enable
Disable 1000 Full
Down
Enable
Enable
Disable 1000 Full
Down
Enable
Enable
Enable
Down
Enable
Enable
1000 Full
Check the switch running configuration. Check running configuration with r command or i for the one interface configuration.
4
Collect detailed statistics with ss command. Detailed statistics show current statistics without comparing to the previous result.
Issue: 02B
DN0976768
59
Hardware troubleshooting
t
Troubleshooting Multicontroller RNC
Note You can clear detailed statistics using: ss SPF7 clear or ss m clear all For example: # zjane ss SFP20 ZJANE @RNC-37BOX: 1Motherboard:BCNMB-ACPU:Octeon+Fri Oct 17 11:45:43 CEST 2014 Fri Oct 17 11:45:43 CEST 2014 : Getting L2 statistics: Target is EXT switch port SFP20(0/14). ############################################### show interface ethernet 0/14 Total Packets Received (Octets)................ 0 Packets Received 64 Octets..................... 0 Packets Received 65-127 Octets................. 0 Packets Received 128-255 Octets................ 0 Packets Received 256-511 Octets................ 0 Packets Received 512-1023 Octets............... 0 Packets Received 1024-1518 Octets.............. 0 Packets Received > 1522 Octets................. 0 Packets RX and TX 64 Octets.................... 0 Packets RX and TX 65-127 Octets................ 0 Packets RX and TX 128-255 Octets............... 0 Packets RX and TX 256-511 Octets............... 0 Packets RX and TX 512-1023 Octets.............. 0 Packets RX and TX 1024-1518 Octets............. 0 Packets RX and TX 1519-1522 Octets............. 0 Packets RX and TX 1523-2047 Octets............. 0 Packets RX and TX 2048-4095 Octets............. 0 Packets RX and TX 4096-9216 Octets............. 0 Total Packets Received Without Errors.......... 0 Unicast Packets Received....................... 0 Multicast Packets Received..................... 0 Broadcast Packets Received..................... 0 Total Packets Received with MAC Errors......... 0 Jabbers Received............................... 0 Fragments Received............................. 0 Undersize Received............................. 0 Alignment Errors............................... 0 FCS Errors..................................... 0 Overruns....................................... 0 Total Received Packets Not Forwarded........... 0 Local Traffic Frames........................... 0 802.3x Pause Frames Received................... 0 Unacceptable Frame Type........................ 0 Multicast Tree Viable Discards................. 0 Reserved Address Discards...................... 0 Broadcast Storm Recovery....................... 0 CFI Discards................................... 0 Upstream Threshold............................. 0 Total Packets Transmitted (Octets)............. 0
60
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
Hardware troubleshooting
Packets Transmitted 64 Octets.................. Packets Transmitted 65-127 Octets.............. Packets Transmitted 128-255 Octets............. Packets Transmitted 256-511 Octets............. Packets Transmitted 512-1023 Octets............ Packets Transmitted 1024-1518 Octets........... Max Frame Size................................. Total Packets Transmitted Successfully......... Unicast Packets Transmitted.................... Multicast Packets Transmitted.................. Broadcast Packets Transmitted.................. Total Transmit Errors.......................... FCS Errors..................................... Tx Oversized................................... Underrun Errors................................ Total Transmit Packets Discarded............... Single Collision Frames........................ Multiple Collision Frames...................... Excessive Collision Frames..................... Port Membership Discards....................... 802.3x Pause Frames Transmitted................ GVRP PDUs received............................. GVRP PDUs Transmitted.......................... GVRP Failed Registrations...................... GMRP PDUs Received............................. GMRP PDUs Transmitted.......................... GMRP Failed Registrations...................... STP BPDUs Transmitted.......................... STP BPDUs Received............................. RSTP BPDUs Transmitted......................... RSTP BPDUs Received............................ MSTP BPDUs Transmitted......................... MSTP BPDUs Received............................ EAPOL Frames Transmitted....................... EAPOL Start Frames Received.................... Time Since Counters Last Cleared............... sec
5
0 0 0 0 0 0 1522 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 day 0 hr 59 min 40
Collect unicast packet statistics form both main and external switch port. Get statistics of send and received traffic with s. The s command allows checking the unicast traffic in selected interface. For example: # zjane s e 0/16 ZJANE @RNC-37BOX: 1Motherboard:BCNMB-ACPU:Octeon+Fri Oct 17 11:46:43 CEST 2014 Fri Oct 17 11:46:43 CEST 2014 : Getting L2 Tx and Rx counters from BCM, loop mode (press Ctrl+C for exit...) ------------------ 11:46:43 ---------------------EXT 0/16 Rx: 0 packets EXT 0/16 Tx: 0 packets ------------------ 11:46:49 ---------------------EXT 0/16 Rx: 0 packets EXT 0/16 Tx: 0 packets
Issue: 02B
DN0976768
61
Hardware troubleshooting
Troubleshooting Multicontroller RNC
------------------ 11:46:54 ---------------------EXT 0/16 Rx: 0 packets EXT 0/16 Tx: 0 packets ------------------ 11:47:00 ---------------------EXT 0/16 Rx: 0 packets EXT 0/16 Tx: 0 packets ------------------ 11:47:05 ---------------------EXT 0/16 Rx: 0 packets EXT 0/16 Tx: 0 packets ------------------ 11:47:11 ---------------------^CFri Oct 17 11:47:12 CEST 2014 : INFO :: trap handle started. Fri Oct 17 11:47:12 CEST 2014 : INFO :: wait 2sec.... # zjane s e "0/11 0/16 m "0/7 0/9 0/11 0/13"" m "0/7 0/9 0/11 0/13" ZJANE @RNC-37BOX: 1Motherboard:BCNMB-ACPU:Octeon+Fri Oct 17 11:48:54 CEST 2014 Fri Oct 17 11:48:54 CEST 2014 : Getting L2 Tx and Rx counters from BCM, loop mode (press Ctrl+C for exit...) ------------------ 11:48:57 ---------------------MAIN 0/7 Rx: 241 packets MAIN 0/7 Tx: 390 packets MAIN 0/9 Rx: 275 packets MAIN 0/9 Tx: 455 packets MAIN 0/11 Rx: 158 packets MAIN 0/11 Tx: 249 packets MAIN 0/13 Rx: 203 packets MAIN 0/13 Tx: 281 packets EXT 0/11 Rx: 27 packets EXT 0/11 Tx: 12 packets EXT 0/16 Rx: 0 packets EXT 0/16 Tx: 0 packets ------------------ 11:49:06 ---------------------MAIN 0/7 Rx: 234 packets MAIN 0/7 Tx: 381 packets MAIN 0/9 Rx: 240 packets MAIN 0/9 Tx: 398 packets MAIN 0/11 Rx: 145 packets MAIN 0/11 Tx: 239 packets MAIN 0/13 Rx: 198 packets MAIN 0/13 Tx: 285 packets EXT 0/11 Rx: 32 packets EXT 0/11 Tx: 16 packets EXT 0/16 Rx: 0 packets EXT 0/16 Tx: 0 packets ------------------ 11:49:14 ---------------------MAIN 0/7 Rx: 234 packets MAIN 0/7 Tx: 384 packets MAIN 0/9 Rx: 855 packets MAIN 0/9 Tx: 1356 packets MAIN 0/11 Rx: 524 packets MAIN 0/11 Tx: 860 packets MAIN 0/13 Rx: 645 packets MAIN 0/13 Tx: 917 packets EXT 0/11 Rx: 99 packets EXT 0/11 Tx: 48 packets EXT 0/16 Rx: 0 packets
62
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
Hardware troubleshooting
EXT 0/16 Tx: 0 packets ------------------ 11:49:42 ---------------------MAIN 0/9 Rx: 234 packets MAIN 0/9 Tx: 374 packets MAIN 0/11 Rx: 173 packets MAIN 0/11 Tx: 278 packets MAIN 0/13 Rx: 169 packets MAIN 0/13 Tx: 250 packets EXT 0/11 Rx: 29 packets EXT 0/11 Tx: 14 packets EXT 0/16 Rx: 0 packets EXT 0/16 Tx: 0 packets ------------------ 11:49:51 ---------------------^CFri Oct 17 11:49:54 CEST 2014 : INFO :: trap handle started. Fri Oct 17 11:49:54 CEST 2014 : INFO :: wait 2sec....
6
Check MAC addresses that the mcRNC learned. The mcRNC can learn MAC addresses of other network devices. You can check the MAC address table using the following command: #zjane mac The mac command shows addresses learned by ports from the main and extension switch.
Port monitoring Steps
1
Connect the laptop to one of mcRNC ports.
2
Start the Wireshark on your laptop.
3
Copy the traffic from the source port to the port where the laptop is connected to. The cp command enables copying the traffic form the source port to the destination port. If the port you want to copy traffic from is disabled the question port enabling is shown. #zjane cp
Issue: 02B
DN0976768
63
Hardware troubleshooting
t
Troubleshooting Multicontroller RNC
Note The port you want to copy traffic to might be Disabled. To enable port you can force change of the state with f command. #zjane f The port state can be up or down. For example: Figure 1
Forcing change of the SPF post state
For example:
64
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
Figure 2
g
Hardware troubleshooting
Example usage of the cp command
Note It is possible to copy traffic from more than one port. The traffic of all the chosen ports is copied to the destination port. cp e "0/14 0/15 0/16 " 0/14 cp e "0/11 0/12" 0/15
w
Issue: 02B
Attention The traffic form all chosen ports is summed and copied to the destination port. Calculate the traffic, to not overload the bandwidth of the destination SFP port.
DN0976768
65
Hardware troubleshooting
4
Troubleshooting Multicontroller RNC
Show monitored ports. To see all monitored ports use m command. Commands m m and m e show respectively the main and the external port monitoring. For example: Figure 3
5
Checking the port monitoring
End port monitoring with rm. End the port monitoring before closing Wireshark. The rm command also works with removing all m main and e external ports monitoring. For example:
66
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
Figure 4
Hardware troubleshooting
Removing port monitoring
Debugging Octeon registers 1
Check the Octeon registers. The Octeon registers can be checked with the following commands: • • • •
Issue: 02B
ipd pip pko pow
DN0976768
67
Hardware troubleshooting
g
Troubleshooting Multicontroller RNC
Note To check the single register or group of the registers enter: #zjane For example: #zjane pko PKO_REG_READ_IDX
68
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
Performance management troubleshooting
9 Performance management troubleshooting 9.1 Threshold monitoring alarm is not sent to NetAct Description Threshold monitoring alarm is not received in NetAct when the threshold limit is exceeded in OMS Threshold Monitoring.
Symptoms Threshold monitoring alarm not sent to NetAct One of the following threshold monitoring alarms is not received in NetAct: 71005 THRESHOLD MONITORING LIMIT EXCEEDED, 71006 WCEL THRESHOLD MONITORING LIMIT EXCEEDED or 71007 MEASUREMENT THRESHOLD MONITORING LIMIT EXCEEDED.
Recovery procedures Checking why threshold monitoring alarm is not sent to NetAct Steps 1
Check alarm history. For transport and HW measurements the alarm 71005 THRESHOLD MONITORING LIMIT EXCEEDED set by Threshold Monitoring is of the type disturbance that does not need to be cancelled and it is not shown in the active alarms list in NetAct. For radio network measurements the threshold alarms 71006 WCEL THRESHOLD MONITORING LIMIT EXCEEDED and 71007 MEASUREMENT THRESHOLD MONITORING LIMIT EXCEEDED are one star alarms that are active until their live time expires. The live time should be set so that it is five minutes longer than the used measurement interval.
2
Check that the measurements for which the threshold has been set, are active. Note that creating threshold rules with NE Threshold Management GUI does not start corresponding measurements automatically.
3
Check the following with NE Threshold Management GUI: • • •
Issue: 02B
Threshold rule is correct. Write Log and Make Alarm is selected from the Log Action drop-down menu. Threshold Evaluation has been created for the rule and its target objects are correct. The evaluations are shown in the Thresholds Tree tab under Threshold rule objects.
DN0976768
69
Performance management troubleshooting
4
Troubleshooting Multicontroller RNC
Check with RNW Measurement Presentation GUI that measurement data has been received for the measurement type used in the threshold rule.
9.2 Transport and HW measurement management in mcRNC 9.2.1 McRNC measurement management concepts The mcRNC Transport&HW measurements are managed by using the mcRNC command line interface. There is no management functionality for these measurements in the OMS, but the OMS processes the measurements whenever data is received from the mcRNC. The measurement data is stored in the OMS database and sent to NetAct with the same interval at which the mcRNC produces the data. It is possible to define explicit object lists for the measurements but usually that is not necessary. By default, the measurement is active for all the objects when the object list is not created at all. Measuring all the objects means that also the objects added later in the RNC configuration (for example new IP-based routes when a new BTS is taken into use) are included automatically in the measurement. When activating a measurement, the first task is to create a measurement job. The job defines the measurement type and the measurement interval. The supported measurement intervals are 15 min (900 s), 30 min (1800 s) and 60 min (3600 s). The interval is given in seconds in the job creation command. The measurement scheduling is synchronized as follows: • • •
15 minute interval measurement data is collected at xx:00, xx:15, xx:30, and xx:45 30 minute interval measurement data is collected at xx:00 and xx:30 60 minute interval measurement data is collected at full hours xx:00
The following measurements are managed by using the principle described in this chapter: • • • • • • • • • •
g
70
568 IP Based Route measurement 569 IP TWAMP Statistics 609 DSP service statistics 610 L2 Resource Utilization 802 RNC capacity usage 803 RNC RTP/RTCP 804 RNC IP CAC 2002 CPU Usage 2004 Networking Logical 2007 Node Level CPU load The mcRNC measurement management sCLI shows additional measurements that are not mentioned in the list above. However, they are not supported and should not be activated.
DN0976768
Issue: 02B
Troubleshooting Multicontroller RNC
Performance management troubleshooting
9.2.2 McRNC measurement management commands Transport & HW measurements in the mcRNC are managed using stats command group in sCLI.
Measurement management in sCLI 1. Start sCLI: [root@CFPU-0(RNC-57) /root] # fsclish
2. Create a measurement job for measurement type 2002 with a 900 second (15 minutes) interval: root@CFPU-0 [RNC-57] continuous new job id: 1
g
> add stats m-job name test omes 2002 granularity 900
The supported measurement intervals are 900 seconds, 1800 seconds and 3600 seconds. Measurement intervals shorter than 900 seconds are not supported. 3. Activate the measurement for the previously created job ID: root@CFPU-0 [RNC-57] > set stats m-job id 1 enabled Measurement job id: 1 enabled.
4. Check measurement status: root@CFPU-0 [RNC-57] > show stats m-job all Job ID and name : 1 - test Measurement type : 2002 - Cpu Usage Object list ID and name : [ALL] Start time : 2012.12.31 19:05:42 Stop time : Non-stop Granularity period (ROP) : 900 seconds Number of periods : 0 Job type : Continuous Admin state : disabled Operational state : disabled
5. Stop the measurement: root@CFPU-0 [RNC-57] > set stats m-job id 1 disabled Measurement job id: 1 will be disabled.
6. Delete the measurement job: root@CFPU-0 [RNC-57] > delete stats m-job id 1 Measurement job id: 1 deleted.
g
Issue: 02B
After starting or stopping the measurements, the LDAP configuration must be saved. Otherwise the settings are lost after the next system restart. Save the configuration with sCLI command “save snapshot”. It is not necessary to save the configuration after every measurement management action. The operator can start or stop multiple measurements, and when all the operations are finished, saving must be performed.
DN0976768
71
View more...
Comments
