Description
Basic KPI Troubleshooting
Basic Key Performance Indicators •
RRC Success Rate - This is to evaluate Accessibility of the network. “ConnectionEstablishmentSuccess/ConnectionEstablishmentAttempt”
•
Connection Drop Rate (CDR) – This is evaluate disconnections/drops for all kind of services (QCI1 to QCI9) “E-RAB Release whilst Active/(Initial ERAB Setup Success+ Additional ERAB Setup Success)”
•
Voice over LTE Drop Rate (VDR) - This is to evaluate call drops during Volte call. “E-RAB Release whilst Active*/(Initial ERAB Setup Success*+ Additional ERAB Setup Success*)” *Use values from only QCI1
•
Handover Success Rate (HOSR) – This is to evaluate mobility performance of the network. “Inter eNB handover Success/Inter eNB handover Attempt”
•
Throughput DL/UL - Cell level Data speed in DL and UL “DL Throughput_HSI =(('DlMacTrafficBytes.9'[eCell,Time]*((8.0*0.743)*1000.0))/(('DlActTti.9'[eCell,Time]*1024.0))” “UL Throughput_HSI = ((('UlMacTrafficBytes.9'[eCell,Time]*(8.0*0.2*1000))/(('UlActTti.9'[eCell,Time])*1024.0)”
Copyright 2016 AIRSPAN NETWORKS INC.
2
RRC Success Rate Troubleshooting 1. PCI & RSI Planning Validation- Refer Basic Parameter Planning Guideline * 1. *If Auto PCI and Auto RSI is enabled (Check SON Profile from Netspan) then ignore this step. 2. Clock Issue: Refer “Alarm Document” for clock related issues. 1. If No clock alarms but RRC Success Rate < 50% then swap L2 Switch & monitor the performance 2. If multiple cells are having poor RRC Success Rate then try CSS reboot. 3. Congestion : If “Max. Connected Users” > 30 then 1. Check if neighbouring cells are OOS - resolve it and monitor the performance. 2. Check # connected users at NBR cells, if possible increase the power of NBR cell & monitor the performance. 3. If SMCs are not planned tightly at affected floor then add small cell for traffic offload and monitor the performance. 4. If SMCs are tightly planned then do re-design the floor with additional capacity and get validated from NPE team . 4. Planning and design validation 1. Check if eNBs placed in close proximity is causing SMC to SMC interference. 2. Physical optimization if installed locations are different than plan 3. If above two conditions are met then re-design. 5. Walk Test: Do CVT/Walk to validate coverage gap 1. Increasing in eNB power can help to improve RRC success rate. 2. If power is at maximum, plan additional small cells. 3. if the outdoor signal strength is high and causing interference then increase SMC powe rif not improved, coordinate with Samsung Optimization team to do physical optimization Copyright 2016 AIRSPAN NETWORKS INC.
3
RRC Success Rate Flow Chart
Copyright 2016 AIRSPAN NETWORKS INC.
4
HOSR Troubleshooting 1. PCI & RSI Planning Validation- Refer Basic Parameter Guideline * 1. *If Auto PCI and Auto RSI is enabled (Check SON Profile from Netspan) then ignore this step. 2. Clock Issue: Refer “Alarm Document” for clock related issues 1. If No clock alarms but RRC Success Rate < 50% then swap L2 Switch & monitor the performance 2. If multiple cells are having poor RRC Success Rate then try CSS reboot. 3. Audit NRT table – 1. Validate SMC installation plan along with surrounding Macro locations. 2. Check if all possible neighbours are reported in NRT and their X2 links are setup to form required NBR relationship. 3. Check stats to find out specific NBR relationship causing handover failures. 4. Neighbour cell Availability- Check if neighbouring cells are OOS and causing handover failure because of coverage gaps or poor coverage areas generated. 5. Neighbour cell Accessibility – Check if the neighbouring cells are having Accessibility issues, Refer to RRC Success Rate troubleshooting module. 6. Walk Test A. Pingpong HO 1. To identify Ping-Pong Handover, do walk test and check if Handovers are happening back n forth at same locations. 2. If both SMCs are strong (RSRP) then 1. Reduce power or 2. Physical optimization required. 3. If Macro interference causing this then increase power to make SMC dominant. Copyright 2016 AIRSPAN NETWORKS INC.
5
HOSR Troubleshooting 4. else coordinate with Samsung Optimization to optimize the macro coverage B. Coverage Gaps 1. Do walk test to identify if Handover failures are happening in poor coverage and bad SINR 2. Increase eNB power to coverage and check HOSR. 3. If power is at maximum, additional small cell planning is required.
Copyright 2016 AIRSPAN NETWORKS INC.
6
HOSR Flow Chart
Copyright 2016 AIRSPAN NETWORKS INC.
7
CDR & VDR Troubleshooting 1. PCI & RSI Planning Validation- Refer Basic Parameter Guideline * 1. *If Auto PCI and Auto RSI is enabled (Check SON Profile from Netspan) then ignore this step. 2. Check HO Performance - Need to check if HO failures are increased. 1. Refer HO Success Rate module 3. Planning and design validation 1. Check if eNBs placed in close proximity is causing SMC to SMC interference. 2. Physical optimization if installed locations are different than plan 3. Else do redesign and NPE will validate. 4. Walk Test: Do CVT/Walk to validate coverage gap 1. Increasing in eNB power can help to improve CDR & VDR. 2. If power is at maximum, plan additional small cells. Raise to NPE for redesign
Copyright 2016 AIRSPAN NETWORKS INC.
8
CDR & VDR Flow Chart
Copyright 2016 AIRSPAN NETWORKS INC.
9
Throughput DL/UL Troubleshooting 1. PCI Planning Validation- Refer Basic Parameter Planning Guideline * 1. *If Auto PCI is enabled (Check SON Profile from Netspan) then ignore this step. 2. Congestion : If “Max. Connected Users” > 30 or “DL PRB %” > 75% & “UL PRB %” > 75% then 1. Check if neighbouring cells are OOS - resolve it and monitor the performance. 2. Check # connected users at NBR cells, if possible increase the power of NBR cell & monitor the performance. 3. If SMCs are not planned tightly at affected floor then add small cell for traffic offload and monitor the performance. 4. If SMCs are tightly planned then raise to NPE/NOC team for re-design the floor with additional capacity. 3. Backhaul Testing: Run Iperf test with maximum data load. 1. If throughput lower than data sent then raise issue to Backhaul team 4. Planning and Design validation 1. Check if eNBs placed in close proximity is causeing SMC to SMC interference. 2. Physical optimization if installed locations are different than plan 3. Else raise to NPE/NOC team for re-design. 5. Walk Test: Do CVT/Walk to validate coverage gap 1. Increasing in eNB power can help to improve Throughput DL/UL. 2. If power is at maximum, plan additional small cells.
Copyright 2016 AIRSPAN NETWORKS INC.
10
Throughput Flow Chart
Copyright 2016 AIRSPAN NETWORKS INC.
11
Alarm & Troubleshooting
Small Cell Alarm List Sr. No Alarm ID
1
100
Alarm Description
Node Lost Comms
Severity
Critical
Probable Cause
Action
Power Failure / Backhaul Issue
1. Check Reachability through CSS 2. Check power availability on site 3. Check physical backhaul issue 1. Check clock status 2. Make sure MME connectivity is OK 3. Check for any PCI Allocation Failure alarm
2
557
Channel Out of Service Critical
Clock Issue / MME not reachable
3
556
MME connectivity lost Major
Loss of the S1AP SCTP connection to 1. Verify the extent of the alarms on the nearby nodes a specific MME 2. Contact NOC for further advise
4
580
SGW connectivity lost Major
SGW is not reachable
1. Verify the extent of the alarms on the nearby nodes 2. Contact NOC for further advise
5
565
Auto PCI Allocation fail Major
No PCI resource available in the PCI pool
Report to NOC for further advise
6
587
Wi-Fi Board Connectivity
Major
Wi-Fi board is not active, Wi-Fi software version is not updated
Report to NOC for further advise
7
579
eSON Server connection Lost
Major
Server unreachable
Report to NOC
Copyright 2016 AIRSPAN NETWORKS INC.
13
Small Cell Alarm List – PTP / GPS Sr. Alarm No ID
Alarm Description
1
572 Node Loss of Sync
2
570
Severity Critical
Node PTP Grand Master Major Lost
Probable Cause The node does not have a valid sync source and remain in the OOS state
1. For ISC, check the GM connectivity 2. For OSC, check GPS connectivity
GM unreachable
1. Check status in Netspan connection to GM 2. Report to IP team to check connectivity to the GM
PTP Grand Master Lost PTP Degraded Stream
1. Verify additional alarms involving GPS or GM. 2. Rectify physical issues for GPS alarms 3. Report to IP team for GM-related alarms
3
571 Node in Holdover
Major
6
599 PTP Degraded Stream
Warning High PDV
4
5
213 GPS SNR Low
224
GPS Signal Unacceptable
Minor
Minor
Action
1. Check additional alarms and events in Netspan 2. Report to NOC.
Skyview partially obstructed GPS antenna faulty GPS antenna cable faulty
1. Check onsite if GPS location has clear skyview 2. Check connectors, they should free from moist 3. Check gps cable on both ends make sure properly tighten 4. Try to swap new gps antenna
No satellite detected
1. Check onsite if GPS location has clear skyview 2. Check connectors, they should free from moist 3. Check gps cable on both ends make sure properly tighten 4. Try to swap new gps antenna
Copyright 2016 AIRSPAN NETWORKS INC.
14
Node Lost Comms Description – The node is not reachable from Netspan
Yes
Probable Cause : Power / IP / Backhaul issue Impact : Services will be suspended , No Ping possible.
Yes
Actions to be taken :
1. Management VLAN (IP) Issue a.
Request IP team to check reachability from CSS to Netspan
b.
If not reachable check Netspan route in CSS
c.
If Netspan is reachable from CSS, check reachability from CSS to eNB
d.
Check reachability of L2 SW from CSS
Check power on the site
b.
LED should be ON in the node
No
Check eNB Reachabi lity
Check Routes in CSS
No Check Power
No
Report to concern team
Yes Check Physical connecti vity
2. Power Failure: a.
From CSS Check Netspan Reachabil ity
Yes
Take necessary actions
No
3. Backhaul Issue
Report to NOC team
a.
Check port status on L2 (Up / Down)
b.
Check the physical connections between CSS to L2 Switch and L2 Switch to SMC e-NB. (Loose connection check )
c.
Clean SFPs used at L2 Switch and SMC e-NB. (Dust /Moisture check )
d.
Check fiber/CPRI cable (if damaged, bent) Copyright 2016 AIRSPAN NETWORKS INC.
15
Channel OOS Description – The node is manageable but RF is not radiating Probable Cause : Clock Issue, MME connectivity, PCI allocation failure Impact : Services will be suspended
No
Actions to be taken : 1 . Clock Issue
a. In Netspan, clock status should be connected & Locked
Check MME Alarm
Yes Check PTP Alarms
Report to IP Team
Yes
b. If it is not Locked, check Grand Master (ISC) or GPS (OSC) connectivity or refer to PTP & GPS alarm chart
No
2. MME Connectivity Issue a. Check MME Alarm (Details in MME Connectivity Lost Alarm)
Check PCI failure Events
3. PCI Allocation Failure a. Check PCI alarms / events (Details in PCI Allocation failure) b. If alarm is present, possible no PCI resources available then report to SON team
Copyright 2016 AIRSPAN NETWORKS INC.
Report to NOC team
16
MME Connectivity Lost Description – Loss of the S1AP SCTP connection to a specific MME Probable Cause : MME is not reachable Impact : If alarm present on all defined MMEs then service affecting (Channel OOS)
Actions to be taken : 1. Verify other nodes at the same site with the same MME connection if it has the same alarm a. If it has same alarm report to IP team and check MME status b. Otherwise, report to IP team to check:
i.
Check 602 vlan tagging on L2 switch
ii.
Check 602 gateway reachability with source as eNb on CSS
2. If everything is OK contact NOC team
Copyright 2016 AIRSPAN NETWORKS INC.
17
SGW Connectivity Loss Description – Loss of S1 connection to the SGW Probable Cause : SGW is not reachable Impact : If alarm present on all defined SGWs then service affecting
Actions to be taken : 1. Verify other nodes with the same SGW connection if it has the same alarm a. If it has same alarm report to IP team and check SGW status b. Otherwise, report to IP team to check :
i.
Check 601 vlan tagging on L2 switch
ii.
Check 601 gateway reachability with source as eNb on CSS
2. If everything is OK contact NOC team
Copyright 2016 AIRSPAN NETWORKS INC.
18
Auto PCI Allocation Fail Alarm Description – Automatic allocation of the PCI failed Probable Cause : No PCI resource available in the PCI pool ( Channel OOS) Impact : Services will be affected Actions to be taken : 1. Report to NOC for further advise
Copyright 2016 AIRSPAN NETWORKS INC.
19
Wi-Fi Board Connectivity Alarm Description – The unit is not able to communicate with the Wi-Fi board (applies to AirVelocity with Wi-Fi only) Probable Cause : Wi-Fi board is not active, Wi-Fi software version is not updated Impact : Wi-Fi services will be affected
Actions to be taken : 1. Report to NOC for further advise
Copyright 2016 AIRSPAN NETWORKS INC.
20
eSON Server Connection Lost Description – Connection with eSON server is lost Probable Cause : eSON server unreachable Impact : Not service affecting but some SON performance will be affected
Actions to be taken : 1. Check Netspan status 2. Report to IP team to check connection to eSON server
Copyright 2016 AIRSPAN NETWORKS INC.
21
PTP and GPS Alarm Chart
GPS SNR LOW
PTP Degraded stream High PDV (Event)
2.5 Min
Check Sky, antenna no obstruction
GPS Signal Unacceptable
Node in Holdover
Node in Holdover
Node PTP Grand Master Lost
8 Hrs
Node Loss Sync
Node Loss Sync
Channel OOS
Copyright 2016 AIRSPAN NETWORKS INC.
22
Node Loss of Sync Description – The maximum holdover time has expired, node has no valid sync source Probable Cause : GM is not reachable or GPS connectivity issue Impact : Services will be suspended (Channel OOS) Actions to be taken : 1 . For ISC, check the GM connectivity a.
In Netspan, clock status should be connected
b.
If it is not Locked, check Grand Master or refer to PTP & GPS alarm chart
c.
Report to IP team
2. For OSC, check GPS connectivity a.
In Netspan, clock status should be locked
b.
Ensure GPS is well installed having sky clearance
c.
Check physical connectivity between eNB and GPS antenna (Moist, connector)
d.
If problem persist then try to replace GPS antenna
Copyright 2016 AIRSPAN NETWORKS INC.
23
Node PTP Grand Master Lost Description – Node has no valid 1588 clock source Probable Cause : GM not reachable Impact : Service will be suspended (Channel OOS) Actions to be taken : 1. Check status in Netspan connection to GM
2. Report to IP team to check connectivity to the GM.
Copyright 2016 AIRSPAN NETWORKS INC.
24
Node in Holdover Description – Node has no valid clock source Probable Cause : GM not reachable, High PDV Impact : May lead to Holdover expire and probable service-affecting
Actions to be taken : 1. Verify additional alarms in Netspan like GPS SNR Low and Node PTP Grand Master Lost and events like High PDV 2. Report to IP team to check connectivity to the GM. 3. If GPS problem, refer to the GPS SNR Low action item.
Copyright 2016 AIRSPAN NETWORKS INC.
25
PTP Degraded Stream Description – PDV conditions in the 1588 PTP stream have changed Probable Cause : PDV reaches the threshold level Impact: May lead to Holdover and probable service-affecting
Actions to be taken : 1. Check additional alarms and events in Netspan 2. Report to NOC.
Copyright 2016 AIRSPAN NETWORKS INC.
26
GPS SNR Low Description – Low SNR of the GPS signal Probable Cause : Sky view partially obstructed ,GPS antenna faulty , GPS antenna cable faulty Impact : May lead to GPS lock loss Actions to be taken : 1. Check onsite if GPS location has clear skyview 2. Check connectors, they should free from moist 3. Check gps cable on both ends make sure properly tighten
4. Try to swap new gps antenna
Copyright 2016 AIRSPAN NETWORKS INC.
27
GPS Signal Unacceptable Description – Satellite not detected Probable Cause : Sky view partially obstructed ,GPS antenna faulty , GPS antenna cable faulty Impact : May lead to GPS lock loss Actions to be taken : 1. Check onsite if GPS location has clear sky view 2. Check connectors, they should free from moist 3. Check GPS cable on both ends make sure properly terminated
4. Try to swap new gps antenna
Copyright 2016 AIRSPAN NETWORKS INC.
28
THANK YOU
View more...
Comments
