ACME Checks and Learnings_2

ACME Checks and Learning

24th January 2016

Table of contents              

ACME Test Scenario Network Architecture VOLTE Call Flow Multiband Settings and Learnings Neighbours Deletion eNode B Prechecks True Call Based Analysis CSL Based X2 Analysis GTP Loss X2 Learnings S1AP – Critical for Uptime S1AP Preservation S5 Interface Learnings Core Network Learnings

ACME TEST Scenarios V to V

•VOLTE to VOLTE

V to C

•VOLTE to CS

C to V

•CS to VOLTE

V to P

•VOLTE to PSTN

 Around 300 to 400 Calls for each scenario  PASS CRITERIA AS MINIMUM 98% SUCCESS  Call Set Up and Call Drops added together to be less then 2% 2016 © Samsung Electronics

3

ENB – Interface and Protocol The Protocols that run between UE and eNB are called Access Stratum Protocols or AS Protocols The Protocols that run between UE and Core Network are called as NAS Protocols or Non Access Stratum Protocols

2016 © Samsung Electronics

4

MME – Interface and Protocol

2016 © Samsung Electronics

5

MME – Interface and Protocol

2016 © Samsung Electronics

6

SAE GW – Interface and Protocol

2016 © Samsung Electronics

7

SAE GW – Interface and Protocol

• • • • •

Ip Packet is tunneled between P GW and UE using GPRS Tunneling Protocol GTP Each Bearer has associated QCI Each QCI has GBR/ Non GBR, Priority, packet delay , Packet Error Rate defined PCRF decides the Bearer Type P GW performs the DHCP functionality = allocates IP addresses to each UE

2016 © Samsung Electronics

8

VoLTE Call Flow

9

VoLTE – Logical Architecture ENUM – Electronic Number Mapping System To translate a telephone number into IP address

2016 © Samsung Electronics

10

VoLTE – Logical Architecture

2016 © Samsung Electronics

11

VoLTE - UE Attachment and IMS Registration message

CCR Credit Control Request CCA Credit Control Answer 2016 © Samsung Electronics

Part 1 of 2 12

VoLTE - UE Attachment and IMS Registration message

Part 2 of 2 2016 © Samsung Electronics

13

VoLTE - UE to UE Call establishment (Originating Side)

2016 © Samsung Electronics

14

VoLTE - UE to UE Call establishment (Terminating Side)

2016 © Samsung Electronics

15

VoLTE - UE to UE Call clearing (Initiated message)

2016 © Samsung Electronics

16

VoLTE - UE to UE Call clearing (Received message )

2016 © Samsung Electronics

17

VoLTE - VoLTE UE to CS call establishment (Originating Side )

2016 © Samsung Electronics

18

VoLTE - VoLTE UE to CS call establishment (Originating Side )

2016 © Samsung Electronics

19

VoLTE - VoLTE UE to CS call establishment (Terminating

2016 © Samsung Electronics

Side )

20

VoLTE - VoLTE UE to CS call Clearing (Initiated )

2016 © Samsung Electronics

21

VoLTE - VoLTE UE to CS call Clearing (Received )

2016 © Samsung Electronics

22

Multiband Settings & Learnings

2016 © Samsung Electronics

23

Idle Mode (2300 and 1800) Service Target Idle Mode

2.3G

Parameter

2.3G Priority QrxLevMin S_NonIntraSearch Thresh_Serving Low

1.8G 1.8G

7 64 -128 12 -104 dBm 12 -104 dBm

ThrehX_Low ThrehX_High

1.8G

2.3G

6 64 -128 12 -104 dBm 12 -104 dBm

12 -104 dBm 14 -100 dBm

(-Q_Rxlevmin + 2 * Thresh_Serv_Low) dBm 1. 2. 3. 4. 5.

Equal priority Band and higher priority Bands, are always measured UE on 1.8 G will always measure 2.3G The UE starts measuring other lower priority band neighbors when Serving Cell < S_nonintrasearch UE on 2.3G measures 1.8G when 2.3G is lower than -104 dBm UE does cell reselection from 2.3G to 1.8G when 2.3G is lower than -104 dBm and 1.8G is better than -104 dBm 6. UE returns to 2.3G when 2.3G is better than -100 dBm 2016 © Samsung Electronics

24

Idle Mode Parameters Behavior -75dBm

-80dBm

-85dBm

-90dBm

-95dBm

-100dBm

-105dBm

-110dBm

-115dBm

-70dBm

-75dBm

-80dBm

-85dBm

-90dBm

-95dBm

-100dBm

-105dBm

Band 40 (2300MHz) -65dBm

Band 3 (1800MHz)

•

The UE starts measuring other lower priority band neighbors when Serving Cell < S_nonintrasearch.

•

Equal priority band And higher priority Bands, are always measured.

1. For Cell Reselection from 2300 to 1800 happens when 2300 is weaker than Thresh_Serving_Low and 1800 is stronger than Thresh_Xlow (higher to lower priority) 2. For Cell Reselection from 1800 to 2300 happens when 2300 is stronger than ThreshX_High (lower to higher priority)

Example Idle Mode

Band 3 (1800MHz)

Band 40 (2300MHz)

2300 has higher priority than 1800 -104 dBm 2016 © Samsung Electronics

25

Handover : Event in Active Mode  Event Type  Intra RAT (Intra FA / Inter FA) : Event A

—Event A1 (Serving becomes better than threshold) —Event A2 (Serving becomes worse than threshold) —Event A3 (Neighbor becomes offset better than serving) —Event A4 (Neighbor becomes better than threshold) —Event A5 (Serving becomes worse than threshold1 and neighbor becomes better than threshold2)

2016 © Samsung Electronics

26

LTE Handover Events  Event A1 Serving becomes better than threshold RSRP TimetoTrigger Thresh

Hys

 Event A2

Hys RSRP Serving Cell Serving Cell Leave A1 Condition Enter A1 Condition

Time Enter A1 Condition

Event A1

Hys Thresh Hys

TimetoTrigger

Serving becomes worse than threshold Time Enter A2 Condition

2016 © Samsung Electronics

Enter A2 Condition Leave A2 Condition

Event A2

27

LTE Handover Events  Event A3 

Neighbor becomes offset better than serving

RSRP/RSRQ

Serving Cell Neighbor Cell TimetoTrigger

A3 Offset Hys Hys

Hys

Time Enter A3 Condition When Ofn, Ocn, Hys, Ofs, Ocs, Off are all zero

2016 © Samsung Electronics

Event A3 Leaving A3 Condition When Ofn, Ocn, Hys, Ofs, Ocs, Off are all zero

Enter A3 Condition When Ofn, Ocn, Hys, Ofs, Ocs, Off are all zero

28

LTE Handover Events  Event A5  Serving becomes worse than threshold1 and neighbor becomes better than threshold2 RSRP/RSRQ

Neighbour Cell Stronger than Threshold 2

TimetoTrigger Serving Cell

Thresh2 Thresh1

Hys Hys Hys Hys

Enter Condition1

Leave Condition1 Enter Condition2 Enter Condition1

2016 © Samsung Electronics

Time

Event A5

Serving Cell weaker than Threshold 1

29

-140 dBm + A2

Active Mode 2300-1800 Service Target GBR Service (VoLTE)

2.3G QCI based A2 Threshold QCI based A1 threshold (Meas Gap Deactivation) QCI based A3 threshold

Non-GBR Service (FTP DL)

2.3G

Parameter

A2 Threshold A1 threshold(Meas Gap Deactivation) A3 Threshold A5 Threshold (Th1) A5 Threshold (Th2)

1.8G 1.8G

1.8G

35 -105 dBm

35 -105 dBm

-90 dBm

-90 dBm

6 3 dB

6 3 dB

6 3 dB

27

80

-113 dBm

-60 dBm

-100 dBm

-50 dBm

6 3 dB

6 3 dB

-113 dBm -110 dBm

2.3G

6 3 dB

-60 dBm -110 dBm

1. For GBR for transition from 2.3G to 1.8G, as 2.3G becomes than weaker than -105 for GBR, UE starts measuring other band and transitions when 1.8G is stronger than 2.3G by A3 Offset which is 3 dBm 2. For Non GBR - A5 for transition from 1.8G to 2.3G (as 2.3G would be weaker than 1.8G in most cases ) i.e. when serving cell in 1.8G is weaker than -60 dBm and the target cell in 2.3G is better than -110 dBm 3. A2 and A3 are used for GBR for intra & inter band handover and for intra band handover for non GBR 4. A5 is used for inter band handover for non GBR

2016 © Samsung Electronics

30

Active Mode Parameters Behavior -75dBm

-80dBm

-85dBm

-90dBm

-95dBm

-100dBm

-105dBm

-110dBm

-115dBm

-70dBm

-75dBm

-80dBm

-85dBm

-90dBm

-95dBm

-100dBm

-105dBm

Band 40 (2300MHz) -65dBm

Band 3 (1800MHz)

For GBR - For transition from 2300 to 1800, as 2300 becomes than weaker than A2 threshold, UE starts measuring other band and transitions when 1800 is stronger than 2300 by A3 Offset (which would always happen except for Micro cells)

For Non GBR - A2 and A3 for transition intra band A5 for transition inter band – especially from 1800 to 2300 (as 2300 would be weaker than 1800 in most cases )

-113 dBm Example Data Mode

Band 40 (2300MHz)

Band 3 (1800MHz)

VoLTE Mode 2016 © Samsung Electronics

-105 dBm

31

Expected Relative Behavior in Connected Mode Band 40 (2.3Ghz)

-90dBm

-95dBm

-100dBm

-105dBm

-110dBm

-113dBm

-120dBm

-125dBm

-130dBm

-80Bm

-85dBm

-90dBm

-95dBm

-100dBm

-103dBm

-110dBm

-115dBm

-120dBm

Band 3 (1.8GHz) Continues Data Services

Call Start

Data Service Data Service

Data Services to VoLTE Before -113dBm Data Service

VoLTE Call Start

VoLTE call end Before –110 dBm

Data Service

Data Service

Data Service

VoLTE Call END

Data Services to VoLTE BEFORE -113dBm Data Service

VoLTE Call Start

Data Service

VoLTE call end After –110 dBm

VoLTE Call END

Data Services to VoLTE After -113dBm Data Service VoLTE call end After –110 dBm 2016 © Samsung Electronics

Data Service

Data Service Data Service

VoLTE Call Start

VoLTE Call END

Data Service 32

Bhopal Learning : Why it Happened ? A1 should always be higher than A2

Overview: 

Parameter Template 1: Configuration with Multi-band Implementation



Parameter Template 2: Default Parameter implementation during Golden Parameter audit & corrections



Audit & corrections carried out on 24-Oct over-ride A2 value earlier implemented as a part of Multi-band Parameter list

Technical Analysis: Default Parameter List (Used in Work Stream 1) A1: Not Set

vs

Multi-band Services Implementation (Used in Work Stream 2)

Remarks

A1: -100dBm ( value: 40 )

If RSRP is better than A1, it will stop all other measurements

A2: -90dBm ( value: 50 )

A2: -113dBm ( value: 27 )

If RSRP is lower than A2, UE has to start measurement of neighbor band

Result:  A2 got configured as -90dBm in place of -113dBm, with A1 as -100dBm 

When UE RSRP goes less than -90dBm (A2), inter-band measurements will trigger.



However since RSRP is better than A1 ( -100dBm ), UE will be instructed to stop the inter-band measurements.

 Thus conflicting settings for A1 & A2, leads to high messaging & thus eNB going into Racing condition. 2016 © Samsung Electronics

33

Learnings – from incorrect EARFCN settings CELL_NUM 0 0

FA_INDEX 0 1

DUPLEX_TYPE TDD ( TDD ) FDD ( TDD )

1 1

0 1

TDD ( TDD ) FDD ( TDD )

2 2

0 1

TDD ( TDD ) FDD ( TDD )

3 3

0 1

FDD ( FDD ) TDD ( FDD )

4 4

0 1

FDD ( FDD ) TDD ( FDD )

5 5

0 1

FDD ( FDD ) TDD ( FDD )

STATUS EQUIP ( EQUIP EQUIP ( N_EQUIP ) EQUIP ( EQUIP EQUIP ( N_EQUIP ) EQUIP ( EQUIP EQUIP ( N_EQUIP ) EQUIP ( EQUIP EQUIP ( N_EQUIP ) EQUIP ( EQUIP EQUIP ( N_EQUIP ) EQUIP ( EQUIP EQUIP ( N_EQUIP )

EARFCN_UL 38800 ( 38800 ) 19294 ( 38800 )

EARFCN_DL 38800 ( 38800 ) 1294 ( 38800 )

PRIORITY 7(7) 6(7)

)

38800 ( 38800 ) 19294 ( 38800 )

38800 ( 38800 ) 1294 ( 38800 )

7(7) 6(7)

)

38800 ( 38800 ) 19294 ( 38800 )

38800 ( 38800 ) 1294 ( 38800 )

7(7) 6(7)

)

19294 ( 19294 ) 38800 ( 19356 )

1294 ( 1294 ) 38800 ( 1356 )

6(6) 7(7)

)

19294 ( 19294 ) 38800 ( 19356 )

1294 ( 1294 ) 38800 ( 1356 )

6(6) 7(7)

)

19294 ( 19294 ) 38800 ( 19356 )

1294 ( 1294 ) 38800 ( 1356 )

6(6) 7(7)

)

EARFCN settings for eNode B and for Multiband should be the same Else Handovers would get impacted

2016 © Samsung Electronics

34

Active & Idle Mode – All Three Bands Service

Parameter

2.3G

Target GBR Service (VoLTE)

2.3G 1.8G

850

1.8G

1.8G 2.3G

850

850

35

35

35

-105 dBm

-105 dBm

-105 dBm

-90 dBm

-90 dBm

-90 dBm

850M 2.3G

1.8G

QCI based A2 Threshold QCI based A1 threshold (Meas Gap Deactivation)

6

6

6

6

6

6

6

6

6

3 dB

3 dB

15+5 dB

3 dB

3 dB

3 dB

3 dB

3 dB

3 dB

QCI based A3 threshold Non-GBR Service (FTP DL)

27

80

80

-113 dBm

-60 dBm

-60 dBm

-100 dBm

-50 dBm

-50 dBm

6

6

6

3 dB

3 dB

3 dB

A2 Threshold A1 threshold(Meas Gap Deactivation) A3 Threshold

Service Target Idle Mode

A5 Threshold (Th1)

-113 dBm

-113 dBm

-60 dBm

-113 dBm

-60 dBm

-113 dBm

A5 Threshold (Th2)

-110 dBm

-93 dBm

-110 dBm

-110 dBm

-110 dBm

-110 dBm

2.3G 1.8G

850

1.8G 2.3G

850

850M 2.3G

1.8G

14 -100 dBm

14 -100 dBm

Parameter

2.3G Priority QrxLevMin S_NonIntraSearch

7

6

ThrehX_Low

ThrehX_High 2016 © Samsung Electronics

850 5

64

64

64

-128

-128

-128

12

12 -104 dBm 12 -104 dBm

12 -104 dBm 12 -104 dBm

-104 dBm 12

Thresh_Serving Low

1.8G

-104 dBm

12 -104 dBm

12 -104 dBm

14 -104 dBm 14 -100 dBm

35

Neighbor Deletion Analysis

36

Neighbor Deletion Based on HO Success  Neighbor Table





Auto build by LSMR and UE based . Max NBR in table are 256



Key parameter are Neighbor cell PCI (Unique/Band), Cell ID , TAC, eNB ID

Wrong Neighbor impacting HO performance degradation

2016 © Samsung Electronics

37

HO Performance Improvement  Radio Interface Failure are mainly happened by Poor RF after/Before HO Failures.

 HO Failure Cases and Resolution 

Wrong Neighbor Relation 2.UE Failed HO due to wrong PRACH Sequence 1.HO Preparation done due to wrong neighbor

UE Real Target Cell PCI : “A”

Source cell Very far and signal is not detected

Wrong target cell, PCI : “A”

— Needs to turn on HO Success Rate Based NBR Deletion — The feature deletes Neighbor that shows HO failure count greater than “Th1” if total HO Attempt is greather than “Th2”

 PCI Confusion Case1

Neighbor Cell PCI : “A”

Source cell

Case2

Neighbor Cell PCI : “A”

Source cell



Neighbor Cell PCI : “A”

Very far but signal is detected

PCI Reallocation  SON PCI needs to be ON Neighbor Cell PCI : “A” Overshooting, RF Optimization needed

HO Command Reception Failure due to Poor RF — Too late HO, Needs to optimize RF condition

2016 © Samsung Electronics

38

Neighbor Relation Table - Optimization  Methods for Wrong Neighbor deletions  MR Based This feature removes Neighbor Relation (NR) which cannot receive Measurement Report (MR) messages among the NRs included when the network was initially created, so that only valid NRs could be included in the Neighbor Table. Deployed at Ahemdabad, Ujjain and Hyderabad

 Neighbor Deletion based on HO Success This feature removes Neighbor Relation based on HO Success Rate Deployed in network  Neighbor Deletion based on NBRDEL-CAUSE This feature removes Neighbor Relation based on 2 cause reported Cell not available & Unknown Target ID Deployed in network  Manual Neighbor Deletion This is based on troubleshooting HO call failure observed in CSL/Truecall.

Not Applicable

2016 © Samsung Electronics

39

Neighbor Deletion based on MR: Ahmedabad Plan Day

eNB Configuration Parameter

Objective

PCI Conflict Notification

nrDelFlag = True (Enable Feature)

Day 1

MR Based Neighbor Deletion

thNumMrNrDel = 3 (Count of MR)

KPI & Performance Indicator Checks

Total NBR Relation to reduce after Aggressive MR based Deletion

thTimeNrDel = 1 (Days)

NBR Count RRC Connection Setup Success Rate, ERAB Success Rate, Attach Success Rate Call Drop Rate, RRC Connection Re-establishment Attempts and Success Rate Handover Preparation Success Rate, Handover Completion Success Rate PCI Conflict Notification

thNumMrNrDel = 5 (Count of MR) Day 3

MR Based Neighbor Deletion thTimeNrDel = 3 (Days)

NBR Count Change in MR Thresold to reduce NBR deletion count.

RRC Connection Setup Success Rate, ERAB Success Rate, Attach Success Rate Call Drop Rate, RRC Connection Re-establishment Attempts and Success Rate

Handover Preparation Success Rate, Handover Completion Success Rate

2016 © Samsung Electronics

40

Neighbor Deletion based on MR: Ahmedabad  MR Based Deletion  Enabled MR based NR deletion on 13th Jan . Before enabling this feature the total NRs in Ahmedabad city are 407192. on 14th Jan the MR based deletion algorithm deleted the most of the NBRs and the count on 15th Dec is 44K.

2016 © Samsung Electronics

Date

NBR Addition

NBR deletion

1/12/2016

970

61

1/13/2016

1152

84

1/14/2016

28169

379783

1/15/2016

1187

12665 41

Handover Statistics

2016 © Samsung Electronics

42

UE Based ANR  Overall Call Flow

#UE reports unknown PCI #eNB orders UE to report eCGI by sending “RRCConnectionReconfig”. Point 1) Samsung eNB does not trigger report CGI process for those UE with GBR bearer.  VoLTE UE does not report eCGI

#In order for UE to get eCGI information, UE has to go to Idle mode with help of DRX mode or Meausurement GAP (only applicable for inter FA) to get eCGI information in SIB1 of the target cell. #Data Scheduling has higher priority than DRX mode by 3GPP Standard. Point2) If UE continuously receives or transmits traffic, UE cannot go to DRX mode. UE-based-ANR is triggered before LSM-based-ANR 2016 © Samsung Electronics

43

MR Based Neighbor Deletion  Feature Description This function removes Neighbor Relation (NR) which cannot receive Measurement Report (MR) messages among the NRs included when the network was initially created, so that only valid NRs could be included in the Neighbor Table.

 Feature Configuration nrDelFlag = True thNumMrNrDel = 5 in Ujjain (Range 0 to 1000K) thTimeNrDel = 1 in Ujjain ( Range 1 to 7 days)  Ujjain Observations Total eNB 84 , Neighbors 40,000 18Dec - Enabled MR based NR deletion 20Dec - MR based deletion algorithm deleted the most of the Neighbors. 24Dec – 3413 Neighbor created mostly by UE based ANR. Ujjain Analysis – MR Based Neighbor Deletion 100.00% 99.80%

35000 30000

25000 20000 15000 5000 0

2100 400 1200

3413

3083

InterX2HOSuccRate

Number Of NBRs

45000 40000 40000

10000

InterX2 HO Attempts

HO Performance Stats 26

24

InterX2 HO Failure

15

18 70

91

99.60%

103

99.40% 99.20% 99.00% 98.80% 7557

5627

7107

12110

12880

9315

11888

22-Dec-15

23-Dec-15

24-Dec-15

25-Dec-15

26-Dec-15

27-Dec-15

28-Dec-15

98.60%

2016 © Samsung Electronics

44

Neighbour Deletion based on HO Success Pre-Post Analysis 6 cities  In 6 cities analysis , 3K to 6K Neighbors having HO Attempt greater than 10 and very few neighbors having HO Success less than equal to 5. City

Total Neighbors Daily Average

Ahmedabad Bangalore Hyderabad Chennai Kolkata Bhopal

167225 380499 311600 191142 435667 92736

Neighbor Count HO Attempt Neighbors Count Attempt GT Greater than 10 10 & HO Success less than 5 Daily Average Daily Average 6600 (3.95%) 6000 (1.58%) 5500 (1.77%) 3200 (1.67%) 3700 (0.80%) 1250 (1.35%)

3 4 2 1.5 1.5 1

 Neighbor Deletion threshold of HO Attempt GT 10 and HO Success less than 5 are not met in most of the cases  1 to 4% of neighbors having HO Attempt greater than 10 indicates less users.  Next Step – Review the HO Attempts and HO Success every month for any optimization of threshold.

2016 © Samsung Electronics

45

eNode B Prechecks

46

Pre-requisites/ Checklist (RAN) 1.

2.

3.

4.

5.

6.

…..Contd.

SCFT: 100% completed and all sites to be radiating. • Run command “RTRV-CELL-STS” in all eNBs. • Cells with Status = DISABLE needs to be checked. No Service affecting alarm in eNB • Run command “RTRV-ALM-LIST” in all eNBs. • Service impacting alarms needs to be rectified. eNB on 5.0.0. – all patches to be loaded , SW & FW Audits • Go to Configuration -> Software -> Verify. Select all eNBs and verify the Software and Firmware. • eNBs with “NOK” status, should be noted and packges should be upgraded in planned event. • Please refer Software/Firmware Verification in attached pre-requisite file. OCNS to be removed • Run command “MON-TEST” in all eNBs. • eNBs with OCNS running should be noted. Run command “TERM-TEST” to remove OCNS, in all eNBs. • Please refer OCNS Check section for procedure in attached pre-requisite file. Multiband/Golden Parameter Settings • Go to Configuration -> Parameter Manager -> Select appropriate parameter planner.  Golden Parameter: pkg_500new  Multiband Parameter : Multi_Carrier_23n18_CircleName (For eNBs having 2300 MHz and 1800 MHz bands) Multi_Carrier_3band_CircleName (For eNBs having 2300 MHz and 1800 MHz bands) • Select all eNBs and compare the parameter manger. • eNBs with “NOK” status, should be noted and parameter planner should be applied in planned event. • Please refer Golden Parameter/Multiband Parameter Settings section for procedure in attached pre-requisite file. PCI SON Notifications to be resolved • Go to SON -> Log Dump Management. Select the time period and get the dump. • From dump, check source and target eNBs and solve the PCI collision if both neighbors are required. • Please refer PCI SON Notifications section for procedure in attached pre-requisite file.

2016 © Samsung Electronics

47

Pre-requisites/ Checklist (RAN) 7.

…..Contd.

Inactivity Timer 30 Sec, All Sites on “Same Power Option”. — Run command “RTRV-INACT-TIMER” for all eNBs. Internal USER_INACTIVITY_TIMER should be 30 for all QCI. — Run command :CHG-INACT-TIMER” for changing the USER_INACTIVITY_TIMER.

8.

Inter LSMR Neighbors to be added manually or using UE based ANR. — Inter LSMR required neighbor list would be provided by RF team. — Run command “RTRV-NBR-EUTRA” for provided cells. — In case, neighbor does not exist, it needs to be added using GUI. — Please refer Neighbour Addition/Deletion section for procedure in attached pre-requisite file.

9.

Inter Band Neighbors to be checked for 2300, 1800 & 850. — Run command “RTRV-NBR-EUTRA” for all cells and if inter band neighbor exists. — In case ,intra eNB inter band neighbor does not exist, neighbors need to be added using GUI. — Please refer Neighbour Addition/Deletion section for procedure in attached pre-requisite file.

10. PCI repetition within 2 km to be removed for all cases, Additional Audit for 4 km and first tier /second tier neighbor all PCI conflict cases to be plotted on map info and corrective actions identified. — Go to SON -> Log Dump Management. Select the time period and get the dump. — This dump needs to be provided to RF team, RF team will map conflict cases on map . — RF will recommend to change PCI of some sites. — PCI of sites can be changed using GUI. — Please refer PCI/EARFCN Change section for procedure in attached pre-requisite file.

2016 © Samsung Electronics

48

Pre-requisites/ Checklist (RAN) 11.

…..Contd.

Small Cells: Neighbor Definitions to be checked within Small cells (if in a cluster). — List neighboring sites of small cells and run “RTRV_NBR_EUTRA” on all cells. — If neighbors are found missing, add the neighbors using GUI. — Please refer Neighbour Addition/Deletion section for procedure in attached pre-requisite file.

12. RRH Log Analysis to ensure all sites radiating . — Run “RTRV-RRH-STS” for all eNBs. — Check if RRH are in operational state and NO OF FA =1. 13. EARFCN definitions to be audited for multiband operation for all elements (Macro & Small cells). — Run command “RTRV-CELL-IDLE” in all eNBs. — Check UL and DL EARFCN of all eNBs with EARFCN allocated to respective circle. — Please refer PCI/EARFCN Change section for procedure in attached pre-requisite file.

2016 © Samsung Electronics

49

TrueCall Based Analysis

50

True Call Report – Cell Wise RLF & X2 Fail (Sample) eNB ID RLF 1076 4147 2915 2166 2790 936 2918 318 1556 2919 4092 4242 1337 1418 1348

eNB ID X2 Fail 1055 1386 2166 2176 2791 1266 2016 © Samsung Electronics

Cell Name 5 13 109

16 14 1 147 78 4 24 40 38 98 5 2 4 21 3 25

17 13 11

123 5 13 1

5 23

3

16 3 14 7 3 266

17 56 11 80 104 19 21 56 70 19 79 15

18 112 96 8 35 124 86 53 2 8 61

30 4 5

24 27

35 22 3 66 24 41 12 29 1 28

4 9 7

17 1 168 76 447 5 43

18 1175 1108 18 67 26 50

36 20 4 2 26 1 48 13 114 28 22 125 2 7 3 50

Cell Name 35 1 4 980 19 2 32

37 104 76 10 16 61 13 17

48

15 14 23

49 12 39

1 3

9

7 4 1 55 1

36 328 6 11 5 3 21

1 21 17

37 17 22 25 37 20 3

9 20 18

48

Total

ACME Zone

353 339 313 283 268 235 233 225 213 174 146 134 127 125 124

New Mumbai JNPT kalyan New Mumbai JNPT New Mumbai JNPT New Mumbai JNPT New Mumbai JNPT New Mumbai JNPT New Mumbai JNPT kalyan New Mumbai JNPT thane central south New Mumbai JNPT kalyan west

49 1

2

469 4

1 1

Total

ACME Zone

1523 1336 1124 582 532 420

New Mumbai JNPT thane central New Mumbai JNPT New Mumbai JNPT kalyan New Mumbai JNPT 51

Abnormal Release Causes Abnormal Release Cause GTP Pa th Fa i l ure

Abnormal Release Cause

RLC - Inva l i d Pa ra meter

eNB - Ra di o Li nk Fa i l ure

eNB - Rel ea s e Due To eNB Genera ted Rea s on

S1AP - Reduce Loa d In Servi ng Cel l

MAC - Inva l i d Cel l Id

RRM - UE S-TMSI Dupl i ca te eNB Ti meout - Inter X2 Ha ndover Comma nd Compl ete eNB Ti meout - RRC Connection Setup Compl ete Mes s a ge Not Recei ved eNB Ti meout - RRC Connection Reconfi gura tion Compl ete Mes s a ge Not Recei ved eNB Ti meout - Intra Ha ndover Comma nd Compl ete

eNB - Rees tabl i s hment Fa i l ure Due to Inva l i d State S1AP - Ra di o Network Uns peci fi ed eNB Ti meout - X2 SN Status Tra ns fer eNB Ti meout - Inter S1 Ha ndover Comma nd Compl ete RRM - MME Overl oa d eNB Ti meout - S1Rel ocOvera l l Expi ry

eNB Ti meout - X2 Rel ocOvera l l Expi ry

eNB - Recei ved X2 Res et Reques t

eNB Ti meout - RRC Connection Rees tabl i s hment Compl ete Mes s a ge Not Recei ved

eNB Ti meout - Interna l Rees tabl i s h Control

eNB - ARQ Ma xi mum Retra ns mi s s i on

eNB Ti meout - RRC Securi ty Mode Compl ete Mes s a ge Not Recei ved

eNB - Ra di o Li nk Fa i l ure by RRC Connection Rees tabl i s hment

RLC - Ca l l Is Not Active

eNB - DSP Audi t RLC MAC Ca l l Rel ea s e

S1AP - Ca us e NAS Uns peci fi ed

S1AP - Authentica tion Fa i l ure

S1AP - HO Fa i l ure In Ta rget EPC eNB Or Ta rget Sys tem

eNB - ARQ Ma xi mum Retra ns mi s s i on by RRC Connection Rees tabl i s hment

eNB Ti meout - Interna l Da ta Tra ns fer Start

eNB - S1 SCTP Out of Servi ce

eNB Ti meout - S1 MME Status Tra ns fer Not Recei ved

X2AP - Ca us e Mi s c Uns peci fi ed

MAC - Inva l i d Pa ra meter

eNB Ti meout - RRC UE Ca pa bi l i ty Informa tion Mes s a ge Not Recei ved eNB Ti meout - S1 Ini tia l Context Setup Reques t eNB Ti meout - S1 Pa th Swi tch Reques t Ack Not Recei ved eNB - Recei ved Cel l Rel ea s e Indi ca tion from eNB

RRM - MME Not Ava i l a bl e for PLMN eNB Ti meout - Interna l Res ource Setup GTP Setup Fa i l ure PDCP - Inva l i d RB Id X2AP - Cel l Not Ava i l a bl e S1AP - Ra di o Connection Wi th UE Los t S1AP - Loa d Ba l a nci ng TAU Requi red S1AP - CS Fa l l ba ck Tri ggered eNB - Recei ved Res et Reques t From eNB RRM - CQI Res ource Fa i l ure GTP Recei ve Error Indi ca tion From GW X2AP - Trel ocprep Expi ry

2016 © Samsung Electronics

52

True Call – Sample Report Abnormal Release Abnormal Release Cause

Mumbai

12474 S1AP - Reduce Loa d In Servi ng Cel l 12151 RRM - UE S-TMSI Dupl i ca te 8547 eNB Ti meout - Inter X2 Ha ndover Comma nd Compl ete 5796 eNB Ti meout - RRC Connection Setup Compl ete Mes s a ge Not Recei ved 4367 eNB Ti meout - RRC Connection Reconfi gura tion Compl ete Mes s a ge Not Recei ved 4214 eNB Ti meout - Intra Ha ndover Comma nd Compl ete 3020 eNB Ti meout - X2 Rel ocOvera l l Expi ry 2613 eNB Ti meout - RRC Connection Rees tabl i s hment Compl ete Mes s a ge Not Recei ved1625 eNB - ARQ Ma xi mum Retra ns mi s s i on 729 eNB - Ra di o Li nk Fa i l ure by RRC Connection Rees tabl i s hment 628 eNB - DSP Audi t RLC MAC Ca l l Rel ea s e 477 S1AP - Authentica tion Fa i l ure 215 eNB - ARQ Ma xi mum Retra ns mi s s i on by RRC Connection Rees tabl i s hment 196 eNB - S1 SCTP Out of Servi ce 187 X2AP - Ca us e Mi s c Uns peci fi ed 164 eNB Ti meout - RRC UE Ca pa bi l i ty Informa tion Mes s a ge Not Recei ved 99 eNB Ti meout - S1 Ini tia l Context Setup Reques t 63 eNB Ti meout - S1 Pa th Swi tch Reques t Ack Not Recei ved 36 eNB - Recei ved Cel l Rel ea s e Indi ca tion from eNB 32 GTP Pa th Fa i l ure 30 eNB - Ra di o Li nk Fa i l ure

2016 © Samsung Electronics

21.57 21.01 14.78 10.02 7.55 7.29 5.22 4.52 2.81 1.26 1.09 0.83 0.37 0.34 0.32 0.28 0.17 0.11 0.06 0.06 0.05

Kolkata

8276 2235 5274 4117 2274 1408 2771 1248 2072 786 308 358 18 127 26 12 33 8 18 22 987

25.47 6.88 16.23 12.67 7.00 4.33 8.53 3.84 6.37 2.42 0.95 1.10 0.05 0.39 0.08 0.04 0.10 0.03 0.05 0.07 3.04

Delhi

6756 1242 8753 5809 4141 2053 2955 2728 2499 1050 283 440 55 162 417 77 43 114 65 32 127

16.88 3.10 21.86 14.51 10.34 5.13 7.38 6.81 6.24 2.62 0.71 1.10 0.14 0.41 1.04 0.19 0.11 0.28 0.16 0.08 0.32 53

Abnormal Release Trend True Call Implemented Only in Gujrat, Mumbai, Delhi, AP, Karnataka, Kolkata, MP

5.00 4.80 4.60 4.40 4.20 4.00 3.80 3.60 3.40 3.20 3.00

51931704

4.50

50000000

44683001 4.24

39154166

60000000

53543901

4.22

40000000

3.98

30000000

Release Count

% Abnormal Failure

Total Abnormal Release (%) (PAN India)

20000000 2289216

1980235

1843142

2218316

10000000 0

Dec Week3

Dec Week4 Total Release

Jan Week1 Abnormal Release

Jan Week2 % Failure

800000

Absolute Failures

700000 600000 500000 Dec Week3

400000

Dec Week4

300000

Jan Week1

200000

Jan Week2

100000 0 Gujrat

Mumbai

Delhi

AP

Karnataka

Kolkata

MP

Circles With True Call Implemented 2016 © Samsung Electronics

54

Abnormal Release Breakup (4th~10th Jan) Sr No.

Failure Cause

Mumbai

AP

Delhi

Gujrat

Karnataka Kolkata

MP

1

eNB - Radio Link Failure

21.59%

25.98%

16.90%

31.32%

20.15%

25.48%

27.68%

2

S1AP - Reduce Load In Serving Cell

21.03%

2.89%

3.11%

5.45%

6.74%

6.88%

5.94%

3

RRM - UE S-TMSI Duplicate

14.80%

15.81%

21.90%

17.40%

17.09%

16.24%

18.63%

4

eNB Timeout - Inter X2 Handover Command Complete

10.03%

9.71%

14.53%

6.77%

9.35%

12.68%

8.09%

5

eNB Timeout - RRC Connection Setup Complete Message Not Received

7.56%

10.90%

10.36%

7.04%

13.29%

7.00%

8.24%

6

eNB Timeout - RRC Connection Reconfiguration Complet e Message Not Received

7.30%

4.40%

5.14%

4.79%

5.14%

4.33%

4.83%

7

eNB Timeout - Intra Handover Command Complete

5.23%

3.76%

7.39%

4.04%

5.27%

8.53%

4.90%

8

eNB Timeout - X2 RelocOverall Expiry

4.52%

5.38%

6.83%

5.09%

5.07%

3.84%

6.26%

9

eNB Timeout - RRC Connection Reestablishment Complet e Message Not Received

2.81%

5.33%

6.25%

8.83%

7.14%

6.38%

6.83%

10

eNB - ARQ Maximum Retransmission

1.26%

1.81%

2.63%

2.00%

1.80%

2.42%

1.96%

11

X2AP - Cause Misc Unspecified

0.28%

7.93%

0.19%

0.66%

4.20%

0.04%

1.89%

12

GTP Path Failure

0.05%

1.41%

0.32%

0.17%

1.16%

3.04%

0.30%

2016 © Samsung Electronics

55

Detail Analysis for Mumbai (7th Jan)



Total Release

Abnormal Release

% Fail

1781047

79516

4.46

Out of 79516 Abnormal release more than 95% failures contributed by below shown 10 Cause codes. 25.00

% Contribution to Total Failures

20.99

20.87

20.00

14.02

15.00

11.92 10.00

7.14

6.63 4.56

5.00

4.56

2.79

1.61

0.00 S1AP - Reduce Load In Serving Cell

2016 © Samsung Electronics

eNB - Radio Link Failure

RRM - UE S-TMSI eNB Timeout - eNB Timeout - RRC eNB Timeout - RRC eNB Timeout - X2 Duplicate Inter X2 Handover Connection Connection Setup RelocOverall Command Reconfiguration Complete Message Expiry Complete Complete Message Not Received Not Received

eNB Timeout - eNB Timeout - RRC S1AP Intra Handover Connection Authentication Command Reestablishment Failure Complete Complete Message Not Received

56

Analysis on Reduce load in serving cell  Specific device model is contributing to this release cause  Sony, Motorola and Unknown device types Make

Model

Total Failure

MOTOROLA MOBILITY LLC A LENOVO COMPANY

MOTOROLA LX12506245

19701

SONY MOBILE COMMUNICATIONS

TBD

7296

MOTOROLA MOBILITY LLC A LENOVO COMPANY

INDR001345

4414

MOTOROLA MOBILITY LLC A LENOVO COMPANY

INDR000945

3177

N/A

TAC: 35234207

2915

N/A

TAC: 35236007

1856

MOTOROLA MOBILITY LLC A LENOVO COMPANY

MOTO E (2ND GEN) WITH 4G LTE ST12424645

1818

N/A

TAC: 35548407

1265

N/A

TAC: 35235907

902

SONY MOBILE COMMUNICATIONS

XPERIA E4G DUAL

859

MOTOROLA MOBILITY LLC A LENOVO COMPANY

CA12459845

730

N/A

TAC: 35386307

488

N/A

TAC: 35235607

186

N/A

TAC: 35627407

117

N/A

TAC: 35627307

103

2016 © Samsung Electronics

57

Analysis on Reduce load in serving cell  Device Capability Issue  If device does not support “TDD-FDD HO” or “Inter FA HO”, eNB sends the UE to other carrier with “Redirection” with release cause of “Reduce Load in Serving Cell” in current eNB Pkg, instead of “Handover”  Needs to check Feature Group Indicator 13, 25 and 30 in UE Capability Information for all devices

 Call Flow UE

MME

eNB 862

MR EVENT A2(LteHo) RRC Conn ReConfiguration RRC Conn Reconfiguration Comp MR EVENT A5(IntraLteHandover)

RRC Connection Release (with redirectedCarrierInfo)

As UE does not support Inter frequency HO or FDD TDD handover then eNB will initiate redirection

UE Context Release Request(Reduce load in serving cell(19)) UE Context Release Command UE Context Release Complete 2016 © Samsung Electronics

58

Radio Link Failure Analysis 

Out of 16596 RLF observed in entire Mumbai, 11.6% fails happened in 11 cells.



Top 3 Cell Analysis 

4294.16 is IBS of TC22, and supporting data for further analysis not available from CSL for 05/01/2016.

Cell ID

Total Fail

4294.16

473

2761.50

217

2257.16

201



For 2761.5 & 2257.16,finding given below.

4294.17

171



In both cell, specific single IMSI is contributing to most of the fails

2257.17

154

820.37

143

4142.37

129

1076.18

114

2918.37

110

75.35

108

1076.37

104

Total

1924

eNB Id

2761.50 I-MU-KLYN-ENB-6008

2257.16 I-MU-MUMB-ENB-7334

2016 © Samsung Electronics

IMSI Detail

Total Failure

405874000046676

212

405874000025017

4

405874000047743

1

405874000026883

199

405874000062987

1

405874000016480

1

Remarks Outskirt facing cell having no site within 2-3 km area. Failures from far location suspected. Last CQI Reported is of Value 5 (SINR < 0). Also fails observed in one specific IMSI only. No site upto 1 km and very high rise clutter observed. Failures from far and high-rise indoor location suspected. Last CQI Reported is Of Value 3 (SINR < 0). Also fails observed in one specific IMSI only.

59

Radio Link Failure (2761.50 - I-MU-KLYN-ENB-6008) 

Outskirt facing cell having no site within 2-3 km area. Failures from far location suspected.



Also fails observed in one specific IMSI only.

2761.50

> 1 km

2016 © Samsung Electronics

60

Radio Link Failure (2217.56 - I-MU-MUMB-ENB-7334) 

No site upto 1 km and very high rise clutter observed. Failures from far and high-rise indoor location suspected.



Also fails observed in one specific IMSI only.

Non-Integrated Site

> 1 km

2217.56

2016 © Samsung Electronics

61

Inter X2 HO Fail Analysis 



Out of 9481 fails observed in entire Mumbai, 21% fails happened in 1 cells only.

Cell ID

Total Fail

1350.18

1953

Top 1 Cell Analysis 

When checked IMSI wise fails, only one IMSI contributing to 77% of the fails.



Need to trace that IMSI and do the further testing.



For remaining fails in this cell, wrong neighbor deletion/PCI conflict to be checked.

eNB Id

1350.18 I-MU-MUMB-ENB-1327

2016 © Samsung Electronics

IMSI Detail

Total Failure

405874000050068

1498

405874000051815

61

405874000064118

53

Other IMSI

308

62

[Issue-1] Missing Events 

TrueCall is not capturing all the events.



As seen below, in TrueCall Logs, there are no events between 01/06/16 1:26:56 to 1:28:41, whereas, as per CSL logs, 3 events occurred during same time.

TrueCall Logs

CSL Logs eNBNAME

Call Release Cause

eNB_1484 eNB_2166 eNB_4241 eNB_4103 eNB_3417 eNB_964 eNB_4091 eNB_1662 eNB_2686

0x00000FFF 0x00000FFF 0x00000FFF 0x00000FFF 0x00000308 0x00000324 0x00000236 0x00000343 0x00000308

2016 © Samsung Electronics

Call Attempt Time

Call Setup Time

2016-01-06 10:55:48:650 10:55:48:744 2016-01-06 11:19:49:869 11:19:49:962 2016-01-06 12:04:18:839 12:04:18:944 2016-01-06 13:12:01:035 13:12:01:190 2016-01-06 13:27:25:845 255:255:255:65535 2016-01-06 13:26:57:707 13:26:57:801 2016-01-06 13:28:15:081 255:255:255:65535 2016-01-06 13:28:28:082 13:28:28:188 2016-01-06 13:30:44:682 255:255:255:65535

FreqBandI cgi.PLMN ndicator ID

40 3 40 40 40 40 40 40 40

405 874 405 874 405 874 405 874 405 874 405 874 405 874 405 874 405 874

cgi.CI

379921 554532 1E+06 1E+06 874768 246802 1E+06 425488 687633

PCI

Call Release Time

PCI cgi.PLMNID

76 2016-01-06 10:56:17:923 76 208 2016-01-06 11:20:21:337 208 332 2016-01-06 12:05:08:922 332 501 2016-01-06 13:12:27:062 501 252 2016-01-06 13:27:28:855 252 236 2016-01-06 13:27:30:859 236 501 2016-01-06 13:28:16:620 501 3 2016-01-06 13:28:41:234 3 181 2016-01-06 13:30:47:693 181

cgi.CI

405 874 379921 405 874 554532 405 874 1085714 405 874 1050384 405 874 874768 405 874 246802 405 874 1047313 405 874 425488 405 874 687633

IMSI

405874000017389 405874000017389 405874000017389 405874000017389 405874000017389 405874000017389 405874000017389 405874000017389 405874000017389 63

CSL Based X2 Failure Analysis For Wrong Neighbour

64

[0x0308]EccTmout_InterX2HandoverCmdComplete – Case Study 

[0x0308] Tags When A call is released when the target eNB cannot receive the RRC Connection Reconfiguration Complete message during the X2 handover.



As Per UE Logs UE has sent MR For 118 PCI which It Found better so A3 Event Triggered. But After that UE has Started PRACH Procedure but failed as done on wrong Cell.

Source cell PCI 53

UE

Wrong Target cell PCI 118

Actual Target PCI 118

A3 Event Triggered MR

HO Preparation

HO Command PRACH FAIL MSG2

3000ms

EccTmout_InterX2HandoverCmdComplete 2016 © Samsung Electronics

65

How To Identify Wrong Nbr Based Upon CSL – (1/2) Target CI

Call Release Cause

Call Attempt Time

Call Setup Time

0x00000308 0x00000308 0x00000308 0x00000308 0x00000308 0x00000308 0x00000308

2016-01-04 00:05:04:044 2016-01-04 00:05:45:940 2016-01-04 00:05:46:505 2016-01-04 00:06:48:003 2016-01-04 00:06:50:607 2016-01-04 00:07:17:781 2016-01-04 00:07:20:331

255:255:255:65535 255:255:255:65535 255:255:255:65535 255:255:255:65535 255:255:255:65535 255:255:255:65535 255:255:255:65535

Source CI

cgi.CI[0] cgi.CI[1] 689445 354851 542225 887077 947728 557073 212259

89125 287762 936466 735781 869155 239633 541475

IMSI

PCI

405874000000000 405874000000000 405874000000000 405874000000000 405874000000000 405874000000000 405874000000000

203 147 31 323 45 4 105

 By CSL we get Cause Code/ Cell Id From UE History, so that we can Find Source & Target eNB .

 Please Check Call Release Cause “0x0000 308” is for A call is released when the target eNB cannot receive the RRC Connection Reconfiguration Complete message during the X2 handover  By True call Please check Column Name “Last HO Source Cell ID” & “Last HO Target Cell ID”, For Call Release Casue “eNB Timeout – eNB Timeout - Inter X2 Handover Command Complete 2016 © Samsung Electronics

66

How To Identify Wrong Nbr Based Upon CSL – (2/2) eNB275

Wrong Id

eNB1231

Correct Id eNB2155

eNB2155 Source Id

Wrong Target Id

2155.37 275.17

Distance (Km)

Total Failure

5.07

172

Source PCI Target PCI Nearst eNB Nearst Distance

53

118

1231

0.76

Remarks

DELETE

 Ue has sent measurement report for eNB1231 PCI 118 but UE Does RACH With eNB275, Due to Wrong eNB Defined in eNB2155 PCI 53.  Hence Handover Fail So we can Check Nearby Same PCI & Delete Wrong One To Reduce Drop Rate by Handover Fail

S1 AP Links – Uptime Very Critical

21-Jan-2016

68

UE Temp ID’s – S1AP, GTP-C, GTP-U

2016 © Samsung Electronics

69

SCTP Link Fluctuation –MT UE in IDLE Mode  MT UE was connected to eNB-1 and moved to IDLE mode due to User Inactivity.  Incoming call received for the UE.  As the UE was in IDLE mode, MME has sent Paging Request to eNB-2.  As the SCTP link with the eNB-1 where UE is currently present, UE will not be able to recei ve Paging Request message

2016 © Samsung Electronics

70

SCTP Link Fluctuation –MO Call  MO UE attached to MME-1 and GW-1 via eNB – 1 and VoLTE call is on-going  SCTP link to MME-1 with eNB-1 and eNB-2 are down.  UE sent Service Request to MME-2 via eNB-2(as eNB2 -> MME-1 link is down)  MME-2 will send service Reject which results in UE sending Fresh Attach Request.  As UE sent new Attach, the previous call gets dropped

2016 © Samsung Electronics

71

Network issue related to Backhaul

Introduction  GTP Loss  GTP Loss  Root Cause of GTP Loss  Statistic & Troubleshooting

 MME&IND Communication Fail  MME&IND communication  Root Cause of MME&IND Communication  Statistic & Troubleshooting

 X2 Communication Fail  X2 communication Fail  Root Cause of X2 Communication Fail  Statistic & Troubleshooting

2016 © Samsung Electronics

73

GTP Loss  GPRS Tunneling Protocol (GTP) is a group of IP-based communications protocols used to carry general packet radio service (GPRS) within LTE networks  GTP Loss means packet drops between SAEGW and eNB. It is only calculated on bearer down link(SAEGW --> eNB direction).  GTP Loss can happen only by user traffic but cannot occur by ping or traceroute between SAEGW and eNB.  In case of Samsung SGW & PGW both are at same location (called SAEGW). So we will always check GTP loss between SGW and eNB.

2016 © Samsung Electronics

74

Root Cause of GTP Loss There are two reasons for GTP loss (packet drop) in backhaul Buffer Overflow (Backhaul Bandwidth & Buffer size defined)  Physical drops

 1- Buffer OverflowWhen network element send data packets with high speed and receiving node cannot forward it with the same speed due to bottleneck in network, And has small buffer t o store packets so it will start discarding excess packets once its buffer gets full. This situation is called Buffer overflow.

 2- Physical drops The next common issue that can lead to drops would be a physical component that is malfunctioning. Physical issues like SFP module problem, cable problem, loosed connection, line card problem, duplex mismatch, etc. are the most common reasons for packet drop in network.  These physical drops and backhaul media Buffer overflow results in low throughput.

2016 © Samsung Electronics

75

GTP Loss- Trouble-shooting  Check GTP loss counter(GtpSnEnbDeltaLossRate) in LSMR and if GTP loss more than 0.01% observed then check for below 3 cases 1. Physical drops (drop rate requirement is less than 0.01%). 2. Small buffer in the bottleneck (Buffer size < 512KBytes and bottleneck bandwidth network capacity  Congestion  The packet loss happened intermittently, even if packets sending under network capacity.

 TCP congestion control HSS S6a

MPLS Network

GTP-U

GTP-U GTP-C GTP-U

FTP

2016 © Samsung Electronics

86

TCP Packet loss  Intermittently, the packet loss occurs by Rate limitation in the normal network environment.  Even if the packet loss happened with an unspecified that it does not stop the DT test.

 If constantly experiencing ‘packet loss’ or find out the ‘Rate limitation’ then stop the DT test and have the action in order to fix it. — The MOP that is BH checking process already shared by RAN TF

 If you find out the Low Throughput somewhere during test in the field.    

As you well known, Please check the RF Condition including eNB status (Is it down or not ?) Please check the current alarm (Related BH, Related RU, Related RF, Etc.) Check the Rate limitation using shared MOP If you do not find out the reason, then call the RAN BH TF

2016 © Samsung Electronics

87

What is GTP Loss  GPRS Tunneling protocol is an important IP based protocol used in GSM, UMTS and LTE core networks. It is used to encapsulate user data when passing through core network and also

carries bearer specific si

gnaling traffic between various core network entities.  It provides mobility. When UE is mobile, the IP address remains same and packets are still

forwarded

since tunneling is provided between PGW and eNB via SGW.  Multiple tunnels (bearers)can be used by same UE to obtain different network QoS  Main IP remains hidden so it provides security as well.

2016 © Samsung Electronics

88

What is GTP Loss (Contd) Note-

 GTP Loss means packet drops between SAEGW and eNB.  It is only calculated on bearer down link(SAEGW --> eNB direction).  GTP Loss can happen only by user traffic but cannot occur by ping or traceroute

betwee

n SAEGW and eNB.  GTP tunnel is formed only between SAE-GW and eNB so GTP loss has no relation with UE RF conditions.

2016 © Samsung Electronics

89

Detail picture showing data flow inside GTP tunnel-

2016 © Samsung Electronics

90

90

GTP OOS  When UE tries to download any data then TCP session is established between UE and server.  When this data comes to SAE-GW from server then It encapsulates this data using P, IP, L2 and L1 headers and forwards this data to eNB through GTP tunnel.

GTP, UD

 GTP header consists of Sequence no. & tunnel end point Id (TE ID) which is assigned to each packet.  This encapsulated data transfers through GTP tunnel with Sequence no. 1,2,3,4,5….. and so on and reaches to eNB.  If after 2nd packet eNB receives 4th packet instead of 3rd packet then this event would be cou nted as loss in GTP loss counter but after receiving 3rd packet this event would be counted as Out of Sequence packet(OOS) as shown below-

2016 © Samsung Electronics

91

91

GTP Delta Loss

 That means this packet was not lost but came after words (Out of sequence packet) so this will not contribute to GTP loss. This can be seen in GtpSnEnbDeltaLoss column which is calculated as-

2016 © Samsung Electronics

92

92

Why GTP Loss happens There are basically two reasons for GTP loss (packet drop) in backhaul— Buffer Overflow — Physical drops

 Buffer OverflowWhat is Buffer in a network device?

-Small amount of memory used for temporary storage of data, usually to compensate for differe nces in between receiving and transmitting speeds on a network device. It serves as a reservoir i n which the higher speed ingress data to be dumped which is then 'trickled' to the slower one. What is Buffer overflow? -When server is sending data packets with very high speed and receiving node cannot forwar d it with the same speed due to bottleneck in network and it has small buffer to store packets so it will start discarding excess packets once its buffer gets full . This situation is cal led Buffer overflow.

2016 © Samsung Electronics

93

93

Buffer Overflow

 Reason for Buffer overflowRate MismatchCase 1

2016 © Samsung Electronics

Case2

94

94

Physical Drops  Physical dropsThe next common issue that can lead to drops would be a physical component that is

malfunct

ioning. -Physical issues like SFP module problem, cable problem, loosed connection, line card m, duplex mismatch, etc. are the most common reasons for packet drop in a

proble

network.

-If hardware is not working properly, it will usually lead to error messages being seen on the con sole of the device or within system logs but there are other software issues also which can lead t o packet drop and main concern is there is no alarm indication

mechanism for these issu

es in the system

2016 © Samsung Electronics

95

95

GTP Loss As an Indication  GTP loss can’t be avoided in commercial mode when multiple TCP session establishes to access data ,making congestion in network bottleneck point.

 GTP Loss cannot be used as any index or indicator which means backhaul

normal

ity because of speed mismatch in backhaul environment (10G -> 1G on AG1 to CSS ring, 1G -> 470, 250, 150, 100, 45 on Microwave or Leased

ttleneck buffer overflow cannot be avoidable in Multi TCP sessions keep trying to increase TCP

line), GTP Loss by bo

commercial mode where window size so congestion happ

ens in a bottleneck point and so buffer overflow occurs.

2016 © Samsung Electronics

96

96

Where and how to see GTP Loss  GTP loss can be seen in LSMR in performance statistics or On demand monitoring. In GTP statistics, there are two items of GTP Loss and GTP OOS. The real packet loss can be calcul ated by (GTP Loss Count - GTP OOS count) and it's shown in GtpSnEnbDeltaLoss. So GtpSnEnbDe ltaLossRate shows the real loss rate.  How to check GTP loss ? -Generate the traffic on the UE and check the GTP loss counter (GtpSnEnbDeltaLossRate) in on d emand monitoring. — — — —

Login to LSMR. Go to PERFORMANCE-On Demand. Select target. Register-In Statistics family, go to GTP & select GTP sequence number per eNB-period (30sec)duration ( as per testing requirement).Then click on register. — After this click on site which is registered then Monitor option will be highlighted. Click on monitor and select GtpSnEnbDeltaLoss(count) and GtpSnEnbDeltaLossRate(%)-click Apply and then click Start.

NOTE-GTP Loss must be less than equal to 0.01% for proper throughput. (>50Mbps)

2016 © Samsung Electronics

97

97

Example Report Example-

Note- Please note that the GTP loss counter is only valid for downlink throughput analysis. 2016 © Samsung Electronics

98

98

Troubleshooting Steps  Check GTP loss from LSMR as explained above and if GTP loss occurs perform IPERF tween SGW and eNB to check

test be

— Physical drop — Backhaul Bandwidth — Buffer size

Put iperf.octeon64 software in LENA and in eNB, use below command for Iperf test in k.

Downlin

 eNB Side: iperf.octeon64 -s -V -B eNB_Bearer_IP -u -l 8000 -i 1 -p 8001  SGW(LENA) :

iperf.octeon64 -c eNB_Bearer_IP -u -V -i 1 -b 1m -l 8000 -t 9999 -p 8001 Note – Change the mode of iperf file using command “chmod777 iperf.octeon64” be fore executing commands. Where, s->server, V->IP version6, B->binds eNB IP, u->UDP traffic, l 8000->length of datagram size(bytes), p->port, c->client, t->time, b->B.W. to be pumped. First step is to check if any physical drop is happening.

2016 © Samsung Electronics

99

99

Identification  How to identify Physical issue ? - Perform IPERF test between SGW and eNB on lower B.W.(less than equal to 10Mbps) - If packet drop is observed while pumping lower B.W from SGW to eNB then that means there is some physical issue in backhaul. -Ask cisco team to check backhaul. Note- Please note that on Lower B.W. ( SAEGW ---- > MPLS network ---- > eNB ---- > UE) -Put iperf software in FTP server and download iperf tool in UE from playstore to perform test. DOWNLINK UDP (FTP server to UE)FTP server side: iperf -c IP_of_UE -u -i 1 -t 90 -b 50m -p 5002 UE side: iperf -s -u -i 1 -p 5002 UPLINK UDP (UE to FTP serve)UE side: iperf -c Service_IP_of_FTP_server -u -i 1 -t 90 -b 10m -p 5002 FTP server side: iperf -s -u -i 1 -p 5002

2016 © Samsung Electronics

108

108

Iperf Commands  TCP IPERF test can also be performed between FTP and UE using below ands-

comm

DOWNLINK TCP (FTP server to UE)FTP server side: iperf -c IP_ of_UE -i 1 -t 90 -w 1m -p 8002 UE side: iperf -s -i 1 -p 8002

UPLINK TCP (UE to FTP server)UE side: iperf -c Service_IP_of_FTP_server -i 1 -t 90 -w 1m -p 8002 FTP server side : iperf -s -i 1 -p 8002

2016 © Samsung Electronics

109

109

Summary-Even though low throughput is observed in the field, if the below 3 cases are not in aul, definitely there's no backhaul issue causing low throughput.

backh

-Physical drops (drop rate requirement is less than 0.01%) -Small buffer in the bottleneck Buffer size < 512KBytes & bottleneck B.W. 1518 size on UDP instead of sending on TCP --- Issue is with Nokia R&D team

Open

IMS

Open

IMS

Close

MME

Close

OCS/PCRF/ DRA

Close

Cisco

IMS NOT forwarding SIP INVITE, 183 Patch need to be applied to fix the multiple message Session Progress, SIP UPDATE, forwarding issue at TAS & SCSCF. (200OK) for INVITE UE find RLF during X2 Handover process due to X2 connectivity fails/X2 link not available. X2 HO failure procedure fails due to These causes QCI-1 bearer deactivation and hence call gets “Tx2 timer expiry” or due to radiodropped. procedure failure Preservation mode is enabled in MME for SA1P cause code# 1, 6, 21,26 to make sure Post successful X2 HO procedure, While requesting for S1U Source change from old eNB to new eNB, (“path switch Post Successful X2 HO , Call drops request”) timer expires at eNB due to delay in response due to Patch Switch request timer ( i.e. >3 sec) from OCS or PCRF or DRA to respond to CCR-U expiry message which is triggered after Modify bearer request from MME to SAEGW. While X2-HO between two eNB served by different MMEs, S5 Interface issue - Packet loss btw SGW relocation get triggered while PGW remains same as S-GW and P-GW due to SGW earlier. relocation process while X2-HO - Because there was no IP routing configuration between between two eNBs served by SGW and PGW each located at different sites, GTPU packets different MMEs on the S5-interface were dropped. - After IP routing configuration problem resolved.

2016 © Samsung Electronics

133

[3] MO received SIP-503 “service unavailable”

eNB

 Issue Description: QCI-1 bearer setup procedure failed in eNB  Resolution: Issue got fixed after changing A2 Parameter configuration change in eNB which was generating race condition within eNB and were not responding to MME for ERAB Setup requests.

2016 © Samsung Electronics

134

[5] MO Received SIP – 503 – Issue at OCS

OCS

 Issue Description: Delay from OCS for CCA response resulted in VoLTE Call setup Failure with ASR error “Session released - service based local policy function aborted session".  Resolution: Temporarily OCS has been bypassed for ACME testing.

2016 © Samsung Electronics

135

[7] Received 500 Server Internal Error for INVITE

Issue Description:

IMS

IMS is not forwarding 200 Ok for Invite towards MO UE.

Resolution: IMS patch applied at TAS.

2016 © Samsung Electronics

136

[13] X2 HO - SGW change Btwn Noida/Vikaspuri 6 Pair of SAEG Ws (i.e. 36 combina tions to check)

G 6

•

Possible Impacted Case: For S ingle PLMN, two Sites distribu ting traffic with S5.

•

Applicable Sites - Mumbai, D elhi and Chennai.

• •

Scenario: On Registration Co-located SA EGW get selected Initial PGW provides IP to use r Once HO happens between S GW Group, New SGW will get selected but PGW remains sa me. Signaling & Traffic moves fro m new SGW to old PGW on S 5 interface.

6

G

• •

•

In Delhi Case – S5 GTP-C was ok hence bearer was Modified correctly but S5 GTP-U was down hence Voice/Data was not working. 2016 © Samsung Electronics

137

[13] S5 Failure (X2 HO with SGW change) (1/2) S

2016 © Samsung Electronics

T

Cisco

138

[13] S5 Failure (X2 HO with SGW change) (2/2)  GTPU packets on the S5 Interface getting dropped due to missing IP Route  Uplink packets are getting dropped in between SGW(Noida) -> PGW(Vikaspuri)  Downlink packets are getting dropped in between PGW(Vikaspuri) -> SGW(Noida)

2016 © Samsung Electronics

139

No Response for SIP Bye(Backbone) Issue-1  Previous call was not completed. SIP Bye was sent for which response was not received.  Even after multiple retransmission(till 12:30:32), even TCP Ack was not received.  No SIP Invite seen at 12:30:14 in the network logs

City – Mumbai Date – 06-Jan-2016

Similar issue is observe d @ 13:10:02 also

2016 © Samsung Electronics

140

UE Issue - No Response for SIP Invite  MT UE was in connected mode  MT UE was not responding the SIP Invite message sent.  Even after multiple retransmissions, there was no response from MT UE.

City – Mumbai Date – 06-Jan-2016

2016 © Samsung Electronics

141

Call Setup failure due to User Inactivity(30 sec)  MT UE has not answered the call within 30 sec.  As User Inactivity timer is 30 sec, eNB has sent UE Context Release with reason USER-INAC TIVITY.  So MME deleted the QCI-1 bearer which resulted in VoLTE Call Setup failure.

City – Mumbai Date – 07-Jan-2016

2016 © Samsung Electronics

142