CLOT KPI Analysis Guideline(RJIL)_v1.0

CLOT KPI Analysis Guideline (RJIL) March, 2015 Solution Lab.

Responsible for this document: 

배경 • • •



작성범위 • •



본 문서를 통해, RJIL CLOT 시험결과에 대한 원활한 KPI Pass/Fail 분석 업무 수행을 위해 초기 KPI Fail 사례 등을 바탕으로 작성 하였음 작성시점 기준 ActixOne을 활용한 분석이 용이하지 않아, 향후 AxtixOne을 활용한 분석 방법 기술 필요(VoLTE 관련 항목 등) 본 문서는 프로젝트 목표 달성을 위해 관련자 모두에게 공유 가능하며, 수정사항 필요 시 수정 및 추가 가능함

본 문서는 KPI 기준 및 ActixOne/XCAP/Anyplan-Optimizer활용 분석 방안을 제시 함 Reference CLOT 진행 시 발생 한 KPI Fail 사례를 통해 이슈를 정의 함

Change History: Changes

Author

Date

Version

Initial Version

Joohyun Kim/Jaewook Jung

29.03.2015

1.0

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

1

Table of contents  CLOT KPIs  How to Analyze KPI Items • • • • • • • • • • • •

RSRP SINR (Info) Overlapping Servers within 5dB PCI for all Sectors Observed (Info) Mac DL Throughput >= 2Mbps Mac UL Throughput >= 256kbps Average Mac DL Throughput Average Mac UL Throughput Call Drop TAU Success Rate (Info) Handover Success Rate Handover Interruption Time < 100ms

2

Table of contents  How to Analyze KPI Items • • • • • • • • • •

RACH Success Rate Attach Success Rate VoLTE Setup Success Rate Attach Setup Time < 1.5s IMS Registration Time < 1.5s (Report Only) VoLTE Setup Time < 1.5s VoLTE Jitter < 40ms VoLTE RTP Packet Loss Rate Average BLER DL (Info) Average BLER UL (Info)

 Appendix

3

CLOT KPIs No

KPIs

Call Model

Category

Target Value

NonExclusion

1

RSRP

LDL1, LUL

KPI

> 95 %

Don’t Consider

2

SINR >= -2 dB

LDL1, LUL

Info

-

Don’t Consider

3

Overlapping Servers within 5 dB 60 %

Don’t Consider

4

Overlapping Servers within 5 dB 90 %

Don’t Consider

5

PCI for all Sectors Observed

LDL1, LUL

Info

-

Don’t Consider

6

MAC Throughput DL >= 2 Mbps

LDL

KPI

> 95 %

Don’t Consider

7

MAC Throughput UL >= 256 kbps

LUL

KPI

> 95 %

Don’t Consider

8

Average MAC Throughput DL (kbps)

LDL

KPI

> 16,500

Don’t Consider

9

Average MAC Throughput UL (kbps)

LUL

KPI

> 2,500

Don’t Consider

10

Call Drop Rate

LDL1, LUL, VoL

KPI

98 %

Don’t Consider

13

Handover Interruption Time < 100ms

LDL1, LUL, VoL

KPI

> 95 %

Don’t Consider

14

RACH Success Rate

VoS

KPI

> 99 %

Don’t Consider

15

Attach Success Rate

VoS

KPI

> 99 %

Don’t Consider

16

VoLTE Setup Success Rate

VoS

KPI

> 99 %

Don’t Consider

17

Attach Setup Time < 1.5 s

VoS

KPI

> 95%

Don’t Consider

18

IMS Registration Setup Time < 1.5 s

VoS

Info

> 95%

Don’t Consider

19

VoLTE Setup Time < 1.5 s

VoS

KPI

> 95%

Don’t Consider

20

VoLTE Jitter < 40 ms

VoL

KPI

> 95 %

Don’t Consider

21

VoLTE Latency < 100 ms

VoL

KPI

> 95 %

Don’t Consider

22

VoLTE RTP Packet Loss Rate

VoL

KPI

95%

LDL1, LUL (Merge)

Formula •

(The number of passing RSRP samples / The total number of RSRP samples) within exclusion zone — Passing RSRP samples : the binned RSRP samples which meet ‘Coverage Objective RSRP Threshold’ within exclusion zone — The total number of RSRP samples : the all binned RSRP samples within exclusion zone

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

7

1. RSRP 

Analysis using ActixOne and AnyPlan-Optimizer •

Check KPI value in ActixOne Report

•

Find poor RSRP area using ActixOne and AnyPlan-Optimizer — Check RSRP plots with BSP in AnyPlan-Optimizer ※ Poor RSRP area outside BSP does not have to be considered in the analysis

•

Find the issue eNBs related with poor RSRP area — Check PCI plots in ActixOne (or AnyPlan-Optimizer)

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

8

1. RSRP 

Analysis using ActixOne and AnyPlan-Optimizer •

Check the coverage of the issue eNBs — Check PCI plots in ActixOne(or AnyPlan-Optimizer) — Execute ‘Sector Swapped Site’ and ‘Tx Off Site’ in AnyPlan-Optimizer — Check ‘Sector Plot’ for each issue eNB in AnyPlan-Optimizer

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

9

1. RSRP 

Analysis using ActixOne and AnyPlan-Optimizer •

Check the issue eNBs alarm status during drive test from LSM — Check the alarms such as RRH/Site/SNMP Down, ECP/UMP/GPS Function Fail , VSWR etc.

I-OR-SUNA-ENB-9005 Alarm History in eNB No.

Severity

Code

541

Critical

2007

556

Critical

2007

ProbableCause

Alarm time

Clear time

Clear type

Protocol Status Alarm ( Communication Fail : PING )

2015-02-03 12:37

2015-02-03 12:43

Auto

Protocol Status Alarm ( Communication Fail : PING )

2015-02-03 11:49

2015-02-03 11:49

Auto

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

10

1. RSRP 

Analysis using ActixOne and AnyPlan-Optimizer • • •

Execute ‘AOP’ in AnyPlan-Optimizer Fine-tune ‘AOP’ recommendation manually if necessary Check the prediction results — RSRP as well SINR and Serving Sector area should be also checked together

※ Refer to ‘AnyPlan-Optimizer User Guideline’ for the details ※ If RSRP KPI is not improved after finishing ‘AOP’/’Modify Sector Parameter’, it should be checked UE’s normality or the measurement environments such as obstacles Baseline

Prediction

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

11

1. RSRP 

Analysis Results •

Antenna Recommendation — Azimuth, M-Tilt and E-Tilt

•

The issue eNBs lists ※ All alarms not only for issue eNBs but also for all eNBs in the cluster should be checked and cleared before drive test

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

12

2. SINR (Info) 

Description •



Target Value •



-

Used Call Model •



Ratio of SINR that meets ‘SINR Outage Threshold’ value (-2dB) within exclusion zone

LDL1, LUL (Merge)

Formula •

(The number of passing SINR samples / The total number of SINR samples) within exclusion zone — Passing SINR sample : the binned SINR samples which meet ‘SINR Outage Threshold’ value (-2dB)’ within exclusion zone — The total number of SINR samples : the all binned SINR samples within exclusion zone

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

13

2. SINR (Info) 

Analysis using ActixOne and AnyPlan-Optimizer •

Check KPI value in ActixOne Report

•

Analyze poor SINR area : Analysis methodology would be the similar with RSRP analysis ※ Refer to ‘RSRP analysis’ and ‘AnyPlan-Optimizer User Guideline’ for the details — Find Poor SINR area — Find issue eNBs and check alarms — Execute ‘AOP’/’Modify Sector Parameters’ in AnyPlan-Optimizer etc. ex) removing dominant interference or enhancing the serving signal — Check the prediction results



Analysis Results •

Antenna Recommendation — Azimuth, M-Tilt and E-Tilt

•

The issue eNB lists ※ All alarms not only for issue eNBs but also for all eNBs in the cluster should be checked and cleared before drive test © SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

14

2. SINR (Info) 

Improve Point • Even through RSRP is good, SINR was very low due to missing neighbor ▶ Check Neighbor relation between two cells

SINR

RSRP

Despite of relatively high RSRP, SINR is very low due to missing neighbor

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

15

3. Overlapping Servers within 5 dB 

Description •



Target Value •



> {60% | 90%} for 1 server and 2 servers respectively

Used Call Model •



The average number of sectors having RSRP difference with serving best server within 5 dB, including best server signal within exclusion zone ※ The number of overlapping sectors is calculated first, then it is averaged by binning

LDL1, LUL (Merge)

Formula •

(The number of passing overlapping samples / The total number of bin samples) within exclusion zone — Passing overlapping samples : the binned samples which meet the average number of overlapping sectors within 5 dB = 2Mbps 

Analysis using ActixOne, AnyPlan-Optimizer and XCAP •

Check KPI value in ActixOne Report

•

Find poor DL Throughput area using ActixOne and AnyPlan-Optimizer — MAC Layer Throughput Download could be checked in ActixOne report ※ Only ‘PayloadRate’ can be displayed in ActixOne — Check DL Throughput plots with BSP in AnyPlan-Optimizer ※ Poor DL Throughput area outside BSP does not have to be considered in the analysis

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

23

5. DL MAC Throughput >= 2Mbps 

Analysis using ActixOne, AnyPlan-Optimizer and XCAP •

Find the issue eNBs related with poor DL Throughput area — Check PCI plots in ActixOne (or AnyPlan-Optimizer)

•

Check the issue eNBs alarm status during drive test from LSM — Check the alarms such as RRH/Site/SNMP Down, ECP/UMP/GPS Function Fail , VSWR etc. ※ Refer to ‘RSRP analysis’ for the details

•

Modify Antenna Parameters in AnyPlan-Optimzier (Improve DL Edge SINR) — Execute ‘AOP’ in AnyPlan-Optimzier — Execute ‘Modify Sector Parameters’ in AnyPlan-Optimzier if necessary ※ Refer to ‘RSRP analysis’ and ‘AnyPlan-Optimizer User Guideline’ for the details

I-OR-SUNA-ENB-9005 Alarm History in eNB No. Severity Code 541

Critical

556

Critical

ProbableCause

Alarm time Clear time

Protocol Status Alarm 2015-02-03 2015-02-03 2007 ( Communication Fail : 12:37 12:43 PING ) Protocol Status Alarm 2015-02-03 2015-02-03 2007 ( Communication Fail : 08:53 11:49 PING )

Clear typ e Auto

Auto

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

24

5. DL MAC Throughput >= 2Mbps •

Check other issues in XCAP — Check missing Neighbors (No HO after multiple MR transmission) — Check Backhaul issues ‐ Server Drop (Error message from FTP server) ‐ Socket Close ‐ Initial attach or IdletoActive fails due to no receiving ‘dlInformationTransfer’ message — Check FTP Server issues



Analysis Results •

Antenna Recommendation — Azimuth, M-Tilt and E-Tilt

•

The issue eNB lists ※ All alarms not only for issue eNBs but also for all eNBs in the cluster should be checked and cleared before drive test

•

Missing NBR list etc.

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

25

5. DL MAC Throughput >= 2Mbps 

Fail Cases •

Poor RF (SINR) — DL MAC Throughput was low due to the low SINR

▶ RF Optimization DL MAC Throughput fail

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

26

5. DL MAC Throughput >= 2Mbps 

Fail Cases •

Missing Neighbors — DL MAC Throughput was low due to no HO procedure ‐ No HO procedure after multiple MR transmission due to missing Neighbors ‐ RACH procedure by ‘UL Data arrival’ after multiple MR transmission due to missing Neighbors

▶ Check Neighbor list in LSM (check missing neighbor)

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

27

5. DL MAC Throughput >= 2Mbps 

Fail Cases •

Interference issue — DL MAC Throughput was low due to UL interference ‐ eNB didn’t receive ACK/NACK for FTP DL data on PUSCH due to high IoT in this eNB

▶ Check Interference

LDL PUSCH Tx Power LDL Power Headroom

LDL BLER

LDL Low DL Throughput © SAMSUNG Electronics Co., Ltd. Confidential and Proprietary © SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

28

5. DL MAC Throughput >= 2Mbps 

Fail Cases •

Late Service Request Triggering — UE triggered service request too late after RRC Connection Release, so there happened server drop

▶ Check Backhaul and S1 interface

Server Drop Release DL Low Throughput

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

29

6. UL MAC Throughput >= 256kbps 

Description •



Target Value •



> 95%

Used Call Model •



Ratio of MAC UL Throughput which meets the criteria (256kbps) within exclusion zone

LUL

Formula •

(The number of passing UL Tput samples / The total number of UL Tput samples) within exclusion zone — Passing UL Tput samples : the binned MAC UL Throughput samples which meet 256kbps within exclusion zone — The total number of UL Tput samples : the all binned MAC UL Throughput samples within exclusion zone

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

30

6. UL MAC Throughput >= 256kbps 

Analysis using ActixOne, AnyPlan-Optimizer and XCAP •

Check KPI value in ActixOne Report

•

Analyze poor UL Throughput area : Analysis methodology would be the similar with DL Edge Throughput analysis — — — —

Find poor UL Throughput area Find issue eNBs and check alarms Execute ‘AOP’/’Modify Sector Parameters’ in AnyPlan-Optimizer etc. Check other issues such as missing neighbor, FTP Server issues in XCAP

BSP

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

31

6. UL MAC Throughput >= 256kbps 

Fail Cases •

Poor RF (RSRP) — UL MAC Throughput was low due to the low RSRP

▶ RF Optimization UL MAC Throughput fail

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

32

6. UL MAC Throughput >= 256kbps 

Fail Cases •

Missing Neighbors — UL MAC Throughput was low due to no HO procedure ‐ After multiple MR transmission, UE didn’t start HO procedure due to missing Neighbors ‐ After multiple MR transmission, UE started RACH procedure by ‘UL Data arrival’

▶ Check Neighbor list in LSM (check missing neighbor)

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

33

6. UL MAC Throughput >= 256kbps 

Fail Cases •

Backhaul issue — UL MAC Throughput was low due to several backhaul related issues ‐ FTP Server Drop ‐ Socket Close ‐ Small Buffer Size of UE due to small TCP window size of eNB ‐ Late service request after RRC Connection Release ‐ No receiving ‘Security protected NAS message’ message from MME after RRC connection setup complete in normal RF environment

▶ Check Backhaul and S1 interface

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

34

6. UL MAC Throughput >= 256kbps 

Fail Cases •

Inherence issue — UL MAC Throughput was low due to interference(Tx Power : Max, Power Headroom : low, RF environment : Not bad)

▶ Check interference

UL Throughput

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

35

7. Average DL MAC Throughput 

Description •



Target Value •



> 16,500kbps

Used Call Model •



Summation of the averaged MAL DL Throughput from device 1 and device 2 within exclusion zone

LDL1, LDL2 (Separate)

Formula •

(Average MAC DL Throughput for LDL1 + Average MAC DL Throughput for LDL2) within exclusion zone — Average MAC DL Throughput : (summation of the binned MAC DL Throughput samples / the total number of MAC DL Throughput samples) within exclusion zone

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

36

7. Average DL MAC Throughput 

Analysis using ActixOne, AnyPlan-Optimizer and XCAP •

Check KPI value in ActixOne Report

•

Analyze poor DL Throughput area : Analysis methodology would be the similar with DL Edge Throughput analysis — — — —

Find poor UL Throughput area Find issue eNBs and check alarms Execute ‘AOP’/’Modify Sector Parameters’ in AnyPlan-Optimizer etc. Check other issues such as missing neighbor, FTP Server issues in XCAP

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

37

8. Average UL MAC Throughput 

Description •



Target Value •



> 2,500kbps

Used Call Model •



The averaged MAL UL Throughput within exclusion zone

LUL

Formula •

Average MAC UL Throughput for LUL within exclusion zone — Average MAC UL Throughput : (summation of the binned MAC UL Throughput samples) / (the total number of MAC UL Throughput samples) within exclusion zone

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

38

8. Average UL MAC Throughput 

Analysis using ActixOne, AnyPlan-Optimizer and XCAP •

Check KPI value in ActixOne Report

•

Analyze poor UL Throughput area : Analysis methodology would be the similar with DL Edge Throughput analysis — — — —

Find poor UL Throughput area Find issue eNBs and check alarms Execute ‘AOP’/’Modify Sector Parameters’ in AnyPlan-Optimizer etc. Check other issues such as missing neighbor, FTP Server issues in XCAP

BSP

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

39

9. Call Drop Rate 

Description •



Target Value •



98%

LDL1, LUL and VoL (Separate)

Formula •

The number of Handover Success / The number of Handover Attempts — Handover Success : LTE MAC RACH Attempt Success for ‘rrcConnectionReconfigurationComplete’ — Handover Attempts : Sending ‘rrcConnectionReconfiguration’ (contains HO command)

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

52

11. Handover Success Rate 

Message from XCAP •

From ‘rrcConnectionReconfiguration’ message (with HO Command) to LTE MAC RACH attempt Success

Attempts

Complete

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

53

11. Handover Success Rate 

Analysis using ActixOne and XCAP •

Check KPI value in ActixOne Report

•

Check ‘LTE Handover Fail’ events in ActixOne

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

54

11. Handover Success Rate 

Analysis using ActixOne and XCAP •

Check Handover fail events in Macro and XCAP — Check the specific event time for HO fail

•

Check the detailed messages in XCAP for the fail events

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

55

11. Handover Success Rate 

Fail Cases •

Failure at MSG2 — UE didn’t receive RAR(MSG2) from eNB due to poor RF environments

▶ Check RF environment (RSRP/SINR)

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

56

11. Handover Success Rate 

Fail Cases •

RLF during HO — UE didn’t succeed Handover due to Radio link failure during HO procedure

▶ Check RF Environments (RSRP/SINR)

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

57

11. Handover Success Rate 

Fail Cases •

PCI Duplication — UE didn’t receive RAR(MSG2) due to referring the wrong cell

▶ Check PCI Collision and Confusion in NBR list from LSM

RACH info

SIB2

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

58

11. Handover Success Rate 

Fail Cases •

No Measurement Report — Measurement parameters are not configured in eNB

▶ Check eNB Parameter or configuration

Measurement parameters are not configured

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

59

12. Handover Interruption Time < 100ms 

Description •

Ratio of Handover Success which meets the criteria (100ms) among the intra-frequency LTE Handover Success Procedure — Handover procedure : from RRC Connection Reconfiguration (contains HO command) to LTE MAC RACH Attempt Success for RRC Connection Reconfiguration Complete



Target Value •



Used Call Model •



>95%

LDL1, LUL and VoL (Separate)

Formula •

The number of Handover Success within 100ms / The number of Handover Success — Handover Success within 100ms : LTE MAC RACH Attempt Success for ‘rrcConnectionReconfigurationComplete’ within 100ms — Handover Success : LTE MAC RACH Attempt Success for ‘rrcConnectionReconfigurationComplete’

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

60

12. Handover Interruption Time < 100ms 

Message from XCAP •

From ‘rrcConnectionReconfiguration’ message (with HO Command) to LTE MAC RACH attempt Success

Attempts

Complete

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

61

12. Handover Interruption Time < 100ms 

Analysis using ActixOne and XCAP •

Check KPI value in ActixOne Report

•

Check ‘LTE Slow Handover’ events in ActixOne ※ Handover Interruption Time fail could be different with ‘LTE Slow Handover’ events in ActixOne

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

62

12. Handover Interruption Time < 100ms 

Analysis using ActixOne and XCAP •

Check Handover Interruption time fail events using Macro — Check the specific event time for HO interruption time fail

•

Check the detailed messages in XCAP for the fail events

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

63

12. Handover Interruption Time < 100ms 

Fail Cases •

Multiple MSG2 re-transmission — UE re-transmitted MSG2 due to poor RF environments

▶ Check RF environment (RSRP/SINR)

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

64

13. RACH Success Rate 

Description •

Ratio of Contention-based RACH Success — RACH Procedure (Contention based): from Random Access Request message(MSG1) to Contention Resolution message(MSG4)



Target Value •



Used Call Model •



>99%

VoS

Formula •

The number of RACH Success / The number of RACH Attempts(CT Based) — RACH Success : LTE MAC RACH Attempt Success (Success of Message 4) — RACH Attempts(CT Based) : ‐ LTE MAC RACH trigger followed by ‘rrcConnectionRequest’ ‐ LTE MAC RACH trigger followed by ‘rrcConnectionReestablishmentRequest’

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

65

13. RACH Success Rate 

Message from XCAP •

Initial RRC Connection Setup — From LTE MAC RACH trigger by Connection request to LTE MAC RACH attempt Success Attempts

Complete

•

RRC Connection Re-establishment — From LTE MAC RACH trigger by Radio Link failure to LTE MAC RACH attempt Success Attempts

Complete

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

66

13. RACH Success Rate 

Analysis using ActixOne and XCAP •

Check KPI value in ActixOne Report

•

Check Contention-based RACH fail events using Macro and XCAP — Check the event times for RACH fails ※ RACH triggered by RLF should be also counted

•

Check the detailed messages in XCAP for the fail event

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

67

13. RACH Success Rate 

Fail Cases •

Failure at MSG2 — UE didn’t receive Random Access Response (RAR) from eNB due to poor RF environments — RACH will be aborted when T300 timer is expired

▶ Check RF environment (RSRP/SINR)

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

68

13. RACH Success Rate 

Fail Cases •

Failure at MSG4 due to CT Timer expired — UE didn’t receive MSG4 from eNB within the pre-defined CT timer(56sf)

▶ Check RF environment (RSRP/SINR)

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

69

14. Attach Success Rate 

Description •

Ratio of LTE Attach Success — Attach Procedure : from RRC Connection Setup Complete with Attach Request to IMS Registration



Target Value •



Used Call Model •



>99%

VoS

Formula •

The number of Attach Success / The number of Attach Attempts — Attach Success : Receiving SIP 200 OK for IMS Registration after ‘rrcConnectionSetupComplete’ with Attach Request — Attach Attempts : Sending ‘rrcConnectionSetupComplete’ with Attach Request

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

70

14. Attach Success Rate 

Attach Procedure

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

71

14. Attach Success Rate 

Message from XCAP •

From ‘rrcConnectionSetupComplete’ message (with Attach Request) to SIP 200 OK message for IMS REGISTER

Attempts

Complete

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

72

14. Attach Success Rate 

Analysis using ActixOne and XCAP •

Check KPI value in ActixOne Report

•

Check Attach fail events in ActixOne — ActixOne does not provide the items which can check Attach fails yet

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

73

14. Attach Success Rate 

Analysis using ActixOne and XCAP •

Check Attach fail events using Macro — Check the event times for attach setup fails

— Check the event times for IMS registration fails

• •

Check the detailed messages in XCAP for the fail events Define the issues whether the issue is comes from core network(IMS) or not — Refer IMS issue code in Appendix © SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

74

14. Attach Success Rate 

Fail Cases •

Initial Attach Fail by S1 interface issue — UE didn’t receive “dlInformationTransfer" message after RRC Connection Setup Complete

▶ Check the stability of S1 interface

No message from MME after rrcConnectionSetupComplete

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

75

14. Attach Success Rate 

Fail Cases •

No Receiving TCP ACK for Sync — UE didn’t receive TCP Acknowledgement (SYNC+ACK) for IMS Register after Default Bearer Setup

▶ Check Configuration of N7K router ▶ Check the stability of Backhaul and S1 interface

No receiving ACK message for TCP Sync.

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

76

14. Attach Success Rate 

Fail Cases •

403 Forbidden — UE didn’t receive SIP 200 OK message for IMS registration due to IMS server error

※ Another core side issue codes could be analyzed with the similar way ▶ Check Core network (including IMS server)

No 200 OK for Registration

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

77

14. Attach Success Rate 

Fail Cases •

Initial Attach Fail — After RRC Connection Setup Complete, UE was released due to ‘PDN CONNECTIVITY REJECT’ message from MME ※ Reject Cause : #30 (Request rejected by Serving GW or PDN GW)

▶ Check Core network (including IMS server)

Attach Reject : PDN Connectivity Reject

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

78

15. VoLTE Setup Success Rate 

Description •

Ratio of VoLTE Call Setup Success — VoLTE Setup Procedure : from INVITE to 200 OK for INVITE



Target Value •



Used Call Model •



> 99%

VoS

Formula •

The number of VoLTE Setup Successes / The number of VoLTE Setup Attempts — VoLTE Setup Success : Receiving SIP 200 OK message for INVITE (After a dedicate bearer with QCI 1 is activated) — VoLTE Setup Attempt : Sending SIP INVIVTE message (with originating SDP)

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

79

15. VoLTE Setup Success Rate 

VoLTE Setup Procedure

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

80

15. VoLTE Setup Success Rate 

Message from XCAP •

From SIP INVITE message to SIP 200 OK message for INVITE

Attempts

P-RACK

Complete 200 OK for INVITE

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

81

15. VoLTE Setup Success Rate 

Analysis using ActixOne and XCAP •

Check KPI value in ActixOne Report

•

Check VoLTE Setup fail events in ActixOne ※ ActixOne does not implement analyzing function for VoLTE Call in detail Network Optimization

Call Event Explorer

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

82

15. VoLTE Setup Success Rate 

Analysis using ActixOne and XCAP •

Check VoLTE Setup fail events using Macro — Check the event time for VoLTE setup fails ‐ Long Setup time more than 10 seconds ‐ No Setup time ※ The number of fails found from Macro file could be different with ActixOne (Refer to Appendix)

• •

Check the detailed messages in XCAP for the fail events Define the issues whether the issue is comes from IMS or not — Refer IMS issue code in Appendix © SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

83

15. VoLTE Setup Success Rate 

Analysis using ActixOne and XCAP • • •

Check the fail events found in ActixOne in XCAP also Check the detailed messages in XCAP for the fail events Define the issues whether the issue is comes from core network or not — Refer IMS issue code in Appendix

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

84

15. VoLTE Setup Success Rate 

Fail Cases •

487 Request Terminated — UE didn’t receive SIP 200 OK message for invite due to IMS server error

※ Another core side issue code error could be analyzed with the similar way ▶ Check Core network (including IMS Server)

No 200 OK for INVITE

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

85

15. VoLTE Setup Success Rate 

Fail Cases •

500 Server Internal Error — UE didn’t receive SIP 183 Session Progress message from IMS due to server error

▶ Check Core network (including IMS Server)

No 200 OK for INVITE

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

86

15. VoLTE Setup Success Rate 

Fail Cases •

No 200 OK for PRACK — UE didn’t receive SIP 200 OK message for P-RACK when RF environment was good ※ SIP 200 Status Code : The 200 status code is the ACK message which represents the request was received and processed successfully

▶ Check Core network (including IMS Server)

No 200 OK for PRACK

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

87

15. VoLTE Setup Success Rate 

Fail Cases •

No 100 Trying — UE didn’t receive 100 Trying message after invite even through RF Environment is good

▶ Check Core network (including IMS Server)

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

88

15. VoLTE Setup Success Rate 

Fail Cases •

No 100 Trying — UE didn’t receive 100 Trying message which comes from IMS Server

▶ Check Core network (including IMS Server)

No 100 Trying between INVITE and PRACK

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

89

16. Attach Setup Time < 1.5s 

Description •

Ratio of Attach Setup Success which meets the criteria (1.5s) among the Attach Setup Success Procedure — Attach Setup Procedure : from RRC Connection Setup Complete with Attach Request to Attach Complete



Target Value •



Used Call Model •



>95%

VoS

Formula •

The number of Attach Setup Success within 1.5s / The number of Attach Setup Success — Attach Setup Success within 1.5s : Sending ‘Attach complete’ message (UL EPS MM) after ‘rrcConnectionSetupComplete’ with Attach Request within 1.5s — Attach Setup Success : Sending ‘Attach complete’ message (UL EPS MM) after ‘rrcConnectionSetupComplete’ with Attach Request

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

90

16. Attach Setup Time < 1.5s 

Attach Setup Procedure

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

91

16. Attach Setup Time < 1.5s 

Message from XCAP •

From ‘rrcConnectionSetupComplete’ (with Attach Request) to Attach Complete (UL EPS MM) Attempts

Complete

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

92

16. Attach Setup Time < 1.5s 

Analysis using ActixOne and XCAP •

Check KPI value in ActixOne Report

•

Check Attach setup time fail events in ActixOne — ActixOne does not provide the items which can check Attach setup time fail event yet

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

93

16. Attach Setup Time < 1.5s 

Analysis using ActixOne and XCAP •

Check Attach setup time fail events using Macro — Check the event time for attach setup time fails : Filter out the cases having more than 1.5s

• •

Check the detailed messages in XCAP for the fail events Define the issues whether the issue is comes from core network or not — Refer IMS issue code in Appendix

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

94

16. Attach Setup Time < 1.5s 

Fail Cases •

Security protected NAS message Delay — UE received Security protected NAS message from MME too late

▶ Check the stability of S1 Interface 16:59:59.819

16:00:02.088

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

95

17. IMS Registration Time < 1.5s (Report Only) 

Description •

Ratio of IMS Registration Success within the criteria (1.5s) among IMS Registration Success Procedure — IMS Registration Procedure : from PDN Connectivity Request to 200 OK for IMS Registration



Target Value •



Used Call Model •



>95% (Report Only)

VoS

Formula •

The number of IMS Registration within 1.5s / The number of IMS Registration Success — Attach Setup Success within 1.5s : Receiving SIP 200 OK for IMS Registration after ‘PDN Connectivity Request’ (UL EPS SM) within 1.5s — IMS Registration Success : Receiving SIP 200 OK for IMS Registration after ‘PDN Connectivity Request’ (UL EPS SM)

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

96

17. IMS Registration Time < 1.5s (Report Only) 

IMS Registration Procedure

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

97

17. IMS Registration Time < 1.5s (Report Only) 

Message from XCAP •

From ‘PDN connectivity request’ (UL EPS SM) to SIP 200 OK for IMS registration

Attempts

Complete

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

98

17. IMS Registration Time < 1.5s (Report Only) 

Analysis using ActixOne and XCAP •

Check KPI value in ActixOne Report

•

Check IMS registration setup time fail events in ActixOne — ActixOne does not provide the items which can check IMS registration setup time fails yet

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

99

17. IMS Registration Time < 1.5s (Report Only) 

Fail Cases •

504 Server Time-out — UE received 200 OK for IMS registration too late due to server error

※ Another IMS error code could be analyzed with the similar way ▶ Check Core network (including IMS Server)

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

100

18. VoLTE Setup Time < 1.5s 

Description •

Ratio of VoLTE Setup Success which meets the criteria (1.5s) among the VoLTE Setup Success Procedure — VoLTE Setup Procedure : from INVITE to 200 OK for IVITE



Target Value •



Used Call Model •



>95%

VoS

Formula •

The number of VoLTE Setup Success within 1.5s / The number of VoLTE Setup Success — VoLTE Setup Success within 1.5s : Receiving SIP 200 OK for INVITE after SIP INVITE within 1.5s — VoLTE Setup Success : Receiving SIP 200 OK for INVITE after SIP INVITE

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

101

18. VoLTE Setup Time < 1.5s 

VoLTE Setup Procedure

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

102

18. VoLTE Setup Time < 1.5s 

Message from XCAP •

From SIP INVITE message to SIP 200 OK message for INVITE

Attempts

P-RACK

Complete

200 OK for INVITE

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

103

18. VoLTE Setup Time < 1.5s 

Analysis using ActixOne and XCAP •

Check KPI value in ActixOne Report

•

Check VoLTE setup time fail events in ActixOne — ActixOne does not provide the items which can check VoLTE setup time fails yet

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

104

18. VoLTE Setup Time < 1.5s 

Analysis using ActixOne and XCAP •

Check VoLTE setup time fail events using Macro — Check the event time for VoLTE setup time fails : Filter out the cases having more than 1.5s

• •

Check the detailed messages in XCAP for the fail events Define the issues whether the issue is comes from core network or not — Refer IMS issue code in Appendix

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

105

18. VoLTE Setup Time < 1.5s 

Fail Cases •

QCI1 Activation Delay by IMS — QCI1 Bearer was activated too late

▶ Check Core network (including IMS Server) Time: 16:55.25.687

Time: 16:55.31.093

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

106

18. VoLTE Setup Time < 1.5s 

Fail Cases •

183 Session Progress Delay by IMS — UE received 183 Session Progress message with delay more than 20000ms

▶ Check Core network (including IMS Server) Time: 17:25.13.799

Time: 17:25.18.623

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

107

19. VoLTE Jitter < 40ms 

Description •

Ratio of VoLTE Jitters which meets the criteria (40ms) — VoLTE Jitters : Statistical variation of the RTP data packet inter-arrival time (RFC 3550)



Target Value •



Used Call Model •



>95%

VoL

Formula •

The number of VoLTE Jitters within 40ms / The number of VoLTE Jitters — Lost VoLTE RTP Packets will be calculated based on the received RTP Packet Sequence Number

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

108

19. VoLTE Jitter < 40ms 

Analysis using ActixOne and XCAP •

Check KPI value in ActixOne Report

•

Check VoLTE Jitter fail events in ActixOne — ActixOne does not provide the items which can check VoLTE Jitter yet

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

109

19. VoLTE Jitter < 40ms 

Analysis using XCAP • •

Check Jitter value in XCAP Check the detailed messages in XCAP for the fail events

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

110

20. VoLTE RTP Packet Loss Rate 

Description •

Ratio of Lost VoLTE RTP Packet — Causes for packet loss : network congestion, late arrivals etc.



Target Value •



Used Call Model •



Qualcomm LTE/LteAdv> PDSCH BLER Info> Total Info> Total BLER

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

117

22. Average BLER UL (Info) 

Description •



Target Value •



-

Used Call Model •



Average NACK rate in UL for the initial transmission within exclusion zone

LDL1, LUL (Merge)

Formula •

Average of UL BLER for the binned samples within exclusion zone

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

118

22. Average BLER UL (Info) 

Analysis using ActixOne and XCAP •

Check KPI value in ActixOne Report

•

Check BLER DL in XCAP — Check Graph or Table> Qualcomm LTE/LteAdv> PDSCH BLER Info> Total Info> Total BLER

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

119

22. Average BLER UL (Info) 

Fail Cases •

High UL BLER due to Interference — High UL BLER was detected withtin specific area due to interference ※ UL would be affected more severely than DL due to high sensitivity and antenna gain of eNB

▶ Check IoT level in LSM ▶ Detect Interference source using spectrum analyzer at the field ※ Interference source should be removed before CLOT

High UL BLER area

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

120

Appendix    

CLOT Report Download from ActixOne CLOT Test Scenarios Issue Code – IMS & EMM Automation Process

CLOT Report Download from ActixOne 

Drive log cycle and Cluster name are required to download CLOT report



Log On to ActixOne(http://49.40.2.69/ActixOne/) with provided login credentials •

ID/PW : samsung1/samsung1

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

122

CLOT Report Download from ActixOne 

Follow the below steps sequentially

1. Click on Network optimization module

2. Click on Task Management tab

3. Click on filter tab

4. Select & click create filter

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

123

CLOT Report Download from ActixOne 

Follow the below steps sequentially

5. Put the filter Name you want

6. Keep one row for filter , select other row and delete with clicking “delete row” b utton

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

124

CLOT Report Download from ActixOne 

Follow the below steps sequentially

7. From “Field “ column sel ect “description” and from “operator” column select ‘ contains’ and put Cluster n ame of required report whi ch you want to download in “Value” column and the click on “Save “

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

125

CLOT Report Download from ActixOne 

Follow the below steps sequentially

8. Right click on Task management window

9. click on Reports

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

126

CLOT Report Download from ActixOne 

Follow the below steps sequentially

10. Double click on “R JIL CLOT Acceptance R eports “ template

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

127

CLOT Report Download from ActixOne 

Follow the below steps sequentially

11. Fill the Cluster Name , Task ID detail and click on “Run”

Task ID

Cluster Name

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

128

CLOT Report Download from ActixOne 

Follow the below steps sequentially

12. Save the CLOT report

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

129

CLOT Test Scenario 

Drive Test • •

It will be used 5 handsets to perform five tests using Accuver DM tool installed in mobile handset Each handset will perform one of the test given below 1. 2. 3. 4. 5.

•

FTP DL 1 FTP DL 2 FTP UL VoLTE Long VoLTE Short

All test scenarios are already predefined in Accuver DM tool — Drive test engineer will select different test scenarios per handset

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

130

Issue Codes 

IMS Issue Code for VoLTE

NO

Codes

Description

1

403

Forbidden

2

487

Request Terminated

3

500

Internal Server Error

4

501

Not Implemented

5

502

Bad Gateway

6

503

Service Unavailable

7

504

Service Time-out

8

505

SIP Version Not Supported

9

513

Message Too Large

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

131

Issue Codes 

PDN Connectivity Reject •

If the PDN CONNECTIVITY REQUEST cannot be accepted by the network, then MME shall send a PDN CONNECTIVITY REJECT message to the UE. — This message shall contain an ESM cause value which indicates the reason for rejecting the UE requested PDN connectivity. — The ESM cause IE typically indicates one of the following ESM cause values; ‐ Refer to section 6.5.1.4 in the 3GPP TS 24.301 for detailed information #

EMM Cause

#

EMM Cause

#8

operator determined barring

#35:

PTI already in use

#26

insufficient resources

#38

network failure

#27

missing or unknown APN

#50

PDN type IPv4 only allowed

#28

unknown PDN type

#51

PDN type IPv6 only allowed

#29

user authentication failed

#53:

ESM information not received

#30

request rejected by Serving GW or PDN GW

#54

PDN connection does not exist

#31

request rejected, unspecified

#55

multiple PDN connections for a given APN not allowed

#32

service option not supported

#95

111: protocol errors

#33

requested service option not subscribed

#112

APN restriction value incompatible with active EPS bearer context.

#34

service option temporarily out of order

© SAMSUNG Electronics Co., Ltd. Confidential and Proprietary

132

Automation Process 

Automation Process Improvement •

XCAL-Mobile → FMS → ActixOne ※ XCAL-Mobile : UE Diagnostic Monitor App. in S4mini, FMS : Fleet Management System, ActixOne : Automatic field test log analysis and report generation tool

XCALMobile Content



Collect Log file

File Type



DRX

Content

•

Management Call Scenario, IMEI & Log file

File Type

•

Convert DRX → DRM

Storage

•

600GB → Upgrade 6TB (10DEC ~ 15DEC2014)

Processing

•

Processing 1 = max 2,000 Devices

Save file

•

The type of Log file is DRX saved 3 month (User option)

FMS

DRX

DRM

(Storage 1 / Processing 1)

A

(Storage 1 / Processing 28)

B DRX

Backup Server

Point

ActixOne

Content

•

Save Log files

File Type

•

DRX

Storage

•

10TB

Save file

•

The type of Log file is DRX saved (Not delete)

Content

•

Generation SCFT & CLOT report Provide Plot & Map

File Type

•

DRM

Storage

•

7TB → upgraded 60TB (15JAN2015)

Processing sever

• •

8 → 28 upgraded 1 = 16 files processing

Improvement Plan

•

Target

Owner

A

 To add file check and feedback process in FMS when engineer upload log files.

End of Feb.

Accuver

B-1

 To enhance link bandwidth between FMS and ActixOne to raise file transfer speed up.

ASAP

RJIL

B-2

SAMSUNG Electronics Co., Ltd. Confidential and Proprietary TBD  Need to add status managing ©system between FMS and ActixOne.

Accuver, RJIL

133

Copyright and Confidentiality Copyright © 2014 © SAMSUNG Electronics Co., Ltd. SAMSUNG Electronics reserves the right to make changes to the specifications of the products detailed in this document at any time without notice and obligation to notify any person of such changes. Information in this document is proprietary to SAMSUNG Electronics Co., Ltd. No information contained here may be copied, translated, transcribed or duplicated by any form without the prior written consent of SAMSUNG Electronics.