RU40 TS_SRAN_SW_0002_I5.pdf

Technical Support Note TS-SRAN-SW-0002

3G BTS Critical Problems WCDMA Base Stations SW RU30, RU40  Approval date: 07.05.2014 This document contains following type of information Informative

x

Preventive Corrective Additional categorization Urgent

x

Security Release Upgrade SW Update Information is classified as Internal Public

x

Customer Specific

Nokia Solutions and Networks is continually striving to reduce the adverse environmental effects of its products and services. We would like to encourage you as our customers and users to join us in working towards a cleaner, safer environment. Please recycle product packaging and follow the recommendations recommendations for power use and proper disposal of our products and their components. If you should have questions regarding our Environmental Policy or any of the environmental services we offer, please contact us at Nokia Solutions and Net works for additional information.

TS-SRAN-SW-0002 Copyright © 2014 Nokia Solutions and Networks. All rights reserved. CONFIDENTIAL APPROVED 5.0

Page 1

Table of Contents 1. 2. 3. 4. 5.

Validity .................................................................................................................... 3 Compatibility / Dependencies to other products ...................................................... 3 Keywords ................................................................................................................ 3 Summary ................................................................................................................ 3 Detailed description ................................................................................................. 4 5.1 (New) A problem in RF module SW may cause unexpected reset of RF module after 65 days in operation due to increased memory consumption .......... 4 5.2 Timing Over Packet (IEEE 1588v2) does not synchronize FSMF sufficiently ................................................................................................................... 7 5.3 Site configuration could be lost, if SCF is downloaded but not activated before proceeding BTS SW upgrade .............................................................................. 8 5.4  A problem in WCDMA BTS SW may cause unexpected reset of RF Sharing BTS site (WCDMA+GSM / WCDMA+LTE) due to memory leakage .................... 9 5.5 Incorrect BTS SW version for UltraSite and MetroSite WCDMA BTS in WN8.0.0.3 release build WN8.0 1301_484_61 .................................................. 10 6. Note ...................................................................................................................... 10 7. References ........................................................................................................... 10 Contact: Contact your local Nokia Solutions and Networks support

Summary of changes: 07-05-2014

5.0

06-05-2014 19-03-2014 14-02-2014 03-02-2014

4.0 3.0 2.0 1.0

Approved. Page 5: table corrected, updated estimation for the next correction delivery Approved Approved Approved Approved

TS-SRAN-SW-0002 Copyright © 2014 Nokia Solutions and Networks. All rights reserved. CONFIDENTIAL APPROVED 5.0

Page 2

Purpose This Technical Support Note is for instant supply of information, which explains critical problem situations and instructs available preventative/recovery procedures. Announcing of a possible critical change in sw or requirements for feature may be covered.

1. VALIDITY Product

System release

Flexi Multiradio WCDMA BTS SW UltraSite WCDMA BTS SW MetroSite WCDMA BTS SW Flexi Lite WCDMA BTS

RU30 RU40 WL7.0 WL8.0

2. COMPATIBILITY / DEPENDENCIES TO OTHER PRODUCTS See detailed description for possible compatibility requirements with the other RAN elements.

3. KEYWORDS Flexi Multiradio BTS, Flexi Multiradio 10, UltraSite WCDMA, MetroSite WCDMA, critical, outage, 3G BTS, NodeB, WCDMA

4. SUMMARY This document contains information of the critical faults in 3G BTS software releases, which may cause outages in traffic service, or initiate issues requiring immediate attention in network operation and maintenance. The document outlines instructions for each case how the problem situation can be prevented or recovered.

TS-SRAN-SW-0002 Copyright © 2014 Nokia Solutions and Networks. All rights reserved. CONFIDENTIAL APPROVED 5.0

Page 3

5. DETAILED DESCRIPTION 5.1 (New)   A problem in RF module SW may cause unexpected reset of RF module after 65 days in operation due to increased memory consumption Product:

Flexi Multiradio RF modules FRGP, FRIE, FRKA, FXCA, FXDA, FXFA, FXFB, FXFC, FRGS, FRGT, FXDB

Exists in:

WN7.0 2.3, WN7.0 2.3.1, WN7.0 3.0.1, WN7.0 3.0.2 WN8.0 1.0, WN8.0 1.1, WN8.0 2.1

Corrected in:

WN8.0 3.0, WN8.0 3.1 (only configurations with FSMB, FSIA/FSIB) WN8.0 3.0, WN8.0 3.1 WN8.0 3.2 for the configurations with FSMB, FSIA/ FSIB

Description: In radio module the RF SW is periodically checking climate control related information, as like information and operational state of the module cooling fans (fan type, ID, rotation speed). A defect in the RF SW area induces that the fan information got in the polling procedure gets accumulated and causes increasing consumption of system memory. As a consequence, over the time, allocation of system memory may fail due to lack of large enough continuous memory block, which results a kernel error and sudden reset of the RF module. The reset clears the problematic situation temporarily as the previous fan information is cleared and all reserved memory blocks become released. After the reset, RF module proceeds fully autonomously, no any manual operation is required and all services are recovered automatically. The time required to get into the problematic memory state is depending on number of fans in the FR module. NSN has been recently reported that occasionally numbers of Failure in RP3 interface –alarms were seen in a cluster due to RF module reset in some sites after a latency period of: -

84 days for FRGP, equipped with two cooling fans 65 days for FXDA and FXFA, equipped with three cooling fans

System Impact: Failure in RP3 interface –alarms may occasionally occur in a BTS cluster after a certain period of operation since the last restart e.g. SW release upgrade.  Accessibility failures, call drop rate and cell availability may be shortly impacted in the BTS cluster at the time of RF module resets. Workaround: RF module block/unblock or BTS restart is scheduled to be set before reaching the limit of 84 / 65 days in operation since the last restart of RF module . Correction: The fan related information is overwritten into the same memory allocation during the polling instead of being accumulated. TS-SRAN-SW-0002 Copyright © 2014 Nokia Solutions and Networks. All rights reserved. CONFIDENTIAL APPROVED 5.0

Page 4

Correction delivery: WN8.0 3.2 introduces correction for the Rel1- Rel2 mixed system module configurations and Flexi Indoor FSIA/FSIB system modules. LE for WN8.0 3.2 delivery is May 16th, 2014. Detailed coverage of fault across the SW releases, RF module versions and BTS configurations are described in the table below: Release/Build

FSM rel1 + FSM rel2 mixed configurations (FSMB + FSMC/D/E)

FSM rel2 (FSMC/FSMD/FSME)

FSM rel3 (FSMF)

WN8.0 3.1 WN8.0_1308_555_54

 Affected RFM HW rel. 2.1 / 2.2

Failure not existing

Failure not existing

WN8.0 3.0 WN8.0_1308_437_11

 Affected RFM HW rel. 2.1 / 2.2

Failure not existing

Failure not existing

WN8.0 2.1 WN8.0_1308_437_08

 Affected RFM HW rel. 2.1 / 2.2 / 2.3

 Affected RFM HW rel. 2.1/ 2.2 / 2.3

 Affected RFM HW rel. 2.1 / 2.2 / 2.3

WN8.0 2.0 WN8.0_1308_437_03

NA

NA

Failure not existing

WN8.0 1.1 WN8.0_1301_842_29

 Affected RFM HW rel. 2.1 / 2.2 / 2.3

 Affected RFM HW rel. 2.1 / 2.2 / 2.3

NA

WN8.0 1.0 WN8.0_1301_842_09

 Affected RFM HW rel. 2.1 / 2.2 / 2.3

 Affected RFM HW rel. 2.1 / 2.2 / 2.3

NA

WN8.0 0.0.4 WN8.0_1301_484_63

Failure not existing

Failure not existing

NA

WN8.0 0.0.3 WN8.0_1301_484_61

Failure not existing

Failure not existing

NA

WN8.0 0.0.2 WN8.0_1301_484_58

Failure not existing

Failure not existing

NA

WN8.0 WN8.0_1301_484_15

Failure not existing

Failure not existing

NA

WN7.0 3.0.2 WN7.0_1302_389_39

NA

NA

 Affected RFM HW rel. 2.3

WN7.0 3.0.1 WN7.0_1302_389_37

NA

NA

 Affected RFM HW rel. 2.3

WN7.0 3.0 WN7.0_1302_375_00

NA

NA

 Affected RFM HW rel. 2.3

WN7.0 2.3.1 WN7.0_1207_1056_18

 Affected RFM HW rel. 2.1 / 2.2

 Affected RFM HW rel. 2.1 / 2.2

NA

WN7.0 2.3 WN7.0_1207_1056_02

 Affected RFM HW rel. 2.1 / 2.2

 Affected RFM HW rel. 2.1 / 2.2

NA



NA – not applicable (the release doesn’t include SW for  a WBTS with the specific Flexi System Module)



For requesting the detailed i nformation of a customer specific SW delivery (OnTop releases), please contact to NSN Customer Support



RF Module version type vs. RFM HW releases are described below:

TS-SRAN-SW-0002 Copyright © 2014 Nokia Solutions and Networks. All rights reserved. CONFIDENTIAL APPROVED 5.0

Page 5

Multiradio RF module HW

Multiradio RF module HW

Multiradio RF module HW

release 2.1

release 2.2

release 2.3

FRGP 472100A

FRGP 472100B

FRGS 472441A

FRIE 471895A

FXCA 472142A

FRGT 472442A

FRKA 472111A

FXDA 472083A

FXDB 472573A

FXFA 472166A

FXFC 472679A

FXFB 472569A FXDJ 472143A FRIE 471895A.202

TS-SRAN-SW-0002 Copyright © 2014 Nokia Solutions and Networks. All rights reserved. CONFIDENTIAL APPROVED 5.0

Page 6

5.2 Timing Over Packet (IEEE 1588v2) does not synchronize FSMF sufficiently Product: Exists in: Corrected in:

Flexi Multiradio 10 WCDMA BTS (FSMF) Flexi Lite WCDMA BTS All WN7.0 and WN8.0 releases and WL7.0 WN8.0.3.1, WN8.0.4.0, WL8.0

Description: Timing Over Packet (ToP) can be used for providing synchronization to the Flexi WCDMA BTS via an Ethernet interface. Synchronization is needed to keep all Flexi WCDMA BTS in the same frequency in order to assure correctly working inter-nodeB handovers. In Flexi Multiradio 10 WCDMA BTS (FSMF) synchronization by ToP is not working correctly. Even if the ToP status is shown as locked, the oscillator of FSMF is not tuning to the input frequency with sufficient accuracy. Because of the fault the oscillator frequency may vary more than allowed and the 3GPP requirement for maximum frequency deviation cannot be guaranteed. The fault might not be reflected in KPI immediately or even after longer time, since the deviation of the oscillator is very slow and depends additionally very much on the internal temperature of the system. System Impact: The frequency is set in the factory to a default value that is close to real operation in the field. After first startup the deviation will start, but it may be close enough to other Flexi WCDMA BTS to allow successful handovers. However, with the time the deviation can increase until it has an impact. After that inter-nodeB handover failures can increase until it is failing permanently. Workaround: Only possible option for a workaround is selection of a different synchronization source (e.g. GPS, synchronous ETH or external synchronization interface) if that is available on site or TDM source if an additional TRS module FTIF is installed which offers a TDM input. Correction: Correction will be delivered in WN8.0 3.1, WN8.0 4.0 and WL8.0 release.

TS-SRAN-SW-0002 Copyright © 2014 Nokia Solutions and Networks. All rights reserved. CONFIDENTIAL APPROVED 5.0

Page 7

5.3 Site configuration could be lost, if SCF is downloaded but not activated before proceeding BTS SW upgrade Product: Exists in: Corrected in:

Flexi Multiradio WCDMA BTS, Flexi Direct All WN7.0 and WN8.0 releases WN8.0.3.0 (FSME/FSMD/FSMC/FSMB) WN8.0.4.0 (FSMF)

Description:  A commissioning change for WBTS can be performed b y OMS/NetAct with two separate tasks; first the SCF is downloaded to WBTS, and next the file is activated. If the activation step is not performed at once with download, the SCF however remains stored on the WBTS. The file does not have any effect, if during the next commissioning, locally or remotely, another SCF is downloaded and activated. In this procedure the previously stored file, but not activated yet, is considered as obsolete and then overwritten. However, in case that the stored SCF is not being activated and next an SW update is proceeded, the stored SCF file becomes erroneously activated during the SW update process. As a consequence, the configuration in operation gets replaced by the configuration in the previously downloaded and stored SCF. System Impact: In case the SCF file downloaded in the past does not for instance cover possibly performed manual changes, or it’s configuration is different to the configuration currently in operation, then there is risk for that the new configuration erroneously activated in SW upgrade can be incompatible with required settings. This sudden unplanned change can have a major impact either on TRS ’s or OAM’s interfaces and connections. In such a case, there is risk to cause WBTS fall in to off air condition or up to lose remote connection to the WBTS site.  Any SW update with WN7.0 or WN8.0 releases has the risk to initiate WBTS run into this condition, if an invalid SCF have been downloaded but not activated in the past. Workaround: To avoid the problem in case that download of an invalid or a not up-to-date SCF file has been performed, it is required to download and activate of SCF with a valid configuration before proceeding any SW update to the WBTS. Correction: The potential risk caused by the SW flaw has been fixed. Correction will be delivered in  

WN8.0 3.0 release for BTS configurations with SME/FSMD/FSMC/FSMB WN8.0 4.0 release for BTS configurations with FSMF

When upgrading to the corrective BTS SW release above 8.0.3.0 or any later BTS SW, the stored and not activated SCF file will be now ignored and removed. References WN8.0 3.0 Release Documentation: WN8.0 3.0 CN Document , CN-id: 80354

TS-SRAN-SW-0002 Copyright © 2014 Nokia Solutions and Networks. All rights reserved. CONFIDENTIAL APPROVED 5.0

Page 8

5.4 A problem in WCDMA BTS SW may cause unexpected reset of RF Sharing BTS site (WCDMA+GSM / WCDMA+LTE) due to memory leakage Product:

RF Sharing configurations with Flexi Multiradio System Module FSMC/FSMD/FSME

Exists in:

All WN7.0 releases, excluding WN7.0 2.3 WN8.0 and WN8.0 0.x releases

Corrected in:

WN7.0 2.3, WN8.0 1.0 and later releases

Description: In RF Sharing configuration WCDMA system module performs polling for presence and condition of peer technology’s (GSM or LTE) system module. This is done periodically over the optical RP3 interface used for synchronization between the system modules. A fault in the SW polling procedure causes slow filling and fragmentizing of the memory.  As a consequence, after a prolonged period, spontaneous BTS reset ma y occur due to lack of free memory blocks for normal SW processes. This is more prone to occur when an O&M task (SW download, SW activation, configuration change, plan update etc.) is performed and thus additional blocks are requested on memory already running on the limit. BTS restart clears the problematic situation temporarily as all reserved memory blocks become released and fragmentation is cleared. In case of the spontaneous BTS reset occurs, no any manual operation is needed and all services are recovered automatically. The time required to get into the problematic memory state is estimated to be approx. 6 months of normal BTS operation since the last restart of the BTS. However, this is impacted by the number of O&M operations performed, such as configuration changes, license management task etc. System Impact: When performing mass O&M operation, several RF shared BTS may enter to an unexpected site reset. Also the peer (GSM or LTE) technology will be affected. Workaround: Before performing any O&M tasks (SW update, plan update, re-licensing) NSN recommends to schedule BTS reset for all RF Sharing sites, which have been not proceeded BTS reset during the previous 4+ months. Correction: FSMr2 RP3 handler section has been fixed and delivered in WN7.0 2.3 release (Change Note: WN701144) WN8.0 1.0 release (Change Note: WN80194) and later RU40 BTS SW releases NSN recommends scheduling BTS reset for the RF Sharing sites prior upload of the corrective SW release.  

TS-SRAN-SW-0002 Copyright © 2014 Nokia Solutions and Networks. All rights reserved. CONFIDENTIAL APPROVED 5.0

Page 9

5.5 Incorrect BTS SW version for UltraSite and MetroSite WCDMA BTS in WN8.0.0.3 release build WN8.0 1301_484_61 Product: Exists in: Corrected in:

UltraSite WCDMA BTS, MetroSite WCDMA BTS WN8.0.0.3, build WN8.0 1301_484_61 WN8.0.0.4, build WN8.0 1301_484_63

Description: th The WN8.0.0.3 priority package released on 20  September 2013 has an incorrect sw version for UltraSite BTS and MetroSite BTS. The release build WN8.0 1301_484_61 contains obsolete sw release WN6.0 22.11-139_N for UltraSite platform, whereas the required BTS SW version is WN6.0_22.12298_Z1.  A new build WN8.0 1301_484_63 with a correct UltraSite BTS SW version has been re-created and released as WN8.0.0.4 BTS SW release. There are no any other corrections or changes introduced in WN8.0.0.4 release. th The WN8.0.0.4 BTS SW release is published on 4  October 2013 in NOLS. The issue does not concern Flexi WCMA BTS platform. SW for Flexi WCDMA BTS is equal in the WN8.0.0.3 and WN8.0.0.4 releases. Solution / Workaround: The WN8.0.0.3 release (build WN8.0 1301_484_61) is not recommended upgraded to UltraSite BTS and MetroSite BTS. In network containing UltraSite WCDMA BTS and MetroSite WCDMA BTS, it is recommended to replace WN8.0.0.3 release by the WN8.0.0.4 release in SW Management tools for preventing erroneous download of WN8.0.0.3 to UltraSite BTS and MetroSite BTS. For the network containing only Flexi BTS, there is no difference which of the SW releases, WN8.0.0.3 or WN8.0.0.4, is used. System Impact: UltraSite BTS and MetroSite BTS will operate with an incorrect and obsolete BTS SW release.

6. NOTE 7. REFERENCES

TS-SRAN-SW-0002 Copyright © 2014 Nokia Solutions and Networks. All rights reserved. CONFIDENTIAL APPROVED 5.0

Page 10

Disclaimer

The information in this document is subject to change without notice and describes only the product defined in the introduction of this documentation. This documentation is intended for the use of Nokia Solutions and Networks customers only for the purposes of the agreement under which the document is submitted, and no part of i t may be used, reproduced, modified or transmitted in any form or means without the prior written permission of Nokia Solutions and Networks. The documentation has been prepared to be used by professional and properly trained personnel, and the customer assumes full responsibility when using i t. Nokia Solutions and Networks welcomes customer comments as part of the process of continuous development and improvement of the documentation. The information or statements given in this documentation concerning the suitability, capacity, or performance of the mentioned hardware or software products are given “as is” and all liability arising in connection with such hardware or software produc ts shall be defined conclusively and finally in a separate agreement between Nokia Solutions and Networks and the customer. However, Nokia Solutions and Networks has made all reasonable efforts to ensure that the i nstructions contained in the document are adequate and free of material errors and omissions. Nokia Solutions and Networks will, if deemed necessary by Nokia Solutions and Networks, explain issues which may not be covered by the document. Nokia Solutions and Networks will correct errors in this documentation as soon as possible. IN NO EVENT WILL NOKIA SOLUTIONS AND NETWORKS BE LIABLE FOR ERRORS IN THIS DOCUMENTATION OR FOR ANY DAMAGES, INCLUDING BUT NOT LIMITED TO SPECIAL, DIRECT, INDIRECT, INCIDENTAL OR CONSEQUENTIAL OR ANY LOSSES, SUCH AS BUT NOT LIMITED TO LOSS OF PROFIT, REVENUE, BUSINESS INTERRUPTION, BUSINESS OPPORTUNITY OR DATA,THAT MAY ARISE FROM THE USE OF THIS DOCUMENT OR THE INFORMATION IN IT. This documentation and the product it describes are considered protected by copyrights and other intellectual property rights according to the applicable laws. NSN is a trademark of Nokia Solutions and Networks. Nokia is a registered trademark of Nokia Corporation. Other product names mentioned in this document may be trademarks of their respective owners, and they are mentioned for identification purposes only. Copyright © 2014 Nokia Solutions and Networks. All rights reserved.

TS-SRAN-SW-0002 Copyright © 2014 Nokia Solutions and Networks. All rights reserved. CONFIDENTIAL APPROVED 5.0

Page 11