Scalable Communication Transport Solutions Over Optical Networks

618

Scalable Communication Transport Solutions Over Optical Networks

Working Group D2.35

May 2015

SCALABLE COMMUNICATION TRANSPORT SOLUTIONS OVER OPTICAL NETWORKS Type WG D2.35 Members J.SANDHAM Convenor (IE), M.SCHATZ Secretary (CH), M.ACACIA (BE), F.CASTRO (ES), J.DARNE (ES), J.FEIJOO MARTINEZ (ES), M.FLOHIL (NL), R.IRONS-MCLEAN (UK), M.JANSSEN (NL), A.MOAINI (FR), T.V.PEDERSEN (NO), H.RIIS (DK), A.RUNESSON (SE), S.TANNER (FI)

Corresponding Members C.DI PALMA (AR), C.EVERITT (AU), J.MENDES (PT), P.SCHWYTER (CH), M.SEEWALD (DE), A.SILFVERBERG (FI), V.TAN (AU) Copyright © 2015 “Ownership of a CIGRE publication, whether in paper form or on electronic support only infers right of use for personal purposes. Other uses are prohibited, except if explicitly agreed by CIGRE, total or partial reproduction of the publication for use other than personal and transfer to a third party; hence circulation on any intranet or other company network is forbidden”. Disclaimer notice “CIGRE gives no warranty or assurance about the contents of this publication, nor does it accept any responsibility, as to the accuracy or exhaustiveness of the information. All implied warranties and conditions are excluded to the maximum extent permitted by law”.

ISBN : 978-2-85873-320-0 1|Page

SCALABLE COMMUNICATION TRANSPORT SOLUTIONS OVER OPTICAL NETWORKS Table of Contents 1 INTRODUCTION................................................................................................................ 8 2 REQUIREMENTS FROM USERS .......................................................................................... 9 2.1 Survey Results .................................................................................................................................... 9 2.1.1 Substation applications using IP ............................................................................................ 11 2.1.2 Main operational challenges using IP as communication transport solutions ................ 12 2.1.3 Main psychological barriers with using IP protocols in Substation applications .......... 13 2.1.4 Are all applications compatible with IP at the moment? .................................................. 14 2.1.5 Prediction to migration of all operational communications into IP ................................. 14 2.1.6 How to deal with legacy protocols and equipment? ........................................................ 15 2.1.7 Requisites and concerns for the telecommunication network ........................................... 16 2.1.8 Most promising technologies to provide communications in the access network? ........ 17 2.1.9 Most promising technologies to provide communications in the core network?............ 18 2.1.10 Should the IP network be for operational service or also for corporate services? . 19 2.1.11 Survey Conclusions ................................................................................................................ 19 2.2 List of Services ................................................................................................................................ 20 3 GENERAL CONSIDERATIONS ......................................................................................... 22 3.1 Why are Utilities migrating? ........................................................................................................ 22 3.2 Migration considerations............................................................................................................... 24 3.2.1 Timing ......................................................................................................................................... 26 3.2.4 Communication Requirements Decomposition ..................................................................... 26 3.2.5 Security ...................................................................................................................................... 27 3.2.6 Communication Network Management ................................................................................ 28 3.2.7 IPv4 and IPv6 co-existence .................................................................................................... 29 3.2.8 Segmentation and Virtualization .......................................................................................... 29 3.2.9 Scalability ................................................................................................................................. 30 3.2.10 Availability ............................................................................................................................. 30 4 ASSESMENT OF TECHNOLOGIES .................................................................................... 31 4.1 Synchronous Digital Hierarchy (SDH) ......................................................................................... 32 Life Cycle ............................................................................................................................................. 32 Scalability ............................................................................................................................................ 32 Capability to support EPU Services ................................................................................................ 33 Ease of Implementation and Operation ......................................................................................... 33 Interoperability (other services/technologies)............................................................................... 33 Redundancy, Availability and Reliability ...................................................................................... 34 Quality of Service (QoS)................................................................................................................... 35 Network Management System and Overall Manageability (NMS) ......................................... 35 3|Page

4.2 Dense Wavelength Division Multiplexing & Coarse Wavelength Division Multiplexing . 36 Life Cycle ............................................................................................................................................. 36 Scalability ............................................................................................................................................ 36 Capability to support EPU Services ................................................................................................ 39 Ease of Implementation and Operation ......................................................................................... 39 Interoperability (other services/technologies)............................................................................... 40 Redundancy, Availability and Reliability ...................................................................................... 41 Quality of Service (QoS)................................................................................................................... 41 Network Management System and Overall Manageability (NMS) ......................................... 42 CWDM versus DWDM ....................................................................................................................... 42 4.3 Optical Transport Network (OTN) .............................................................................................. 43 Life Cycle ............................................................................................................................................. 43 Scalability ............................................................................................................................................ 43 Capability to support EPU Services ................................................................................................ 45 Ease of Implementation and Operation ......................................................................................... 46 Interoperability (other services/technologies)............................................................................... 46 Redundancy, Availability and Reliability ...................................................................................... 46 Quality of Service (QoS)................................................................................................................... 46 Network Management System and Overall Manageability (NMS) ......................................... 46 4.4 Dynamic Internet protocol based Multiprotocol Label Switching (IP/MPLS) ..................... 47 Life Cycle ............................................................................................................................................. 47 Scalability ............................................................................................................................................ 47 Capability to support EPU Services (as defined in chapter 2) .................................................. 47 Ease of Implementation and Operation ......................................................................................... 48 Interoperability (other services/technologies)............................................................................... 48 Redundancy, Availability and Reliability ...................................................................................... 48 Quality of Service (QoS)................................................................................................................... 49 Network Management System and Overall Manageability (NMS) ......................................... 49 4.5 Static Multiprotocol Label Switching – Transport Profile (MPLS-TP) .................................... 50 Life Cycle ............................................................................................................................................. 50 Scalability ............................................................................................................................................ 50 Capability to support EPU Services (as defined in chapter 2) .................................................. 50 Ease of Implementation and Operation ......................................................................................... 51 Interoperability (other services/technologies)............................................................................... 51 Redundancy, Availability and Reliability ...................................................................................... 51 Quality of Service (QoS)................................................................................................................... 52 Network Management System and Overall Manageability (NMS) ......................................... 52 4.7 Provider Backbone Bridging (PBB) ............................................................................................. 54 4.8 Ethernet ............................................................................................................................................ 54 Life Cycle ............................................................................................................................................. 54 Scalability ............................................................................................................................................ 55 Capability to support EPU Services (as defined in chapter 2) .................................................. 55 Ease of Implementation and Operation ......................................................................................... 55 Interoperability (other services/technologies)............................................................................... 55 Redundancy, Availability and Reliability ...................................................................................... 55 Quality of Service (QoS)................................................................................................................... 55 Network Management System and Overall Manageability (NMS) ......................................... 55 4.9 Comparison matrix......................................................................................................................... 56 5 CONCLUSIONS AND PROPOSAL FOR FUTURE WORK .................................................. 58 4|Page

Survey ...................................................................................................................................................... 58 Services.................................................................................................................................................... 58 Technologies ........................................................................................................................................... 58 Future Works 1 ...................................................................................................................................... 60 Future Works 2 ...................................................................................................................................... 60 Future Works 3 ...................................................................................................................................... 61 ANNEX 1 SURVEY D2.35 ................................................................................................... 62 ANNEX 2 SURVEY D2.28 IP-BASED SUBSTATION APPLICATIONS.................................... 71 ANNEX 3 WGD2.35 SURVEY RESULTS............................................................................... 74 1 Substation applications using IP ...................................................................................................... 74 2 Substation applications eligible for use with IP ........................................................................... 76 3 Applications outside the substation eligible for use with IP ....................................................... 78 4 Main operational challenges using IP a communication transport solutions ........................... 78 5 Main psychological barriers with using IP protocols in Substation applications .................... 79 6 Are all applications compatible with IP at the moment? ............................................................ 80 7 Prediction to migration of all operational communications into IP ........................................... 81 8 Percentage of the existing applications using IP as communication transport solution? ...... 83 9 How to deal with legacy protocols and equipment? .................................................................. 83 10 Requisites and concerns for the telecommunication network................................................... 84 11 Most promising technologies are to provide communications in the access network? ........ 86 12 Most promising technologies to provide communications in the core network? ................... 86 13 Should the IP network be for operational service or also be for corporate services? ...... 87 14 What type of scalability needs to be addressed? .................................................................. 88 ANNEX 4 NETWORK TIMING ............................................................................................ 90 Synchronisation Types........................................................................................................................... 91 IRIG-B Overview.................................................................................................................................... 92 Global Positioning System (GPS) Overview .................................................................................... 92 One Pulse Per Second (1PPS) Overview .......................................................................................... 94 NTP/SNTP Overview ............................................................................................................................ 94 IEEE 1588 PTP / IEC 61588 PTP ........................................................................................................ 94 Versions of 1588 ................................................................................................................................ 95 NTP vs. SNTP vs. PTP: A Summary...................................................................................................... 95

5|Page

Figures Figure 1 Companies represented in the survey responses ............................................................................. 10 Figure 2 Countries represented in the survey responses ................................................................................ 10 Figure 3 Trends in substation applications using IP ........................................................................................ 11 Figure 4 Trends in operational challenges ...................................................................................................... 12 Figure 5 Trends in psychological barriers ........................................................................................................ 13 Figure 6 Trends in current IP compatibility ...................................................................................................... 14 Figure 7 Trends in estimation of all operational traffic through IP ................................................................... 15 Figure 8 Trends in how to deal with legacy equipment and protocols ............................................................ 15 Figure 9 Trends in requisites and concerns .................................................................................................... 16 Figure 10 Trends in most promising access network technologies ................................................................. 17 Figure 11 Trends in most promising core network technologies ..................................................................... 18 Figure 12 Trends in network separation .......................................................................................................... 19 Figure 13 Traditional Grid Communications Architecture ................................................................................ 23 Figure 14 Tiered approach for New grid applications ...................................................................................... 23 Figure 15 Shift from TDM to packet ................................................................................................................. 24 Figure 16 Migration decision review process .................................................................................................. 25 Figure 17 Communications decomposition table for packet oriented traffic .................................................... 26 Figure 19 SDH Capacity Comparison ............................................................................................................. 32 Figure 18 SDH Add-Drop-Multiplexer .............................................................................................................. 33 Figure 20 SDH Sub-Network Connection Protection ...................................................................................... 34 Figure 22 Optical spectra in fiber ..................................................................................................................... 37 Figure 23 examples of Add-Drop-Multiplexer .................................................................................................. 37 Figure 24 Reconfigurable Optical Add Drop Multiplexer ................................................................................. 38 Figure 25 Working area for light source and receiver ..................................................................................... 38 Figure 21 WDM Multiplexing............................................................................................................................ 39 Figure 26 OTN Architecture ............................................................................................................................. 44 Figure 27 OTN Capacities ............................................................................................................................... 45 Figure 28 MPLS-TP Hierarchy ........................................................................................................................ 51 Figure 29 Protection configuration ................................................................................................................... 52 Figure 30: Substation applications using IP .................................................................................................... 75 Figure 31: Applications eligible for using IP..................................................................................................... 77 Figure 32: Operational challenges ................................................................................................................... 79 Figure 33: Psychological Barriers .................................................................................................................... 80 Figure 34: All applications IP compatible?....................................................................................................... 81 Figure 35: Predication of All operational traffic through IP .............................................................................. 82 Figure 36 Percentage of existing applications using IP ................................................................................... 83 Figure 37How to deal with legacy protocols/equipment .................................................................................. 84 Figure 38 Requisites and concerns for using only IP ...................................................................................... 85 Figure 39 Most promising access network technology .................................................................................... 86 Figure 40 Most promising core network technology ........................................................................................ 87

6|Page

Figure 41 Seperate operational services from corporate services? ................................................................ 88 Figure 42 What type of scalability to address? ................................................................................................ 89 Figure 43 Applications sensitive to time and frequency .................................................................................. 91 Figure 44 Time synchronisation application .................................................................................................... 92

Tables Table 1: (Future) IP usage ............................................................................................................................... 78

7|Page

1 INTRODUCTION The introduction of smart applications in the Electrical Power Utility (EPU) and consequent dispersed intelligence result in a tremendous growth of information exchange across the power system. This implies, in many cases, a change of scale in the requirements of the telecommunication infrastructure and often the deployment of a core data transport network. This may be implemented through a number of different technologies and architectures. The present network of most power utilities is extensively composed of TDM (e.g. PDH/SDH) technology. Packet communication and in particular Ethernet connections are growing very fast and may bring the necessity to adapt and /or replace network technologies. This Technical Brochure aims to identify and analyze solutions and migration plans in the light of data network technology evolutions, new application requirements and EPU’s capability to maintain the system’s operation. During 2009 and 2010, Cigré working group D2.28 conducted a survey amongst Cigré members to identify the current and expected future use of IP networks within Electric Power Utilities. For this Technical Brochure a follow-up survey was conducted to identify trends in the use of networks and network technologies This Technical Brochure contains:  The results of the follow-up survey held for this Technical Brochure  An analysis of trends in the use of networks and network technologies based on a comparison of the survey held for working group D2.28 and the follow-up survey held for this Technical Brochure  General considerations for replacing, refurbishing or extending networks  An assessment of available physical technologies and transport protocols  Recommendations for future works

8|Page

2 REQUIREMENTS FROM USERS To make sure the contents of this Technical Brochure are relevant Working Group D2.35 started by identifying what kind of information would be required by the intended audience. To identify the user requirements a survey was held amongst Cigré members. Working Group D2.28 held a similar survey a few years before so the results of the new survey can be compared to those results to identify trends.

2.1 Survey Results This next section describes the trends that can be identified by comparing the results from the survey held by working group D2.28 with the survey held by working group D2.35. The complete results from the survey held by working group D2.35 can be found in Annex 3 One of the goals of this survey was to try to identify trends in the adoption of IP networks. To that extent, some of the questions in the survey held in 2009-2010 by D2.28 have been re-issued in the survey that was held for this Technical Brochure. This section provides an indication of the trends between the D2.28 survey held in 2009-2010 and the D2.35 survey held in 2012. When comparing the results of the survey held for D2.28 and the survey held for this technical brochure, keep in mind that: -

-

The survey that was held for working group D2.28, while similar, is not exactly the same as the survey held for this Technical Brochure. For a detailed comparison both surveys can be found as an Annex to this TB. The survey for D2.35 was held under all Cigré members, not just the participants in the D2.28 survey. This means that there may be differences between the results of the two different surveys due to differing companies participating in the surveys.

Only questions that were similar or exactly the same between the two surveys are compared below. In total, 83 filled out surveys were received from 29 different countries. Figure 1 shows which companies from which countries have responded. ANSWERS FROM: TRANELSA ENERGINET.DK EGAT ITAIPU BINACIONAL Scottish Power REN SEAS-NVE AXPO AG HUAWEI INEXUS ABB Switzerland AUSGRID POWERLINK ESSENTIAL ENERGY ETSA Utilities TRANSEND ELECTRANET AURORA ENERGY COMMTEL ENDAVOR ENERGEX ERGON FRESNEL NETWORKS SNOWY HYDRO SP AUSNET TRANSGRID TRANSPOWER NZ GDH WESTERN POWER POWER COR ELECTROBRAS Power Grid Corperation of Bangladesh Bhutan Power Corporation Limited Sikkim power GETCO MSETCL PTCUL TSECL Nokia Siemens Networks Nokia Siemens Networks (2) ESKOM DNV KEMA

ANSWERS FROM: Argentina Denmark Thailand Brazil United Kingdom Portugal Denmark Switzerland Belgium United Kingdom Switzerland Australia Australia Australia Australia Australia Australia Australia Australia Australia Australia Australia Australia Australia Australia Australia New Zealand Australia Australia Australia Brazil Bangladesh India India India India India India Finland Finland South Africa The Netherlands

IBERDROLA FINGRID ISREAL ELECTRIC CORP RUGGEDCOM STATKRAFT STATNETT SO-UPS ELECTRIC RESEARCH INSTITUTE NARI RELAYS RTE AGDER ENERGI EGAT ZTE National Grid Saudi Aribia HUAWEI VERBUND Elektroprenos BiH Elektroprivreda BiH Hidroelektrane na Trebisnjici ABB CHINA(1) ABB CHINA(2) ABB CHINA(3) CFE Transmission Yucatan Peninsula CFE Transmnission north CFE Transmission north-east CFE Transmission central Tata Power Co. Ltd EEPCO SaskPower Altalink Management Unknown PSE Operator Hokkaido Electric Power Company Hokuriku Electric Power Company Tokyo Electric Power Company Chubu Electric Power Company Chugoku Electric Power Company Shikoku Electric Power Company Kyushu Electric Power Company NEC Corporation Liandon

Spain Finland Israel Germany Norway Norway Russia Mexico China France Norway Thailand Thailand Suadi Arabia Thailand Switzerland Bosnia and Herzegovina Bosnia and Herzegovina Bosnia and Herzegovina China China China Mexico Mexico Mexico Mexico India Ethiopia Canada Canada China Poland Japan Japan Japan Japan Japan Japan Japan Japan The Netherlands

9|Page

Figure 1 Companies represented in the survey responses Figure 2 shows the different countries from which responses were received and, per country, the percentage that the country contributed to the total results.

Figure 2 Countries represented in the survey responses The vast majority of the respondents work for Transmission-(49%) and/or Distribution-(28%) System Operators.

10 | P a g e

2.1.1 Substation applications using IP

83% 81%

Substation applications using IP:

2012

71% 67% 59%

2010

66%

64% 61% 64%

57%

64%

57%

52% 49% 49% 51%

56% 49% 44%

34%

24%

34%

34% 31%

28% 22%

12%

19% 12% 7%

Figure 3 Trends in substation applications using IP While many of the results from the two surveys are very similar, there are a few differences worth noting. The use of Wireless LAN access has significantly increased since the last survey. There also seems to be an increase in the use of IP for: -

Building control Telephony system Substation RTU to SCADA system

The comparison indicates that there is a decreasing use of IP for: -

Remote IP access to substation assets Inter control center connections Time synchronization

While this could mean that, since 2010, IP communication for these applications has been replaced by non-IP communication, it’s more likely that the differences in these results come from the difference in respondents. 11 | P a g e

2.1.2 Main operational challenges using IP as a communication transport solutions over optical network

Operational challenges: 86% 82% 2010

66%

2012 59%

58%

58% 57%

46% 41%

41%

39%

34% 27%

23%

Figure 4 Trends in operational challenges Again, the results of the new survey are very similar to the results of the previous survey. In the survey held for this technical brochure, fewer of the participants indicated that they see operational challenges in: -

Ruggedness Scope of responsibility

Compared to 2010, more participants indicated that they see Quality of Service (QoS) as an operational challenge.

12 | P a g e

2.1.3 Main psychological barriers with using IP protocols in Substation applications

Psychological barriers:

85% 77%

2010 2012

66% 57% 53% 47%

48%

32% 29% 23%

10% 11%

Figure 5 Trends in psychological barriers The results for both surveys are very similar. The most notable difference in psychological barriers is that the participants in the 2012 survey seem to be more concerned about the Quality of Service (QoS) of IP Protocols

13 | P a g e

2.1.4 Are all applications compatible with IP at the moment?

All applications IP compatible?

2010 79%

21%

2012 76%

18%

Yes

No

Figure 6 Trends in current IP compatibility There do not seem to be notable differences between the results of the 2010 and the 2012 surveys. 2.1.5 Prediction to migration of all operational communications into IP

All operational traffic through IP?

2010

60%

2012

36% 36% 25%

13% 8% 5%

2%

2% 0%

11% 2%

14 | P a g e

Figure 7 Trends in estimation of all operational traffic through IP Compared to 2010, participants seem much more convinced that they will be using IP based communication in the foreseeable future. Only 3% of the 2012 survey participants indicated that they believe IP networks will never host all of their operational traffic. This is a significant decrease compared to the 11% of the 2010 survey participants. 2.1.6 How to deal with legacy protocols and equipment?

How to deal with legacy protocols and equipment? 2010 2012 64% 58%

55%

43% 41%

40%

17% 12%

17% 14%

17% 10%

Figure 8 Trends in how to deal with legacy equipment and protocols While the popularity for each of the different options is comparable for both surveys, participants of the 2012 survey seem to be more selective.

15 | P a g e

2.1.7 Requisites and concerns for the telecommunication network

Requisites and concerns 2010 2012

69%

66%

63%

54%

34%

39% 30% 23%

Existence of Suitable Suitable IP Identification of physical transport addressing space Applications infrastructure technology (PDH, and scheme (IPv4 (fibre, copper, SDH, Ethernet, / IPv6) wireless, etc) etc.)

Figure 9 Trends in requisites and concerns The results for the different surveys are again comparable. The percentages in Figure 9 for each option are lower in the 2012 survey then they were in the 2010 survey, creating the impression that the participants of the 2012 survey have fewer requisites and concerns. FIGURE 9 however does not show option “Suitable substation applications”, because this option was only available in the 2012 survey. 68% of the participants of the 2012 survey chose this option.

16 | P a g e

2.1.8 Most promising technologies to provide secure and reliable transport communications in the access network?

Most promising secure/relaible technology access network 2010 2012 46% 41% 39% 35%

14%

13%

MPLS (IP over Ethernet/IP Ethernet/IP fibre) over SDH over DWDM

Figure 10 Trends in most promising access network technologies While similar, the scope of this question differs between the two surveys. For the 2010 survey, the question was: “Which underlying IP technology is the most promising to provide secure reliable IP communications?”, while the scope for the D2.35 survey was limited to access networks. Nevertheless, MPLS and Ethernet/IP over SDH seem to be popular in both surveys.

17 | P a g e

2.1.9 Most promising technologies to provide secure and reliable transport com munications in the core network?

Most promising secure/relaible technology core network 2010 2012 52%

51% 45%

39% 35%

14%

MPLS (IP over Ethernet/IP Ethernet/IP fibre) over SDH over DWDM

Figure 11 Trends in most promising core network technologies Again, the scope of this question in the 2010 survey differed from the scope in the 2012 survey. For the 2010 survey, the question was: “Which underlying IP technology is the most promising to provide secure reliable IP communications?”,

18 | P a g e

2.1.10 Should the IP network be reserved for operational service or also be used for corporate services?

Seperate operational services from corporate services?

2010 2012

58% 54% 48%

47%

5% 2% One network, no separation

Separate network (physical)

Separate network (virtual/logical)

Figure 12 Trends in network separation There does not seem to be any significant difference between the results for the 2010 survey and the results of the 2012 survey.

2.1.11 Survey Conclusions Many utilities indicate they have concerns and are facing challenges (2.1.2 Main operational challenges using IP as a communication transport solutions over optical network) in implementing and maintaining an IP only network. Still, three out of four utilities indicated that they expect to migrate all their operational traffic to IP within the next 10 years (2.1.5 Prediction to migration of all operational communications into IP). Chapter 3 explains why utilities are migrating to IP despite their concerns and the challenges they face and provides guidelines on what to consider when migrating to IP.

19 | P a g e

2.2 List of Services This section outlines, at a functional level, the service requirements of a typical EPU. Over twenty different systems are identified as supporting operational infrastructure, with many of these systems critical to the operation of the electricity network. The integration or the support for the services listed here will have to be considered by the utilities when they migrate to new network technologies. Bandwidth provisioned on many EPU telecommunications networks fall into two categories - Operational services and Group/Corporate services. Operational services may be as follows:                      

Teleprotection Outage Management Systems Network and Access management SCADA services Energy Management Systems Energy metering Event recorders Switchover of services between control centers Disturbance recorders Real-time PMU Polling telemetry operational communications and Black Start telephony Operational Voice Services Maintenance & Support Private Mobile Radio General site alarms, supervision and surveillance Time distribution using IEEE 1588– alternative to GPS. Video services Physical site security using access control mechanisms. Smart Metering communications. IEC 61850 based communications. Dynamic Line Rating Weather Monitoring & Lightening Detection

Corporate services may be as follow:     

Corporate IP network. Corporate fixed telephony Corporate mobile telephony Corporate voicemail Corporate Video Conferencing

In a lot of existing deployments, where adequate spare capacity exists and economically advantageous to do so, non-operational traffic is accommodated on operational networks. Non-operational traffic is usually a secondary consideration and in general is not a determinant of network build capacity. 20 | P a g e

Many documents from various organizations (e.g. CIGRE, IEC) already include detailed descriptions of the communication / performance requirements for individual applications. It is not the purpose of this document to re-write or repeat these requirements. The following is a (non-exhaustive) list of documents that include such information:       

CIGRE D2.23: The use of Ethernet technology in the EPU environment CIGRE JWG D2B5.30: Line and System Protection using Digital Circuit and Packet Communications CIGRE JWG 34/35.11- TB192: Protection using Telecommunications IEC 61850-90-1: Use of IEC 61850 for the communication between substations IEC 61850-90-2: Use of IEC 61850 for the communication between substations and control centers (under preparation) IEC 61850-90-5: Use of IEC 61850 to transmit synchrophasor information according to IEEE C37.118 IEC 61850-90-12: Wide Area Engineering Guidelines (under preparation) o 90-12 compiles a significant amount of information related to applications and requirements. It is envisaged to become a document which covers a lot of requirements surrounding IEC 61850

Deriving a single mechanism to accurately classify applications based on the information above is quite difficult. For this reason the Technical Brochure categorises differing applications into groups which can be hereafter referred to as traffic types. Traffic types can be classified into 6 areas Samples of possible classifications are listed below: 1. Very low latency, loss intolerant, sequence and symmetry determinant traffic: over the same path a. Differential teleprotection schemes 2. Very low latency, loss intolerant a. Distance Protection 3. Very low latency, loss tolerant traffic: a. Time distribution b. Real-time PMU c. Inter substation event distribution 4. Low latency sequence determinant traffic: a. RS485 b. IEC 61870-101 – IEC 61870-103 c. DNP3 5. Low latency, loss tolerant traffic a. Voice b. Video c. WAMPAC 6. Latency tolerant, loss tolerant a. Packet based SCADA protocols b. File transfer c. Device management

21 | P a g e

3 GENERAL CONSIDERATIONS This section provides an overview of the requirements utilities need to consider as part of a WAN migration strategy, and highlights key services and their performance requirements. Further details of the technologies and services are explained in Chapter 4.

3.1 Why are Utilities migrating? Electrical power grids continue to evolve to meet the challenges of standardisation and grid modernisation in order to enhance existing use cases and applications, while simultaneously enabling new use cases driven by the concept of the smart grid. The addition of new technologies has the potential to allow an EPU to realise a more stable, reliable, efficient and visible power grid capable of handling any type of communication flow. Electric utilities are among the largest users of privately owned and operated wide area network infrastructures. These are built on a hybrid mix of fiber optics, microwave, power line carrier, a variety of licensed and unlicensed wireless incl. Wi-Fi, PDH/SDH, plus more recently packet-based technologies. Communication flow models have largely followed a one-way power delivery flow from generation to consumer. As a result most communication networks consist of multiple point-to-point circuits connecting substations to control centers for SCADA and EMS applications, and point-to-point circuits between substations for protection applications. Many are built and operated for specific applications or solutions, making it more challenging to integrate new use cases and operational processes. The evolution of Smart Grids will result in a large number of intelligent devices being dispersed throughout the electricity grid. New last mile architectures, protocols and technologies will be required to service these devices, many of which will have modest bandwidth requirements, but may be located in areas which traditional networks struggle to reach. New emerging protocols, such as IEC61850, require sub systems based on IP technologies to operate on a local, regional and national level. Telecommunications will be an essential enabler for many Smart Grid applications. The requirements of Smart Grid telecommunications will be met by a combination of traditional networks and media supplemented by emerging last mile technologies, many of which are as yet not fully matured. It is nonetheless almost certain based on the distribution of physical electrical assets that last mile technologies will be wireless based with a greater or lesser role for transmission/distribution line communications. Cost, reliability, criticality, capability, security and performance will be the primary determinant of the technologies selected. The introduction of packet has mostly been restricted to IT, while OT communications have largely remained on traditional TDM networks. The industry is experiencing a steady uptake in TDM to packet migration due to a number of reasons including end of life of many TDM solutions, emerging standards such as IEC 61850, interoperability and the use of IP to future-proof grid communications. Utilities are beginning to consolidate networks on a common infrastructure as part of ongoing refresh cycles, to minimise operational expenditure, and to introduce new use cases. Newer, advanced multipoint use cases, for example, listed below are also driving the need for more bandwidth, multipoint connectivity, forcing us away from the model depicted in FIGURE 13, towards that described in FIGURE 14. This is the case for many traffics other than SCADA also.

22 | P a g e

        

Wide Area Measurements Synchro-check Adaptive relaying Situational awareness State Estimation and on-line security assessment Wide Area Controls - Special Protection Schemes Wide Area Controls - Predictive Dynamic Stability Maintaining System Wide Area Controls - Under Voltage Load Shedding Wide Area Controls - Phenomenon assumption type WAMPAC

Figure 13 Traditional Grid Communications Architecture

Figure 14 Tiered approach for New grid applications The result is that new applications are driving transport models, transport models dictate the WAN design and technology choices. There are a number of drivers to consider as part of a grid WAN migration process. Economic drivers include cash optimisation and margin control, cost effectiveness in grid optimization, OPEX reduction and the opportunity to influence costs of investments for the future. Technology drivers include growing obsolescence of current communication infrastructure due to technology constraints in current platforms and the location of equipment in product life cycles, grid flexibility and scalability, and grid monitoring, control and protection. In addition increasing regulation (e.g. EU 20/20/20 ad M/490 mandates and NERC-CIP for North America) and standardization efforts such as IEC 61850 and IEC 62351, mean that utilities are looking at migration in new ways carefully considering all implications This drives the need for an open network architecture based on open standards, for the implementation of a multiservice network which accommodates multi-vendors, tailored to meet the requirements for all communications services and future ones, with the highest levels of quality, reliability, scalability and costeffectiveness. TDM migration is one of the biggest changes in the electricity network and while many newer applications and use cases are demanding flexibility that TDM does not have, existing applications such as Teleprotection schemes require that packet based transport mechanisms need substantial testing over time to ensure they meet the performance requirements of legacy, traditional and newer services. 23 | P a g e

Figure 15 shows a high level overview of the state of migration. Traditionally most WANs were based on TDM technologies. Over recent years packet technologies have emerged supporting some operational and IT applications. These have either been a combination of packet over TDM, or TDM over packet. There is a gradual shift in the industry towards a hybrid or all packet network architecture.

Figure 15 Shift from TDM to packet

Why are we seeing this transition? The reality is that the communications network is evolving due to changing communications requirements such as precision timing and IEC 61850, increased network functionality including enhanced QoS features and multicast support, and the addition of multiservice capabilities like workforce enablement and physical access solutions.

3.2 Migration considerations There are a number of decision factors to consider and questions to be asked as part of a transition 

 

 



New technologies require new skillsets and engineering approaches. (one of the operational challenges according to the survey results). Is the required skill set and mind set available at utilities for a transition? How does the organizational structure (with OT and IT responsibilities) influence a migration process? Substation equipment refresh: What is the time line for refreshing substation equipment to a new technology? Is this process in progress? Does existing equipment have physical interfaces available to connect using a new technology such as Ethernet/IP? Control Centre capability: Does the EPU management system/s support the new technology choice? What is the time line for this support? Does this require a rebuild and what is the cost for the upgrade? Is an upgrade of existing substation equipment possible, or does it need to be replaced? Cost of substation equipment replacement/upgrade: If no refresh is taking place what is the cost of upgrading equipment to enable adequate interfaces? Can this be justified and is it giving additional functionality? What is the disruptive risk to the overall electrical network and overall electrical supply due to this technology change?

24 | P a g e







Additional cost to integrate legacy applications which cannot be converted to new technology: What is the additional cost to include TDM and traditional SCADA interfaces into the a newer network? Will this include legacy interface support, tunneling and translation of traditional SCADA traffic? Can this cost be justified as the additional capabilities may not be required in 5-10 years? Cost to maintain existing TDM based infrastructure: What is the associated cost to maintain the existing TDM/SDH infrastructure to allow the migration of the legacy devices? Is this infrastructure still supported? What are the security and availability risks in maintaining this Infrastructure? What is the cost of not migrating? What applications and use cases cannot be deployed? What is the cost of supporting two or more infrastructures?

In addition there are a number of network influencing factors. The TDM infrastructure, the business and functional requirements, existing and future communication requirements, and existing infrastructure. Reviewing the questions and influencing factors following a process such as that outlined in Figure 16, and taking into consideration the guidelines already documented in the Technical Brochure from D2.28, the outcome will be the required communications infrastructure.

Figure 16 Migration decision review process

25 | P a g e

An architectural approach, consistent with industry and standards organisations (e.g. NIST, IEC, EPRI, CENELEC), networking companies, and also power vendors is essential to understand the interdependencies and communications between systems that exist, but equally important to provide a seamless platform for future use case integration. An architectural process defines a set of artifacts required to describe a system so that it can be reproduced and maintained over its lifespan. These artifacts provide components, structure, interdependencies, and the guidelines determining the design and evolution over time. This is important as each EPU may have subtly or very different requirements and WAN design will only reflect the use cases and services an EPU wishes to deploy. All of this should be captured using relevant architecture frameworks such as TOGAF or equivalent.

3.2.1 Timing Substation automation and system control are mission-critical processes and electric power utilities must synchronize applications and use cases across large-scale distributed power grid substation networks to ensure grid stability. Precise timing is used to improve reliability, better understand operations of the power system, predict and prevent local and system-wide faults, for testing and verification, and to reduce costs. Refer to ANNEX 4 Network Timing for more details.

3.2.4 Communication Requirements Decomposition When designing any communications network it is essential to consider the various elements that make up communications for an application or service. The communications decomposition table below gives a good starting point for areas to consider and build into the design process.

Figure 17 Communications decomposition table for packet oriented traffic

26 | P a g e

3.2.5 Security Electrical power grids continue to evolve to meet the requirements of standardization efforts and grid modernization in order to enhance existing use cases [1] and applications, while simultaneously enabling new use cases driven by the concept of the smart grid. In addition there has been a surge in the number of non-power applications introduced across the grid, as well as links between networks inside an EPU, and links to external networks. The addition of new technologies allows an EPU to realize a more stable, reliable, efficient and visible power grid capable of handling any type of communication flow. Through the integration of new systems, applications, and technologies to facilitate new use cases across network infrastructure (sometimes common), an EPU’s network becomes multi-service. However with the introduction of any new technology there is an increased risk of security threats which must also be taken into consideration. The highly connected nature of a smart grid has the potential to provide unauthorized users with a greater opportunity to identify and exploit vulnerabilities to attack the electricity grid. There are many examples of attacks on networks worldwide and constant vigilance is necessary to ensure that adequate protective mechanisms are used to ensure the electricity grid is not compromised. While designing and implementing operational networks it will be necessary to comprehensively address cyber security issues. A “defense in depth” (security measures deployed at various physical and conceptual layers of a deployment) approach should be adopted throughout the deployment. Security capabilities must therefore be layered such that defense mechanisms have multiple points to detect and mitigate breaches for all service types that an EPU wishes to deploy. These capabilities should be integral to all segments of the grid infrastructure and address the full set of logical functional requirements, including:      

Physical security Identity and access control policies Hardened network devices and systems Threat defense Data protection for transmission, distribution and storage Real-time monitoring, management, and correlation

Secureness of a technology. All technologies outlined within this document possess specific attributes which at a basic level reduce or introduce additional risks to cyber security, information separation, data privacy and data authenticity. All technologies should be assessed based upon these criterion to evaluate suitability and to highlight any additional activities required to ensure a secure introduction of the technology. Cyber Security encompasses the overall security landscape from a physical level to a user and electrical/electronic level. Information separation is the ability for a technology to transport a specific data stream in a fashion which can be deemed isolated from dependencies of other data streams. Other security considerations:.     

Access Control: Authentication and authorization of all personnel management tools and physical devices Authentication, Authorization and Accounting for data and devices Data Confidentiality and Privacy Securely scalable Tools: Zones, Segmentation, Logging 27 | P a g e

 

Tools: Tamper-resistant design, authenticity and integrity of hardware and software Integrity of Platforms and Devices: Secure devices over the entire life-cycle

3.2.6 Communication Network Management A unified security and network management architecture will be based on a set of applications delivering the Fault, Configuration, Accounting, Performance and Security (FCAPS) functions as defined by the OSI Network Management reference model. A key challenge is the definition of “unified management”. It is unrealistic to expect that a single application can provide all functions and services required when managing such diversity of equipment and communications technologies. More recently applications have been able to manage multi-vendor and multi-technology communications networks, but there is no guarantee to cover all equipment, communication types, and future requirements. It is more realistic defining a set of requirements and features that enables network management and power grid applications to interoperate and exchange information, such as:   

     

Network and security management and power grid management applications must be able to communicate over TDM, IPv4 and IPv6 network layers over the expected lifetime of the deployment. Role Based Access Control for administrators and operators must be able to get user’s authentication from corporate directory services, thus showing interaction between IT and OT networks. Network and security management and power grid management solutions applicable to use cases requiring large scalability, must be able to work in a clustering and load-balancer networking environment. It allows growing the solution over time without reconsidering the architecture. Data flow and API (Application Programing Interface) must be identified, specified and documented for communications between OT and NMS applications. An unified Geographic Information System (GIS) solution should be shared between applications Recommended database (DB) properties should be defined when storing all operational state, device configuration, network event alarm, performance metric, etc. Recommended Historian properties should be defined to store data on a long period of time as required by regulations and operations. Standard protocols, including public vendors’ extensions, should be recommended to facilitate management data exchanges between a device and multiple management applications. Remote Operations Management capabilities should provide utilities asset management functions with real-time communications, improved availability of information, and enhanced visual displays, all of which enable the asset planners and field workers to perform their work with improved efficiency

Benefits A powerful Network Management System enables both condition-based and predictive maintenance strategies so the EPU does not have to rely upon reacting to failures after they happen and scheduling inspections, which normally have to take place even if there are no equipment health issues. In addition, asset data can be used as input for grid state determination and remote power quality monitoring. These benefits also contribute to improved System Reliability 28 | P a g e



Increase Service Restoration Speed: Improvements in the speed and availability of information in the field allows technicians to diagnose and repair assets more quickly, resulting in a more rapid restoration of service.



Increase Reliability Metrics: Increases in the speed and efficiency of delivering service, and increased service restoration speeds will result in higher reliability metrics.

The technologies discussed within this document will be responsible for supporting critical electrical assets, systems and services within an EPU. To ensure that adequate clarity and control of a technology, dynamic, scalable and intuitive management systems need to be put in place. Management systems should be able to accurately present, configure, report and alert on network state in a close to real-time fashion while ensuring compliance with change control and management methodologies.

3.2.7 IPv4 and IPv6 co-existence Even if in future networks the core network devices may be running IPv4, the networks may need to offer IPv6 support for applications.. Co-existence of these protocols is a consideration as part of any migration or accommodation strategy involving Ethernet/IP.

3.2.8 Segmentation and Virtualization With the growing adoption of packet based technologies to converge all data, voice and video traffic from both an operational and IT perspective, there has been a growing need for logical segregation rather than physical. The IEEE and IETF have standardized Layer-2 and Layer-3 mechanisms, enabling links to be shared by all data types which have been in operation for many years. A segmented or virtual network can be seen as decoupled from the physical network but it doesn’t change its basic characteristics. It represents an abstraction that requires proper design, deployment, security and management to be successfully operated. There is a broad range of mechanisms to achieve segmentation and virtualization, in TDM, and Ethernet/IP environments. Any design should match the associated applications requirements in regards of bandwidth, prioritization, security, latency and so on. Example applications could be Serial SCADA over IP, Layer-2 Ethernet or Serial circuit and segregation of operational and corporate data via MPLS VRFs or a separate VC container in SDH. Whatever the selected mechanisms, the “Virtualized Network” building blocks must be designed in considering the following topics:  Segmentation versus virtualization  Separate vs “virtually” separate  Network management – Network management applications must be able to retrieve data from all “Virtualized Networks”  Capacity estimate – when designing and operating the network, capacity estimate allows proper selection of physical and data link layers in order to provide enough bandwidth and optimized latency – particularly important for Teleprotection – for all applications. Capacity estimate of the physical infrastructure must include the network protocol overhead related to the segmentation and virtualization mechanisms. This must include the performance and scalability impacts of the mechanisms on the devices at the various network places.

29 | P a g e

3.2.9 Scalability The migration of EPU Services and applications to new communication technologies will take place over a long period. While future requirements are captured where possible, even more demands and requirements will be presented post deployment. For this reason it is necessary that all technologies are assessed based on the ability to meet:    

future needs of applications future deployment models future capacities future geographical expansion

3.2.10 Availability The availability concept is related to time. That is, availability can be defined as the time in which the system is reliable. In Optical systems, highest availability is achieved by means of certain degree of Redundancy. That redundancy must cover not only the electronic devices such are transmitters, receivers or amplifiers, also must cover the fibre path implementing protected routes between nodes. Availability is also strongly influenced by the network recovery time which depends on the failure detection time and the availability of a protection path.

30 | P a g e

4 ASSESMENT OF TECHNOLOGIES This section discusses a number of technologies which are currently already in use in EPU networks or may be considered for future extensions of the network:      

Synchronous Digital Hierarchy (SDH) Dense Wavelength Division Multiplexing/Coarse Wavelength Division Multiplexing (CWDM/DWDM) Optical Transport Network (OTN) Dynamic Internet Protocol Multiprotocol Label Switching (IP/MPLS) Static Multiprotocol Label Switching (MPLS-TP) Ethernet

This is not a detailed technology tutorial, but has a focus on specific characteristics which are relevant for EPU applications. Therefore for each technology the following is discussed:        

Life Cycle Scalability Capability to support EPU Services Ease of Implementation & Operation Interoperability (other services/technologies) Redundancy, Availability, Reliability Quality of Service (QoS) Network Management System and Overall Manageability (NMS)

31 | P a g e

4.1 Synchronous Digital Hierarchy (SDH) Life Cycle SDH is a mature technology. While SDH is a Legacy technology for Public Telephony, it continues to be the predominant technology for utilities. Thus, it is most likely that SDH prolongs its existence for years to come. This is evident within the survey. SDH is a standardized transmission technology that allows multiple digital bit transfers over optical and electrical links. It was established by a set of ITU-T standards in 1989. When introduced, one of the main concerns was to assure compatibility with existing PDH services without synchronization problems. This was the reason to incorporate standard PDH interfaces. Right now SDH is still the transmission infrastructure of a large number of telecommunication operators and utilities. The most suitable media for SDH is fiber optic, nevertheless electrical SDH line interfaces as well as microwave radio are also common. SDH is a widely adopted technology, it is possible to build SDH equipment using general purpose components. This reduces the risk and the effect of life cycle decisions within the vendor/manufacturing market.

Scalability Huge networks have been built with SDH technology and several European Public Telecom Operators still use SDH as the backbone network. Rates of 10 Gbps (STM-64) are commonly installed and 40 Gbps (STM-256) rates are beginning to appear in the market. SDH is commonly used to build extensive wide area networks. In the following table most common SDH rates are shown. That said, SDH still provides the ability to interface and connect networks and devices at very low data rates (64kbps) in t seemless fashion.

Synchronous Transport Module

Hierachy

(Mbps)

Interface electrical/optical

STM-1

1

155,520

G.703/G.957

STM-4

4

622,020

---/G.957

STM-16

16

2 488,320

---/G.957

STM-64

64

9 953,280

---/G.957

Figure 18 SDH Capacity Comparison 32 | P a g e

Capability to support EPU Services SDH supports a variety of different payload types enabling transport of TDM classical services such as E1 Circuits. SDH network can tunnel point to point Ethernet connections (Ethernet over SDH) and SDH equipment can provide vendor specific Ethernet capabilities - even multipoint connections as an overlay service. A key element in SDH networks is the Add and Drop Multiplexer (ADM) which is the equipment that provides basic capacity for traffic insert and cross-connections.

East

West STM-1,STM-4,STM-16,STM-64

STM-1,STM-4,STM-16,STM-64

Tributaries Electrical intefaces

2M,34M,45M,140M

Optical interfaces

STM-1,STM-4,STM-16

Figure 19 SDH Add-Drop-Multiplexer This capability allows the efficient transport and support supports of legacy and TDM services such as tele control and protections.

Ease of Implementation and Operation From an EPU perspective, SDH’s concepts are familiar and known. They are commonly regarded as “simpler” than data-packet based technologies. This is evident in the questionnaire and in section 3 skillset Moreover, with a lot of matured equipment deployed, there are a lot of technicians familiar with the operation and maintenance of SDH and its theoretical concepts. From this point of view it is relatively simple to establish maintenance strategies due to the fact that know-how is widely available. The fact that SDH is a mature technology that has been very popular ensures also that a lot of spare parts are available. Due to its TDM characteristics, SDH lacks modern data technologies flexibility, and traffic engineering is hard to optimize in big networks generating the fact that some traffic capacity is wasted in meshed networks.

Interoperability (other services/technologies) In comparison with other “newer” technologies, interoperability with legacy interfaces is a strong 33 | P a g e

point. Especially in utilities, SDH in combination with PDH offers interfaces for legacy electrical protection and control devices that can hardly be obtained with other technologies in a native way. SDH provides also a deterministic behavior, which is widely appreciated by traffic engineers. Recent developments have been made in order to accommodate SDH with data based network, such as Ethernet over SDH. SDH works efficiently with WDM.

Redundancy, Availability and Reliability SDH incorporates native protection and resilience mechanisms such as SNCP or MS-SPRING that can provide a 1+1 and N:1 traffic protection. Restoration times of around 50 ms can be achieved. Overall concept for SDH traffic redundancy is to offer an alternative path for connecting origin and destination in a telecommunication circuit. So, in a typical ring configuration two physical different paths are provided. In the following figure an example of SNCP protection is provided.

DXC

ADM

SubNetwork Connection Protection: Simultaneous tranmission by two paths working

protection

ADM

Figure 20 SDH Sub-Network Connection Protection * DXC in Figure 20 refers to a Digital Cross Connect Card protection redundancy schemes are also available. This architecture ensures that in the case of one card failing, the remaining cards can avoid service interruption and single points of failure in equipment. In general, if adequate traffic redundancy is provided, SDH networks have high levels of availability of telecommunication services. 34 | P a g e

Quality of Service (QoS) As a TDM transmission technology SDH supports only reserved bandwidth for each circuit, this mechanism insures a guaranteed QoS especially for “mission critical” applications, but SDH is not optimized in case of multi-services transport. These features are inherently given within SDH standards. The ability to classify traffic within Ethernet circuits is not common across all vendors.

Network Management System and Overall Manageability (NMS) SDH management technology is very well understood, providing inbuilt OAM and monitoring functionalities. SDH management provides alarms supervision and remote provision of services. SDH networks management platforms offer automated management functions. These platforms provide the data that is needed in the day-to-day running of a telecom network and have the possibility of issuing commands to the network infrastructure to activate new services and detect and correct network faults. One of the drawbacks is that SDH management platforms are proprietary. It is possible to connect SDH equipment from different vendors, nevertheless is neither possible to manage a network element from other vendor platform nor feasible to establish automatically a point to point SDH circuit that runs on different vendors regions.

35 | P a g e

4.2 Dense Wavelength Division Multiplexing & Coarse Wavelength Division Multiplexing (DWDM & CWDM) Life Cycle It comes as no surprise that traffic growth is the underlying driver for 100 Gigabit (Gbit) WDM network transport. This was the case with the transition from 2.5G to 10G and from 10G to 40G and now 100G optical channel rates. WDM technology is a physical technology allowing transporting any protocol with variety of bandwidth granularity till 100 Gbps (or more in next future), this capacity of WDM technology will allow covering actual and future data stream needs. Currently Wavelength-division multiplexing WDM transport technology is assuring carrier services for both TDM technology and Ethernet technology. In essence WDM is just a support technology, an infrastructure for the other technologies. More recently WDM is a technology that allows the coexistence of various transmission systems sharing the same fibre pair. SFP optical interfaces are one of the major contributors to the popularity of the WDM because it’s intrinsic easiest way to change the transmitting wavelength simply changing the plug-in module. In fact actually the major part of equipment manufacturers offers a wide variety of SFP possibilities on its optical interface. In the last years the extensive use of the OADM devices, which allows some degree of optical routing, in the Metro networks increases the popularity of that technology. In fact, at the present, all the Optical Transmission Vendors have WDM devices on their portfolio.

Scalability There are two types of WDM architectures: CWDM, and DWDM: A CWDM system typically provides 18 Channel-Wavelength, separated by 20nm, from 1260nm to 1630nm. DWDM is a technology used for a very high capacity links (ITU-T G.694.1). DWDM systems, mainly used by Public Telecom Operators, can multiplex from 32 to more than 100 channels in the range of 1530 – 1624 nm.

36 | P a g e

. Figure 21 Optical spectra in fiber The number of channels, the channel selection (center frequency), and the frequency width of each channel, as well as the channel separation, are important parameters in WDM system design. With a channel spacing of 1nm (or below), DWDM operation requires extremely precise optical sources (laser temperature stabilization) resulting in significantly higher complexity and cost. WDM technologies can accommodate any bit rate providing necessary bandwidth: SDH (STM1, STM4, …), Ethernet (1Gbit/s, 10 Gbit/s) … WDM technology has brought more than an order-of-magnitude increase in the amount of bandwidth that can be transported over fiber. Early implementations were point-to-point only. However, while transport networks may be thought of as the roads of the network, the intersection points are also critical. Natively, WDM technology provides a point to point links, however it is possible to have different type of architectures (ring, mesh …) by mixing WDM with other protocols (Layer 2, Layer 2.5, Layer 3…). Today’s Optical Ethernet networks are much more dynamic than in the past and demand greater flexibility. In order to provide the necessary flexibility, Reconfigurable Optical Add Drop Multiplexers (ROADMs) were developed. They allow operators to access any wavelength at any node at any time – replicating the operational simplicity and flexibility of SONET/SDH networks at the wavelength level.

Figure 22 examples of Add-Drop-Multiplexer

37 | P a g e

State of the art ROADM technology and the use of tunable lasers and filters, allows operators to not only drop any wavelength at any node and any time, but to also send any wavelengths in any direction (directionless) using any available port on the network node (colorless). This so-called triple-A architecture (AAA – any wavelength, any node, any time) is fully flexible and non-blocking. It requires very little technical skill from the operational staff and eliminates the need for meticulous pre-planning. Triple-A ROADMs are the foundation of a fully automated optical network, and they allow for intelligent interworking using the two layer architecture concept, which despite other technologies, have the facility to deploy a network architecture with two stacked switching layers.

Figure 23 Reconfigurable Optical Add Drop Multiplexer

LONG DISTANCE TRANSMISSION In the transmitting side the laser diodes have its output power limited by the signaling speed and, for the same physical reasons, in the receiver side, the quantum energy necessary to reach the detection threshold increases following the signaling speed.

Power [dBm]

Power [mW] 10 mW

+10

TX/Laser Diodes

1 mW

0 - 10

TX/LED

- 20 - 30

1 µW in RX in RX iodes d o t iodes o d h o p t r o o f e ph r limit lanch Lowe r ava o f it r lim Non detectable signal in RX Lowe

1 nW 1M

10 M

100 M

1G

10 G

- 40 - 50 - 60

100 G

Bit Rate [Bit/s]

Figure 24 Working area for light source and receiver 38 | P a g e

For example for 10 Gbit/s bite rate, standard photodiode sensitivity limit is higher than -35 dBm (Red point in the figure above) because the limited energy of the incoming optical pulses. Obviously these characteristics imply that the optical amplification becomes necessary for high speed transmission. An alternative solution is to spread the high speed stream in some medium / low streams and use passive WDM to transmit simultaneously those streams in a single fiber pair

Capability to support EPU Services WDM is a method of combining multiple signals on laser beams at various infrared (IR) wavelengths for transmission along fiber optic media. Each laser is modulated by an independent set of signals. Wavelength-sensitive filters are used at the receiving end to separate the signals.

Figure 25 WDM Multiplexing This is an attractive multiplexing technique that allows:       

Does not directly support termination of many legacy or EPU specific connections (RS-232/485, E1 etc) High bit rate without high speed electronics or modulation Mixing legacy and new network technologies on a single optical fibre Very useful for upgrades to installed fibers Suitable for Mission Critical Networks Extensive use of passive components Loss, crosstalk and non-linear effects are potential problems.

Ease of Implementation and Operation CWDM technologies offer rapid deployments with relative ease in comparison to DWDM solutions. However, many CWDM solutions relay upon subsequent layers of technology to enable “vision” on the network. For this reason, ongoing operations and maintenance of a CWDM solution can become laborious as a network scales. Some CDWM and most DWDM systems include integrated, network management programs that are designed to work in conjunction with other operations support systems (OSSs) and are compliant with the standards the International Telecommunication Union (ITU) has established for Telecommunications Management Network (TMN). Current systems utilize an optical service 39 | P a g e

channel that is independent of the working channels of the WDM product to create a standards– based data communications network that allows service providers to remotely monitor and control system performance and use. Meeting ITU standards and utilizing a Q3 interfaces ensure that end users retain high Operations, Administration, Maintenance, and Provisioning (OAM&P) service. Most systems use SNMP as a standard but in-depth configurations of devices may not be achievable with open standards.

Interoperability (other services/technologies) WDM networks provide purely optical transport for different transmission technologies and different transport protocols. The WDM technology is designed to allow the coexistence of diverse technologies over the same media. As a carrier system many wavelengths can be implemented inside a fibre with total independence, carrying each one the appropriate services. This ensures protocol and format transparency in the network. A major advantage is coexistence with several protocols (SDH, Ethernet …), for example SDH/WDM, Ethernet/WDM, IP/WDM, MPLS/WDM…

40 | P a g e

Redundancy, Availability and Reliability In terms of Availability and Reliability, there are some differences between the WDM implementations. With a channel spacing of 20 nm, CWDM systems tolerate the temperature drift over the whole industrial temperature range of the laser emitter without leaving their allocated channel and without use of special cooling system. The strict frequency stability requirement to match with the DWDM channelling is achieved keeping the laser cavity at constant temperature and limiting the emitter power output. This implies that temperature needs to be kept constant and lowers the MTBF of a system in comparison with CWDM CWDM ruggedness, in addition to considerably lower cost, constitute significant advantages of this technology in the EPU communication network where wavelength multiplexing is not often used for attaining maximum bandwidth but rather for separating networks. Circuit redundancy and failover are not characteristics associated with CWDM/DWDM but are often accomplished via technologies carried on top of WDM

Quality of Service (QoS) QoS provided by WDM technology is based on resource reservation (equivalent to Constant Bit Rate): Each connection has reserved bandwidth (wavelength) without any overflow mechanism. There are a lot of discussions about the suitability of the transmission of Teleprotection signals and / or Differential protection over non dedicated transmission systems. Both Synchronous and Packetized systems present some degree of inconvenience which are discussed in other parts of this Brochure. Direct transmission systems or dedicated point to point equipment. The most suitable media for mission critical communications is fibre optic. There are methods to deploy multiple services via separate optical channels: The use of dedicated optical fibre or the use of a dedicated wavelength. Because optical mixer and receiver filter banks are passive elements the reliability of the transmission path is not affected by choosing one solution or another. WDM technology allows a deterministic end to end delay and jitter.

41 | P a g e

Network Management System and Overall Manageability (NMS) WDM management technology varies a large amount due to the passive nature of some solutions in comparison to the comprehensive nature of other types of solutions. CWDM devices usually rely on subsequent technology layers above to verify connectivity and availability. DWDM can provide extensive optical and circuit based managers providing granular control and vision of provisioned services. CWDM versus DWDM The election of the optical channelling method is a decision which cannot be taken in early stages of a design. This is mainly because both C and DWDM channels may coexist over the same fibre and CWDM is recommended for Mission Critical applications because its highly reliable nature in comparison with DWDM

42 | P a g e

4.3 Optical Transport Network (OTN) Life Cycle OTN adoption was initially slow; the first deployments of OTN were in Japan with little interest in Europe and North American. The reason for this slow adoption is that carriers have invested huge amounts of capital in the existing SDH/SONET and WDM networks. Since the mid 2000s, OTN technology has been essentially proposed for point to point optical links requiring enhanced Forward Error Correction (FEC) capability and higher data rates. Today, OTN technology is deployed in Telco carriers for several network topologies (ring, mesh …), with the possibility to transport transparently any client traffic offering some additional features like enhanced OAM, Redundancy & Resiliency. OTN is supplied by carrier network vendors, mainly the same vendors as SDH. The carrier market for OTN is envisaged to nearly double from 2012 to 2016.

Scalability The basic signal architecture of OTN is shown in the FIGURE 26 below:

43 | P a g e

Client Signal

Optical Channel Data Unit (ODU-k)

Electrical Domain

Optical Channel Payload Unit (OPU-k)

Optical Channel Transport Unit (OTU-k)

Optical Multiplex Unit (OMU)

Optical Domain

Optical Channel (Och)

Optical Transport Module (OTM) Figure 26 OTN Architecture The client signal is passed through two main domains: Electrical domain and optical domain. The OPU-k encapsulates the client signal and performs (if needed) the rate justification. The ODU-k performs the same functions than the Line Overhead in SDH The OTU-k contains the FEC and performs similar functions as the Section overhead in SDH The OUT-k is transmitted over the Och using a wavelength; the OMU performs the multiplexing/de multiplexing of several wavelengths.

44 | P a g e

The standardized bit rates of OTN are: OTN container

Payload Rate

Supported Clients

ODU-0

1.25Gbps

STM-1/4, GE, FC, others

ODU-1

2.5Gbps

STM-16, 2G FC

ODU-2/ODU-2e

10Gbps

STM-64, 10GE

ODU-3

40Gbps

STM-256, 40GE

ODU-4

100Gbps

100 GE

ODU-Flex

N*1.25Gbps

Sub rate Ethernet or constant bit rate like EPON,GPON,CPRI

Figure 27 OTN Capacities Source: Infonetics June 2012 OTN capacities are far beyond potential EPU OT requirements but arguably does present EPU’s with an opportunity to deploy bandwidth as a service for non-operational uses. (corporate or commercial services) Capability to support EPU Services OTN has been developed by ITU-T standardization (G.709, G.879) as a new generation technology for signal transport over optical networks. It allows the following functionalities:     

Transparent signal transport thanks to the service-agnostic mechanism that maps different client services in to the same data frame level. OTN multiplexing bandwidth granularity is one or two orders of magnitude higher than for SDH technology, thus making it more scalable and supporting higher rate. It uses DWDM transport system to reach line rates of 40 Gb/s and 100 Gb/s OTN provides an integrated mechanism for forward error correction (FEC) that allows greater reach between optical nodes and/or higher bit rates on the same fiber. The introduction of enhanced OAM channels built on the experience gained from the existing SDH overhead

These functionalities ensure that adequate capacity and control exist for EP~U activities however OTN solutions do lack the ability to directly interface with legacy technologies and EPU specific applications.

45 | P a g e

Ease of Implementation and Operation Introduction of OTN switching brings major benefits in terms of lower cost and greater functionality. As networks grow and need to accommodate new types of traffic alongside existing traffic, it becomes vitally important to fill the available transmission bandwidth efficiently. This is the area of the market which OTN is designed to be most advantageous to deploy.

Interoperability (other services/technologies) The OTN supports nearly all circuit switched and packet switched services as well as all constant and variable bit rate services. The supported services are listed below: -

SDH/SONET Ethernet (Electrical & optical)/ VLAN based, port based, MAC learning… IP/VPN IP-MPLS, MPLS-TP Ethernet SyncE

Redundancy, Availability and Reliability To achieve maximum availability in the network, OTN nodes support the following redundancy and protection schemes: -

Common module redundancy: Matrix fabric, controller board, power supply, FAN… Tributary and traffic module redundancy: 1+1, 1:1 and 1:n linear protection, Shared Protection Ring (SPRing) for HO and LO ODU-k

OTN supports enhanced OAM channels that allow the monitoring of the traffic in order to trigger switching mechanisms for service protection.

Quality of Service (QoS) OTN QoS is deterministic (like SDH/SONET). In comparison with other technologies, where QoS determinism is solution specific. These mechanisms insure a guaranteed QoS especially for “mission critical” applications. OTN is optimized for multi-services transport.

Network Management System and Overall Manageability (NMS) Generally, network management of the OTN technology falls under two areas: 

OAM

OTN builds on the concept of Tandem Connection Monitoring (TCM) seen in SDH. SDH only allows one tandem connection, whereas OTN supports 6. The optical channel data unit (ODU) carries therefore TCM1-6 in the overhead section. Additionally it carries the path monitoring overhead (PM). TCM1 is used by the User to monitor the QoS that they see. TCM2 is used by the first operator to monitor their end-to-end QoS. TCM3 is used by various domains for Intra domain monitoring. Then TCM4 is used for protection monitoring by the operator. 

Interoperability with different technologies (unique platform for several technologies) 46 | P a g e

The concept of the network management system is very much vendor specific. Multi layer OTN/WDM networks generally require a network management for both the IP and the transport layer. That said, some vendors have the concept of an unique network management system for both the transport layer and the IP network. This will be very helpful for maintenance staff that is used to the circuit switched technology compared with other IP based technology.

4.4 Dynamic Internet protocol based Multiprotocol Label Switching (IP/MPLS) Life Cycle IP/MPLS is now a mature technology used by many organisations and carriers worldwide. Together, Ethernet services and IP/MPLS are widely deployed by public telecom providers. The reason for this, is the technology's ability to reduce cost and improve operational efficiency in organisations. For these reasons it is envisaged that IP/MPLS will be available as a technology for a substantial amount of time. With widely available standards-based implementations by various manufacturers, IP/MPLS adoption by the EPU is an option for the most applications since the standards and RFCs that constitute IP/MPLS are mature. It has been field-proven for many years by telecommunications carriers and more recently the Enterprise, and are well supported by various equipment manufacturers.

Scalability Networks consisting of thousands of nodes have been built with IP/MPLS technology. Most Public Telecom Operators and big Enterprises use IP/MPLS within core and access networks. IP/MPLS is not directly tied to any transport technology and thus can support any link speeds. 100Gbps links are commonly installed and support for faster links is expected as they appear. IP/MPLS networks can also support very slow link speeds. Support multipoint topologies in addition to traditional point-to-point topologies makes IP/MPLS very attractive as a potential future technology for EPU use. Many new operational communication models/protocols/applications are built natively to run in IP networks. IP/MPLS is supported in wide range of different products from small ruggedized models to carrier grade multi-chassis devices. This means it can be deployed in various places, hence addressing a wide variety of EPU use cases.

Capability to support EPU Services (as defined in chapter 2) IP/MPLS networks can support EPU Services using concepts called Virtual Private Networks (VPNs), that provide different network services to different needs. VPNs can be partially divided into native IP services (L3VPNs) and data link layer services (L2VPNs). IP/MPLS supports natively 47 | P a g e

various service topologies (namely point-to-point, multipoint, tree and anycast). Point-to-point L2VPNs are called pseudo-wires and they support most TDM services.

For a technical summary of IP-MPLS, refer to Cigré Technical Brochure from WG D2.28 “Communication Architecture for IP-based Substation Applications”.

A major reservation for EPUs of the use of packet-based technologies, in particular IP-MPLS, is the ability to support the requirements of some of its critical applications, in particular Teleprotection (see 2.2 List of Services). However, research and testing are ongoing in this area.

Ease of Implementation and Operation The use of control plane protocols in IP/MPLS comes with additional management complexity. Additional training on the operations staff is required to troubleshoot and provision IP/MPLS. This, however, can be alleviated by solutions which provide a Management System which is able to abstract the complexities of the protocol. It is therefore important for utilities which are considering IP/MPLS to ensure that the proposed vendor solution includes a capable Management System that automates the configuration of MPLS. IP/MPLS is scalable in that is simple when adding new nodes to an existing MPLS network. The required provisioning involves only configuration of the new nodes. This, of course becomes more complex as other features are used such as MPLS-TE. Expertise in the field of IP/MPLS exist primarily within the enterprise and within public carriers, it should be noted that said expertise do not align with what would be expected for an EPU.

Interoperability (other services/technologies) IP/MPLS is most commonly used as a transport protocol to carry IP traffic on Ethernet. Due to the physical protocol-independence nature of the MPLS label, it can also be carried directly within other physical and data link layers such as C/DWDM, SONET/SDH, and PDH. In terms of the types of data encapsulated by IP/MPLS:    

TDM circuits (E1/T1, SDH, etc.) can be carried by IP/MPLS but can utilise a lot of bandwidth Support for layer 2 Ethernet (point-to-point/multi-point) is good IP multicast over IP/MPLS is still relatively new to utilities Layer-3 VPN (L3VPN) and Layer-2 VPN (L2VPN)

Some IP/MPLS equipment platforms to date support a substantial amount of EPU specific hardware interfaces. Redundancy, Availability and Reliability IP-MPLS supports high availability through traffic engineering (TE) and Fast Reroute (FRR), with fault detection in the order of 50ms using mechanisms such as BFD (bidirectional forwarding detection). 48 | P a g e

The protocols surrounding MPLS-TE and FRR such as RSVP-TE have reached maturity within enterprise and carrier deployments. Link and node protection via FRR can now be computed by most manufacturer devices dynamically, easing deployment of the feature. Protection in IP/MPLS consists of two complementary technologies:  

Fast Reroute (FRR), which provides immediate short term recovery upon a failure. FRR has a deterministic switchover delay. End-to-end path protection, which provides the operator control of the path after failure, and has non-deterministic switchover delay, due to additional signalling required. Hence end-toend path protection is used with FRR to provide both short-term recovery to minimise packet loss immediately upon a failure, and then allow end-to-end path protection to take the time required to calculate the new desired path.

Quality of Service (QoS) IP-MPLS QoS is standardized by IETF in a group of RFCs. QoS in IP/MPLS consists of the following major aspects: 



Coarse-grained admission control using RSVP-TE, where signalling is performed between the LSP tail end towards the head end to determine whether there are sufficient resources to set up a path. Per-class admission control and per-hop-behaviour using DiffServ-TE

With the per-class admission control (Diffserv-TE), although it provides more control on QoS, it is more complex to deploy and requires that all nodes are Diffserv-TE aware.

Network Management System and Overall Manageability (NMS) Main features supported applicable for Power Utilities  

Inbuilt OAM and monitoring Open NMS capability (e.g. northbound interface)

The IP/MPLS protocol have a limited set of OAM defined. New developments to the MPLS standard which evolves to MPLS-TP by the efforts of IETF and ITU-T, seek to define OAM features similar to traditional transport protocols such as SONET and SDH. The OAM toolset in IP/MPLS is not as well-defined as MPLS-TP, and in comparison, IP/MPLS cannot provide transport-like OAM functionality. The OAM toolset within IP/MPLS are LSP ping (to troubleshoot label switched paths) and BFD.

49 | P a g e

4.5 Static Multiprotocol Label Switching – Transport Profile (MPLS-TP) Life Cycle In 2008 ITU-T and IETF started a Joint Working Committee where MPLS-TP started. The core elements of the MPLS-TP standard are completed but some work and agreement still remains to be added. Equipment and software based on the existing standards are available in the market from a number of vendors. MPLS-TP is in its early stage of its life cycle and the maturity of solutions/implementations is low. Scalability The goals of MPLS-TP are to simplify the transport network to a total packet based transport, to lower costs, and to improve the economics of transporting data with a simple-to-operate and resilient end-to-end packet transport network. MPLS-TP is designed for use in very large service provider networks and will scale for large implementations with high numbers of LSP/PW perport, per-card and per-system and supporting huge numbers of protected LSP with sub 50ms failover.

Capability to support EPU Services (as defined in chapter 2) Created with the objective of optimizing and simplifying MPLS for traditional transport operational models, MPLS-TP is a subset of the MPLS protocol suite that is suitable for use in packet-based optical transport networks. Some IP/MPLS features that are unnecessary in a transport context are removed. Extended functionality based on the transport requirements that are necessary to operate in a similar manner to existing transport technologies has been added. This incorporates transport-like OAM, resilience and network management operation. Pseudo wires are used in MPLS-TP to be able to transport layer 1 and layer 2 services like Ethernet, TDM etc in point-to-point applications. The pseudo wire is a connection tunnel between the PE routers that emulates a wire that is carrying L1/L2 frames. MPLS-TP also supports point-to-multipoint and multipoint-to-multipoint services by implementing Layer 2 VPNs. FIGURE 28 below gives an architecture overview of MPLS-Transport Profile (MPLS-TP).

50 | P a g e

Figure 28 MPLS-TP Hierarchy Ease of Implementation and Operation MPLS-TP is a subset of MPLS with some OAM extensions that will make it quite similar to the operation of SDH systems. The ease of implementation and operation is expected be similar to a standard SDH system, but the company may still need a different kind of competency to support(MPLS).

Interoperability (other services/technologies) Due to the physical protocol-independence nature of the MPLS label, it can also be carried directly within other physical and data link layers such as C/DWDM, SONET/SDH, OTN, and PDH. In terms of the types of data encapsulated by MPLS-TP:   

TDM circuits (E1/T1, SDH, etc) can be carried by MPLS but can utilise a lot of bandwidth Support for layer 2 Ethernet (point-to-point/multi-point) is good Layer-3 VPN (L3VPN) and Layer-2 VPN (L2VPN)

MPLS-TP equipment platforms to date support a substantial amount of EPU specific hardware interfaces. Some MPLS-TP equipment platforms to date support a substantial amount of EPU specific hardware interfaces Redundancy, Availability and Reliability MPLS-TP is using OAM packets to trigger different protection mechanisms with the possibility to configure 1:1, 1+1, 1:N and ring protection options. In the 1:1 protection option there is a working and a protection path between two nodes where the protection switching can be triggered by: 51 | P a g e

    

Detected defect condition (AIS-Alarm Indication Signal, LDI-Link Down Indication, and LKRLock Report messages) Administrative action Far end request Server layer defect indication (LOS) Retrieve timer (wait-to-restore)

FIGURE 29 gives an illustration of a 1:1 protection configuration and in MPLS-TP there is sub-50ms protection switchover.

Figure 29 Protection configuration In a 1+1 protected configuration, the standby path will be active, but the PE1 will block the standby LSP until switchover. Quality of Service (QoS) MPLS-TP supports QoS using standard MPLS Differential Services Traffic Engineering (DS-TE) architecture, which is also used in IP-MPLS. Please refer to the IP-MPLS QoS section for a description of DS-TE. Network Management System and Overall Manageability (NMS) Main features supported applicable for Power Utilities 

 

Inbuilt OAM and monitoring functionalities such as: o OAM Continuity Check (CC): proactive (BFD – Bidirectional Forwarding Detection) o OAM Continuity Verification (CV): reactive (LSP-Ping and Trace Route messages) o Alarm Suppression and Fault Indication with AIS (Alarm Indication Signal), RDI (Remote Down Indication), and Client Fault Indication Interoperability with different technologies (unique platform for several technologies) Open NMS capability (e.g. northbound interface)

Operation, Administration, and Management (OAM) have been, and are, the key focus areas of the MPLS-TP standardization work. The reason for this is that today’s TDM transport networks like SONET/SDH are extensively using this type of OAM functions to be able to meet the 52 | P a g e

requirements of the transport networks, and these functions also need to be implemented in MPLS-TP. Functions that are being implemented in MPLS-TP are fault detection (connectivity check, connectivity/path verification), fault localization and performance monitoring (delay and loss measurement). One of the goals of MPLS-TP is to make the establishment of the paths in a similar static way as the TDM systems through a NMS system. Because of this the management of a circuit based and a packet based network will be quite similar. In addition the MPLS-TP standard also has the option of a dynamic control plane (Generalized MPLS), and this could be implemented for scaling reasons and for establishment of more advanced protection functions. MPLS-TP bandwidth can be reserved, and this must be configured explicitly at each hop (on the Label Switched Path), but this bandwidth enforcement requires QoS configuration in the network In the same way as the circuit based transport networks work today, MPLS-TP also needs to support out-of-band management over a dedicated management network. An MPLS-TP transport network should function without the implementation of protocols like IP and dynamic routing protocols, and for that reason an out-of-band management network is needed, so that the NMS can reach the MPLS-TP equipment.

53 | P a g e

4.7 Provider Backbone Bridging (PBB) While aware of this technology, it is apparent that PBB has not gained traction in the market as a transport solution. For this reason, PBB is not considered as a viable option for EPU deployment.

4.8 Ethernet Life Cycle It is difficult to mark Ethernet’s place within its Product Life Cycle. It was first introduced in 1980 however there are no apparent successors to it being developed. For this reason it seems that it is one of the most matured communications technologies as well as one of the most common layer 2 technology deployed today. While a roll out of Ethernet only would be one of the fastest options for deployment. It would prove extremely difficult to manage and operate a network as it scales to large amounts of devices. For this reason the use of Ethernet solely as a transport mechanism would not be envisaged as an adequate method of deploying a network for operational use. Based on this, Ethernet will not be regarded as a potential for use as a transport mechanism

54 | P a g e

Scalability There are three separate axis associated with the scalability of Ethernet. These are:   

Maximum supported individual addressability  Hardware dependent but scalable into a number of thousands Maximum throughput within a network segment  Hardware dependent as with all technologies Maximum throughput per physical connection  Ethernet hardware come with a specific range of throughput values – 10/100/1000Mbps(over optical or electrical interface) half or full duplex mode and 1/100Gbps full duplex (over optical interface)

Capability to support EPU Services (as defined in chapter 2) Ethernet was introduced over three decades ago and has become the main technology to underpin consumer and industrial communications today. It is a mature physical layer technology which is largely deployed within Local Area Networks or in a point to point fashion. Ethernet, though successful and the foundation/end application of a lot of transport technologies discussed, is a local and link layer technology which is not adopted on it’s own as a transport solution. Simple switched Ethernet networks offer broadcast capabilities as well as logical separation of services and applications using VLANS, this enables the ability to deploy basic security functionality. Advanced networking features include spanning tree protocol and rapid spanning tree protocol allows physical redundancy to be implemented and autonomous conversion to take place. Port security and protection features such as MAC lock down, broadcast filtering and virtual LAN’s keep different applications and services separate while using the same physical infrastructure. Ethernet equipment does not directly offer termination of legacy or EPU specific interfaces. Ease of Implementation and Operation For reasons highlighted in the Life Cycle section, further discussion is not necessary.

Interoperability (other services/technologies) As above, for reasons highlighted in the Life Cycle section, further discussion is not necessary.

Redundancy, Availability and Reliability As above, for reasons highlighted in the Life Cycle section, further discussion is not necessary.

Quality of Service (QoS) As above, for reasons highlighted in the Life Cycle section, further discussion is not necessary.

Network Management System and Overall Manageability (NMS) As above, for reasons highlighted in the Life Cycle section, further discussion is not necessary. 55 | P a g e

4.9 Comparison matrix The tables below offer an overview of key features for all technologies within the brochure. Technologies are assessed based on deployments which are widely adopted within the EPU sector. The information gathered to populate these tables comes from two sources:  

Results of the D2.28 Surveys Expertise residing within the D2.35 WG

The tables depict the technical applicability of these technologies to deliver EPU Services however, it should be noted that this does not necessarily reflect a practical suitability of the technology.(Example: the availability of devices for external pole top applications) A PPLICATION

SDH

WDM

Very low latency loss intolerant, sequence and symmetry determinant traffic:





Very low latency loss intolerant





Very low latency, loss tolerant traffic:





Low latency sequence determinant traffic:



Low latency, loss tolerant traffic Latency tolerant, loss tolerant

OTN

IPMPLS E TH MPLS -TP 













































* MPLS-TP is at quite an early stage of adoption within the EPU sector. Wide scale applicability is yet to be proven Organisational considerations (BASED on SDH legacy SDH WDM OTN IP- MPLS Eth control) MPLS -TP    New skillset required?            Organisational structure concerns/opportunities? IT/OT overlap?    Seamless connectivity to all existing endpoint equipment (RTU, Relays, etc.)    Control Centre capability to interface  Non-disruptive rollout      Ease of implementation and operation (OAM)      Scaling towards an Any-to-Any solution platform *1 Requires significantly less expertise IP-MPLS but still much more than SDH

*2 MPLS implementation and operation can vary based on the method used to manage it(NMS, CLI etc) Concerns exist with regard to the maturity of the technology within EPU environments

NMS features and functionality are not standardized or controlled therefore experience can vary

56 | P a g e

Technical appraisal

SDH WDM

OTN

IP- MPLS Eth MPLS -TP

    Network Manageability *1     Scalability      Life Cycle     Interoperation with other transport technologies    Redundancy, availability and reliability – resilient capabilities      QoS *1 – WDM has a number of flavors, some passive solutions lack any capability for vision of the technology and rely upon layers implement above to gain manageability

*2 – The current maturity of OTN is unclear within an EPU sector thus difficult to classify Concerns exist with regard to the maturity of the technology within EPU environments

57 | P a g e

5 CONCLUSIONS AND PROPOSAL FOR FUTURE WORK Survey The brochure builds from information gathered within D2.28, comparing findings to information gathered from reissuing the questionnaire to utilities two years on. Interesting trends identified include: 

Utilities are gaining more of an appetite for new, alternative technologies, Ethernet and IP are becoming more prevalent in operational networks



Most utilities have begun to roadmap a new technology choice within operational environments. The survey from D2.35 however shows that the pace of transition is slower than predicted in the D2.28 survey



Apprehension exists for a number of reasons o Lack of knowledge within the EPU industry o Lack of reassurance of a successful total deployment/replacement



Technology Life Cycle - Utilities are confronted with the phase-out of installed legacy equipment and the risk of installing new solutions with typically much shorter life-cycles than they were used to. It's an opportunity to look into benefits and drawbacks of either purely packet-based technologies or hybrid networks.



Understanding of Smart Grid and IoT – The communication requirements of “the smart grid” are becoming more apparent, and are now beginning outline inadequacies of current solutions while also outlining new requirements for alternative technologies

Services To accurately assess any technology or solution within an organisation it is necessary to ensure that requirements are clearly defined. In this topic, EPU transport solutions, requirements are defined by the services which are carried by a technology. Issues identified by the brochure are: 

No formal definition of service requirements exists within a single document – Currently, not all utilities and stakeholders share a common view to the requirements of services



There is a lot of on-going work within differing IEEE and IEC groups but there is no mechanism to consolidate the status or results of these works



No evident attempts have been made to further classify services with common attributes into a number of concise groups. This will be discussed further in the next paragraph

To enable the comparison of technologies outlined in the brochure a basic structure was defined classifying different services into groups. This enables the uniform assessment of technology suitability. This approach can account for services which are currently deployed and also future services currently lacking concise requirements.

Technologies 58 | P a g e

Technology refresh can have substantial implications at an organisational level. It is important that all opportunities and risks are exposed during the evaluation of suitable technologies. Implications of this are organisation specific. The brochure assumed a control point of an EPU currently operating an SDH solution. Opportunities and risk associated with all technologies can be noted as: 

Migration to an alternative TDM/WDM technology does not have a large impact at an organisational level.



Migration to packet/frame based technologies presents new challenges and risk but also presents new opportunity from and organisational level surrounding OT/IT crossover, skillsets and knowledge.

Features highlighted as most relevant when assessing differing technologies are defined as :      

Predictable timing for time sensitive services and the ability to distribute precision time protocols Security and the secureness of a technology as opposed to the securability of technologies Ability to seamlessly interact with EPU applications and protocols Ability to segment and virtualise Scalability potential of the technology Natural resilience due to the fundamental concept of a technology

Each technology discussed presents unique benefits towards the specifications above. For this reason some technologies tend to support specific applications more easily. One such case would be point to multipoint capabilities presented by IP/MPLS which supports a huge application base. Others include the deterministic nature of SDH to support timing constrained applications such as teleprotection. The Matrix allows comparisons to be drawn between technologies and suitability for specific use cases. From the matrix it is possible to affirm the conclusions of section 4 









Synchronous Digital Hierarchy – SDH, the current incumbent within EPU networks, has the ability to carry all current EPU applications. During the roll out of newer distributed applications, it is evident that SDH does not possess suitable agility and scale to suffice as the only technology in the future. DWDM/CWDM – While the technology ensures controlled and deterministic qualities which do apply very well to support latency sensitive applications such as Differential teleprotection, CWDM is a robust technology, but it does not scale very well, DWDM does lack granularity, control and vision over a network, post deployment. It could be applied as a foundation for subsequent layers of technologies OTN – OTN presents as a technology which could support most if not all needs of a future EPU, concerns do exist surrounding the ability to interface with OTN at a low bandwidth or with EPU specific connections for legacy applications. OTN systems, in large seem to be scaled to support bandwidths well beyond the operational requirements of an EPU. This may be relevant if intent exists to commercialise bandwidth not utilized by the EPU IP-MPLS – The ability exists to support the majority of legacy, current and future applications for utilities. The two most striking concerns relate to the ability of IP-MPLS to carry differential teleprotection schemes over wide areas and the organizational change/skillset refresh involved to migrate from a technology such as SDH. MPLS-TP – Available for a number of years, and set the targets IP-MPLS needs to achieve. While achieving far more granular control during circuit provision and operation, there are 59 | P a g e





shortcomings when implemented at scale. Both technologies (MPLS-TP and IP/MPLS) will develop further in the future. However the overlap of the two will become bigger. Provider Backbone Bridging (PBB) – This technology seems to have been overshadowed within all industries by OTN and for that reason does not show much promise to have a substantial role in EPU networks Ethernet over fibre, SDH, MPLS etc. – Ethernet is not a viable solution for transportation of EPU Services. That being said it could be applied as a foundation for technologies discussed within the document or provided itself a s service “over the top” of a technology

Future Works 1 Views and decisions made by utilities, at a global level, have proven invaluable to the completion of this brochure. It has also allowed the ability to compare, contrast and trend viewpoints on technologies, affirming the conclusions of the brochure. The exercise of distributing D2.28’s questionnaire should become an independent exercise completed on an annual basis aiming to compile a global view into the trends, aspirations and concerns utilities are dealing with. This would allow utilities to individually act knowing that any decision is in line with a globally aligned movement in the same direction.

Future Works 2 No document currently exists defining the full spectrum of EPU applications and communicative requirements associated with them. Without formal definition of requirements it is not possible to classify applications and uniformly approach the assessment of suitable technologies. Work suggested in this area is a working group tasked with:     

Identifying all current and most future applications utilized within utilities and standardized Creating a uniform methodology to assess applications which can be adopted by vendors and understood by end users/network designers Document communication constraints and requirements for each application Categorise and group applications with common communicative needs Define a framework and approach to assessing technologies suitability to all application groups.

A framework, similar to that outlined above would ensure that a global assessment of technologies for EPU’s would give EPU’s, Telco’s, vendors and solution providers a common mechanism to identify, specify, categorise and affirm end-to-end requirements and solutions for all communications networks. The framework would also present an opportunity for Cigré to periodically produce reports assessing the posture of technologies including ongoing suitability, user adoptance and ranking based on advancements and the feedback from information received in Future Works 1. This would align similar to the “magic quadrant” approach to enterprise solutions.

60 | P a g e

Future Works 3 Future Works 1 and 2 highlight the need to document and identify the current standpoint of the EPU market as well as mechanisms to approach the global classification of EPU services and define a set criterion which manufacturers and vendors can meet. What are the services and where are we now. A methodology toward assessment of how to migrate services and also when to migrate services could also prove very beneficial for EPU’s to plan lifecycles and utilize existing transport assets until their end of life. Works constituting this are captured within the proposed working group “Migration Planning and Transformation Control in EPU Telecom Networks”.

61 | P a g e

ANNEX 1 SURVEY D2.35 WGD2.35 – Survey Scalable Communication Transport Solutions over Optical Networks

Survey Objective: Dear reader, Thank you very much for participating in this survey. Your contribution will help D2.35 to identify the trends in the use of IP and other communication transport solutions in the EPU environment and identify the main issues, questions and topics to be addressed in the Technical Brochure. This survey is a continuation of the survey conducted in 2009-2010 by D2.28 and will be used to identify trends, main issues and other topics of importance for the Technical Brochure to be prepared by D2.35 This survey is conducted among CIGRÉ Working Group D2.35 members and CIGRÉ members of SC B5 and D2. Please check and detail your options.

62 | P a g e

Questionnaire: 1.

Are you a:

☐

Transmission System Operator

☐

Distribution System Operator

☐

Consultant

☐

Vendor

☐

Other (please detail)

2.

Within your own company, which substation applications are using IP and IP networks? (For vendors: Which applications do you see at customer´s sites?)

☐

Substation RTU to SCADA Platform

☐

Inter-Control Centre Connections

☐

Condition Monitoring for HV devices

☐

Metering

☐

Disturbance Recorder Data

☐

Telephony System

☐

CCTV / Video Surveillance

☐

Premise Access Control

☐

Building Control (temperature, humidity, lights, etc.)

☐

Network management System and Data Communications Network

☐

Remote IP Access to Substation Assets

☐

Tele protection

☐

Current Differential Protection

☐

Time Synchronization

☐

Access to Office Applications

☐

Wireless LAN Access

☐

Other (please detail)

63 | P a g e

3.

From the list in Question 2, what substation applications (currently not running on IP) could potentially be migrated to IP?

4.

What other applications are you using that could potentially be migrated to IP (applications outside the substation)?

☐

Phasor Measurements

☐

Applications for Distribution Automation (if yes please detail)

☐

Other (please detail)

5.

What are the main operational challenges of using IP as a communication transport solution over optical networks?

☐

Security

☐

Converting Legacy Devices into IP

☐

Interoperability

☐

Ruggedness (environment)

☐

Quality of Service (QoS)

☐

Knowledge and Training

☐

Scope of Responsibility 64 | P a g e

☐

Maintenance

☐

Reliability

☐

Capital Expenditures (CAPEX)

☐

Operational Expenditures (OPEX)

☐

Other (please detail)

☐

Additional Comments

☐

Would the answer be different when using IP for substation specific application? If yes, how:

6.

What are the main psychological barriers with using IP protocols in applications?

☐

Security

☐

Reliability

☐

Quality of Service (QoS)

☐

Familiarity with Ethernet / IP Concepts and Solutions

☐

Rate of Innovation and Obsolescence

☐

Complexity

☐

Lack of Documentation

☐

Capital Expenditures (CAPEX)

☐

Operational Expenditures (OPEX)

☐

Other (please detail)

65 | P a g e

7.

Are all of your applications compatible with IP at the moment?

☐

Yes

☐

No (please detail)

8.

What is your prediction for the migration of all operational communications into IP?

☐

Already Done

☐

Short Term

☐

Within 1 Year

☐

Within 1 – 2 Years

☐

Within 2 – 5 Years

☐

Within 5 – 10 Years

☐

More Than 10 Years

☐

Never – Why?

9.

What percentage of the existing applications within your company are already using IP as communication transport solution?

☐

0%

☐

25%

☐

50%

☐

75%

☐

100% 66 | P a g e

10.

How to deal with legacy protocols and equipment?

☐

Protocol Encapsulation

☐

Vendor to Provide Solution

☐

Gateway (Protocol Conversion)

☐

Emulation

☐

Separate Network

☐

Other (please detail)

11.

Considering that all applications will move to IP, what are the requisites and concerns for the telecommunication network?

☐

Existence of Physical Infrastructure (fibre, copper, wireless, etc.)

☐

Suitable Transport Technology (PDH, SDH, Ethernet, etc.)

☐

Suitable IP Addressing Space and Scheme (IPv4 / IPv6)

☐

Suitable Substation Applications (protection, control, measurements, etc.) for IP

☐

Organisational structure

☐

Identification of Applications

☐

Other (please detail)

12.

Which underlying technologies are the most promising to provide secure and reliable transport communications in the access network?

☐

SDH

☐

CWDM

☐

DWDM

☐

OTN

☐

Ethernet

☐

IP / MPLS 67 | P a g e

☐

PBB / PBT

☐

MPLS – TP

☐

RPR

☐

Other (please detail)

13.

Which underlying technologies are the most promising to provide secure and reliable transport communications in the core network?

☐

SDH

☐

CWDM

☐

DWDM

☐

OTN

☐

Ethernet

☐

IP / MPLS

☐

PBB / PBT

☐

MPLS – TP

☐

RPR

☐

Other (please detail)

14.

Should the IP network be reserved for operational service or also be used for corporate services? Two physically/virtual separate networks or one network?

☐

Single Network, No Separation

☐

Physical separation in the core

☐

Logical separation in the core

☐

Physical separation in the access/edge

☐

Logical separation in the access/edge

15.

What type of scalability needs to be addressed? 68 | P a g e

☐

Number of Users

☐

Number of Applications

☐

Number of Protocols

☐

Variation of Protocols

☐

Number of Interfaces

☐

Number of Nodes

☐

Number of Locations

☐

Number of Services

☐

Bandwidth

☐

Other (please detail)

16.

Are you aware of any activities /documents concerning communication transport solutions over optical networks that may be of interest for D2.35 to consider for the Technical Brochure? If yes, can you please list them below:

Glossary of Terms Abbreviation

Explanation

CWDM

Coarse Wavelength Division Multiplexing

DWDM

Dense Wavelength Division Multiplexing

IP

Internet Protocol

IPv4

Internet Protocol version 4

IPv6

Internet Protocol version 6 69 | P a g e

LAN

Local Area Network

MPLS

Multi Protocol Label Switching

OTN

Optical Transport Network

PBB

Provider Backbone Bridge

PBT

Provider backbone Transport

PDH

Plesiochronous Digital Hierarchy

RPR

Resilient Packet Ring

RTU

Remote Terminal Unit

SCADA

Supervisory Control and Data Acquisition

SDH

Synchronous Digital Hierarchy

WAN

Wide Area Network

70 | P a g e

ANNEX 2 SURVEY D2.28 IP-BASED SUBSTATION APPLICATIONS

WGD2.28 – Survey IP Based Substation Applications Survey Objective: Get an overview of the present use of IP-based applications within the substation environment, between Substations and for Substation - to - Control Centre communications. This survey is conducted among CIGRÉ Working Group D2.28 members and CIGRÉ members. Please check and detail your options.

Questionnaire: 1- Are you a:

Transmission System Operator/Distribution System Operator Vendor Other (please detail):

2- Within your own company, which substation applications are using IP and IP networks? (For vendors: Which applications do you see at customer´s sites?)

Substation RTU to SCADA platform Inter-Control Centre Interconnections Condition monitoring for HV devices Metering Disturbance recorder data Telephony system CCTV/Video Surveillance Premises access control Building control (temperature, humidity, lights, etc.) Network Management System & Data Communication Network Remote IP Access to substation assets Tele-protection Time synchronization Access to office applications Wireless LAN Access

Other (please detail):

3- What are the main operational challenges using IP into the substation environment?

Security

71 | P a g e

Converting legacy interfaces into IP Interoperability Ruggedness (environment) Quality of Service Knowledge and Training Scope of Responsibility Other (please detail): Additional comment:

4- What are the main psychological barriers with using IP protocols in Substation applications?

Security Reliability Quality of Service Familiarity with Ethernet/IP concepts Rate of innovation and obsolescence Lack of documentation Other (please detail):

5- Are all of your applications compatible with IP at the moment?

Yes No (please detail):

6- Prediction to migration of all operational communications into IP?

Already done Short term 6 months 1-2 yrs 5 yrs More than 10 yrs Never - Why?

7- How to deal with legacy protocols and equipment?

Protocol encapsulation Vendor

72 | P a g e

Gateway (protocol conversion) Emulation Separate network Other (please detail):

8- Is the existing communications network ready for IP traffic?

Yes No (please detail):

9- Requisites and concerns for the telecommunication network?

Existence of physical infrastructure (fibre, copper, wireless, etc) Suitable transport technology (PDH, SDH, Ethernet, etc.) Suitable IP addressing space and scheme (IPv4 / IPv6) Identification of Applications Other (please detail):

10- Which underlying IP technology is the most promising to provide secure reliable IP communications?

MPLS (IP over fibre) Ethernet/IP over SDH Ethernet/IP over DWDM Other (please detail):

11- Should the IP network be reserved for operational service or also be used for corporate services? Two physically/virtual separate networks or one network?

One only network, no separation Separate network (physical) Separate network (virtual/logical) 12- Which technology is more promising to separate and provision different types of IP services?

Virtual LAN Virtual Private Networks (Layer 3) Virtual Private LAN Service - VPLS (Layer 2) Multicast filtering Other (please detail):

73 | P a g e

ANNEX 3 WGD2.35 SURVEY RESULTS The following sections contain a summary of the survey results

1 Substation applications using IP The participants were asked what substation applications are currently using IP.

74 | P a g e

Figure 30: Substation applications using IP .

75 | P a g e

Other than the applications listed in the figure above, participants indicated that they use IP for the following applications: -

IEC 61850 station bus LANs Distribution System SCADA – Mesh radios to reclosers Third party IP-WAN for low risk SCADA Next generation service distribution

2 Substation applications eligible for use with IP After having specified the applications that are currently using IP, the participants were asked what applications that are currently not using IP would be eligible for using IP. To be able to interpret the results better, the applications already using IP from the previous question have been added the figure. For example, if only a small number of participants indicate that a certain application would be eligible for use with IP in the future, is this because the application is already using IP or because they think the application is not suitable for IP? The blue bars in the figure below represent the applications that are already using IP form the previous questions, the red bars represent the applications that participants indicated would be eligible for use with IP and the green bars represent the total (current use of IP + eligibility for future use of IP).

76 | P a g e

Figure 31: Applications eligible for using IP The relation between the scores for “existing” and “eligible” can provide a good indication of the current status of IP implementation for the specified applications. Below provides an overview of this relation and the possible conclusions. 77 | P a g e

Score Existing

Score Eligible

Conclusion

Equal

Equal

Migration to IP is currently being done, or will likely be done in the near future Example: Condition monitoring for HV devices

Higher

Lower

This application has mostly migrated to IP Example: Network Management System & Data Communication Network

Lower

Higher

This application is expected to migrate to IP in the future Example: Tele-protection Table 1: (Future) IP usage

The totals, represented by the greens bars in can provide an indication of how suitable the participants think IP is for certain applications. The vast majority of the participants seem to think that Tele-protection and Current Differential protection might not be suitable for use with IP.

3 Applications outside the substation eligible for use with IP When asked what applications participants are using outside the substation that could potentially migrated to IP, 35% of all participants indicated that they are using Phasor measurements. Other applications indicated by the participants include: -

Operational voice radio communication Recloser/RMU connectivity to the substation for information exchange Wide area substation control Climate information for Variable Line Rating and Dynamic Line Rating. Distribution Feeder Automation Scheme Power Quality Monitoring High Speed Monitoring Remote control of RMU Metering Sensors Hydro meteorological Reservoir Monitoring

4 Main operational challenges using IP a communication transport solutions over optical networks When asked what operation challenges participants see for using IP, most participants indicated that they think Security, Knowledge and Training and converting legacy devices into IP are the biggest challenges. An overview of the results can be found below.

78 | P a g e

Figure 32: Operational challenges Other challenges mentioned by the participants are: -

Applications with critical time requirements Rate of innovation and Obsolescence Latency for tele-protection Pv6 (i.e. complex / unknown) Network topology (i.e. Mesh / Ring / L2 / L3) Lack of documentation Complexity By interoperability we are especially concerned at the protocol level (OPC, DNP3, SNMPV3, 61850) People have their “favourite” protocols.

5 Main psychological barriers with using IP protocols in Substation applications After having been asked what the participants saw as the main operational challenges for implementing IP, they were asked what psychological barriers they have for using IP. The results can be below

79 | P a g e

Figure 33: Psychological Barriers Beside security and familiarity with IP being the most important operational challenges, they also represent the most important psychological barriers.

6 Are all applications compatible with IP at the moment? To get an indication of how much work it would be to migrate all applications that utilities are currently using to IP, participants were asked if all their applications are currently IP compatible. The results can be seen below (note that the totals for yes and no might not be 100% since not all participants answered this question).

80 | P a g e

Figure 34: All applications IP compatible? The vast majority of participants indicate that not all of their current applications are IP compatible. Applications of which participants indicated that they are not compatible with IP include:         

Revenue meters Non availability of IP networks at some locations (Tele) protection Differential protection Time synchronization Premise access control Power quality Legacy systems PABX

7 Prediction to migration of all operational communications into IP The results from the previous question clearly show that not all applications are currently using IP. The participants were asked to provide an estimate on when they think all of their operational communication would be IP based. The results can be seen in below.

81 | P a g e

Figure 35: Predication of All operational traffic through IP Almost all participants expect that they will eventually have all of their operational traffic over IP. Almost three quarters of the participants expect to have all their operational traffic migrated to IP within ten years, while twenty-six percent expect that complete migration might take more than ten years.      

 



Existing technology is expected to work for the next 10 years Tele-protection is expected to be migrated later, or might even never be migrated Near real time performance of services over IP and latency variation over redundant IP routes is not yet mature enough Securing an IP network for operational communications is a challenge As assets get replaced, things will migrate to IP. This is already happening but might take more than ten years to complete. Robustness of the service = IP is always considered “flakey”. Acceptance of the technology. Increased Capex and OpEx for IP services. Shorter life cycle components = higher Capex. Increased level of vendor support for software = higher OpEx. Maturing of Deterministic networks. Migration is mainly driven by costs savings, equipment obsolescence and WAN availability Still no solution for latency requirement for legacy teleprotection and Current Differential Protection. Replacement strategies for such are based on direct optical connection and hence not on an IP network Migration must be a step-by-step process verifying the results after each step

82 | P a g e

8 Percentage of the existing applications using IP as communication transport solution? To get an indication on the current state of IP migration, the participants were asked to indicate what percentage of their current applications is already using IP. The results can be seen below

Figure 36 Percentage of existing applications using IP Almost all participants are already using IP, most of them even for at least half of their applications.

9 How to deal with legacy protocols and equipment? Many utilities have legacy equipment that has been designed to last for several decades. It might not be cost efficient to replace this equipment with IP based equipment as long as this legacy equipment is working as it is supposed to. New IP-based equipment will have to be able to communicate (to some extent) with this legacy equipment. Participants were asked what they thought was the best way to deal with these legacy systems. The results can be seen below 83 | P a g e

Figure 37How to deal with legacy protocols/equipment Most of the participants believe that a good solution is to keep the legacy networks separate were possible, and to use gateways or protocol encapsulation where information needs to be exchanged between different networks. Almost twenty percent of the participants think that vendors should provide a solution. Some of the comments made by the participants are:    

Currently rolling out a SDH bearer network with an MPLS network overlaid on it. Legacy protocols will be transported on the SDH network. IP services will use the MPLS network. Each network will have separate VPNs and VRF Migration to IP dedicated operational equipment. Replace legacy equipment

10 Requisites and concerns for the telecommunication network The survey held by Cigré working group D2.28 showed that many utilities have not yet (fully) migrated their communication networks to IP. The participants were asked what their main requisites and concerns were for having IP-only networks. The results can be seen below.

84 | P a g e

Figure 38 Requisites and concerns for using only IP The most pressing concerns for only using IP that the participants indicated are the suitability of their substation applications IP and vice versa. As can be seen, utilities are concerned about all of the specified options. Beside the options shown in the figure below, participants indicated that they have to following concerns:   

Reliability Quality of Service (QoS) traffic load and segmentation 85 | P a g e

 

 

Time and economy Maturity of applications. Experience has shown that many applications are converted to IP rather than being developed with IP in mind. This leads to issues with security, stability, and multiple software revisions. Product testing may not be robust enough. Skills & knowledge Operation & Management

11 Most promising technologies are to provide secure and reliable transport communications in the access network? When asked about the most promising technologies for the access network, results show a clear preference for Ethernet over fibre, SDH and IP/ MPLS. The complete results can be seen below.

Figure 39 Most promising access network technology

12 Most promising technologies to provide secure and reliable transport communications in the core network? Participants believe that SDH and IP/ MPLS are not only suitable for the access network, but also for the core network. Other promising technologies for the core network are DWDM and MPLS /TP. The complete results can be seen below.

86 | P a g e

Figure 40 Most promising core network technology

13 Should the IP network be reserved for operational service or also be used for corporate services? Both operational services and corporate services can be hosted on the same network. A single IP network hosting both corporate and operational services means that all information can potentially be accessed from anywhere with a minimum of required equipment, making such a network cheap to install and maintain. For security reasons however, it would be much safer to have separate networks for corporate and operational services. Participants were asked what they think is the best way to handle this. The results can be seen in below.

87 | P a g e

Figure 41 Seperate operational services from corporate services?

14 What type of scalability needs to be addressed? IP networks can be very scalable, provided the required scalability was taken into account in the network design. A scalable IP network might be more complex, more difficult to maintain and more expensive than an IP network that is only designed to facilitate a pre-defined set of functionality. On the other hand, if functionality would have to be added to a non-scalable IP network, it might mean that the whole network would need to be redesigned, which could cost a lot of time and money. The most efficient network design is one that facilitates only the required scalability. Participants were asked to indicate what scalability they think is needed in their networks, to identify what scalability utilities require. The results can be seen below.

88 | P a g e

Figure 42 What type of scalability to address?

89 | P a g e

ANNEX 4 NETWORK TIMING Substation automation and system control are mission-critical processes and electric power utilities must synchronize applications and use cases across large-scale distributed power grid substation networks to ensure grid stability. Precise timing is used to improve reliability, better understand operations of the power system, predict and prevent local and system-wide faults, for testing and verification, and to reduce costs.

Especially at transmission level, HV-line protection remains a critical application. Line differential protection (with its stringent timing requirements) across Packet Switched Networks is not yet solved from the perspective of the majority of utilities (feedback e.g. from EUTC conference 2013) Utilities use precise timing to accurately timestamp data, measurements and events and share it between power control, protection and measurement equipment. Timing can be used within substation environments, and also across the WAN to exchange information between substations and control centers. Timing can be exchanged within a single application or shared between multiple applications. Precise timing is used for real-time processes as well as post-event analysis. In addition to substations and grid management, timing is also essential in data center and operating center local area networks, and throughout the EPU telecom and wide area networks. By looking at the overall network picture, a utilities form strategies and practices to protect their timing and synchronization infrastructure. Historically millisecond and tens of microsecond accuracy has been provided by SNTP and IRIG-B amongst other mechanisms for clock redistribution. However, newer applications such as PMUs and protocols such as 61850 Sampled Values (SV) require microsecond accuracy. Accurate clocks enable sophisticated analysis of real-time and post-disturbance faults and events within a short amount of time. This leads to faster and more reliable fault isolation and resolution, and better planning of transmission resources on the electric grid.

90 | P a g e

Figure 43 Applications sensitive to time and frequency While all have differing timing requirements, below outlines some of the applications which require accurate timing: Analogue trace recorders Energy Management Systems Lightning strike correlations Merging unit and sample value (IEC 61850-9-2) measurement Power quality measurement SCADA Sequence of event recording Substation automation Synchrophasor measurements System frequency management (AGC) TDM network synchronization Time of use tariff metering Wide area control and protection Networking/communications equipment Line Differential Protection Teleprotection

Electric power utilities use precise GPS based timing at their power transmission and distribution facilities to time stamp data and measurements. Time synchronised data can be analysed to assess causes of faults or disruptions on the power grid, and is an essential component of newer real-time control and protection systems such as System Integrity Protection Schemes (SIPS) and Centralised Remedial Action Schemes (C-RAS). Without accurate time stamps, the collected data cannot be correlated and becomes worthless. This reliance on GPS has been highlighted as a potential weakness of the traffics above and for this reason it would be preferable to attain timing similar to that available within GPS streams through technologies deployed on Fibre Optics

Synchronisation Types There are typically two types of time synchronization in EPU networks: direct and indirect over a communications network. The internal clock of a device will be synchronized to master clock or server which is normally connected to an accurate time source such as GPS, or quantum clock. The accuracy and variance of the time will vary depending on a number of factors including communications network/media, distance, protocols, traffic, interference and errors.

91 | P a g e

For direct synchronization a standalone communications network for timing redistribution is required which can be expensive. These are typically fiber, copper or coaxial networks. This approach is normally implemented for intra-substation timing and uses protocols such as IRIG-B, GPS and 1PPS. Synchronization over a communications network (LAN or WAN) redistributes time over a communications media such as Ethernet, and is usually part of a converged (timing and other data) network, reducing cost. Traditionally NTP/SNTP have been used to synchronise devices, and more recently 1588 PTP has become an alternative or an essential option depending on required accuracy of an application. In addition Synchronous Ethernet (SyncE) can be used to redistribute frequency over the WAN and mitigate clock drift between WAN devices.

Figure 44 Time synchronisation application

IRIG-B Overview Inter-range instrumentation group time codes B (IRIG-B) and is an industry standard for GPS time synchronization. IRIG-B can be applied in a substation for power quality and system stability monitoring, sequence of events recording and accurate time stamping for revenue billing (1 ms). IRIG-B needs to be transmitted via a dedicated timing network (usually of copper or coax cable) and is therefore not a lowcost solution. IRIG-B needs an external time source such as GPS, can be modulated (over a carrier signal) or unmodulated (no carrier signal), and can deliver accuracy in the microsecond range (as low as 10-100 microseconds). It is widely used in the electrical power industry with around 90% of intra-substation timing use cases, and may contain time of year, year, and seconds of day information.

Global Positioning System (GPS) Overview

92 | P a g e

The GPS system is highly redundant (although there are concerns over security of the system) and is used in the EPU industry for both direct time synchronization and as a time source for other time protocols. For direct synchronization an antenna per IED or device is required which can be a costly solution, or GPS can be used as a time source to be redistributed through a timing network (dedicated or shared). The GPS system is capable of delivering timing accuracies of within ten nanoseconds, Global Positioning Systems (GPS) are used to provide a precise time reference for power system devices in an electrical network, ensuring time synchronization even across dispersed networks (substation or distribution networks) and over large geographical distances. In the distributed model a GPS antenna receives a time signal and a receiver converts it to one or more time protocols which are distributed to power system devices which in turn set their internal clock. The Global Positioning System provides a very accurate time reference, however when GPS is not available the system cannot operate as intended for long, and therefore vulnerabilities of the GPS have become a major concern. Other available satellite based technologies using the same principles are GLONASS, GALILEO and COMPASS 3.2.3. Impairments affecting satellite based timing systems

The reception of the signals is subjected to many error sources, which can lead to degraded system performance. The most common sources and their effects are: 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12.

Signal arrival time measurement Atmospheric effects Multipath effects Ephemeris and clock errors Geometric dilution of precision computation (GDOP) Selective availability Anti spoofing Relativity Special and general relativity Signal distortion Natural sources of interference Artificial sources of interference

Contribution of the errors on the position and timing inaccuracies SOURCE OF ERROR Signal arrival time Atmospheric effects Multipath effects Ephemeris and clock errors Geometric dilution of precision Selective availability Antenna spoofing Relativity effects Signal distortion Interfering signals

DISTANCE ERROR ±3m ±7m ±2m ± 2.5 m ± 12 m ± 100 m ±1m 16,19 km/day 90 m ± 30 m

TIMING ERROR 10 ns 23.4 ns 6.67 ns 8.34 ns 40 ns 334 ns 3.33 ns 54 µs/day 300 ns 100 ns

93 | P a g e

Noise in radio path

± 50 m

167 ns

The effects mentioned above may not directly effect any of the applications mentioned within this document but could contribute as a tipping point for overall latency and delay thresholds within a system. These can be mitigated.

One Pulse Per Second (1PPS) Overview Generally a GPS receiver gives out a high precision one pulse per second to mark exact second intervals. This signal is used for precise timing and synchronization. This is then forwarded to each individual user over dedicated communication wiring lines, and is therefore a costly solution to implement. The GPS receiver produces a 1PPS pulse with a defined level and pulse length, and accuracy can be in the tens of nanosecond range (typically 40 ns).

NTP/SNTP Overview NTP (Network Time Protocol, RFC 1305) is a software method to transfer time between clients and servers over packet switched variable latency data networks, and was designed to minimize the effects of variable latency. Although most commonly used over UDP/IP protocol, it can also be transported via TCP/IP. It has been particularly designed to resist the effects of variable latency.

NTP provides reasonable accuracy, from a few milliseconds up to a few hundred milliseconds depending on the nature of the connection between the NTP client and the server. NTPv4 can usually maintain time to within 10 milliseconds over the public communications networks such as the Internet, and can achieve accuracies of 200 microseconds or better in local area networks under ideal conditions. NTP is often used for the time synchronization of communications network devices such as routers and switches on the EPU communications network.

SNTP (Simple Network Time Protocol, RFC 2030) describes a simplified form of NTP. While they both use the same network package format, but lacks some internal algorithms that are not needed for all types of servers.

IEEE 1588 PTP / IEC 61588 PTP PTP stands for Precision Time Protocol and is a standard Ethernet protocol described in the standards IEEE 1588 and IEC61588 for time synchronization. It is seen as a cost effective solution which can be implemented over a shared Ethernet network in a substation, and across a WAN. Software versions of 1588 PTP provided similar time accuracy as NTP on a LAN, and therefore a hardware implementation was required to achieve microsecond consistent accuracy. This also meant that both timing and data could exist on the same communication port on a substation device, as well as a communication device, with sub microsecond accuracy. 1588 is capable of accuracy in the 20 to 100ns range This type of accuracy is essential for newer applications such as SIPS and C-RAS, and protocols like IEC 94 | P a g e

61850 GOOSE and SV. Unfortunately some security problems have been reported concerning to this protocol and his use must be restricted in protected areas where the intruders are kept out. Versions of 1588 1588 (2002) – Revision 1 "Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems" IEC 61588 (2004) – Revision 1 1588 (2008) – Revision 2 introduces =