SAER-6114

Engineering Report SAER-6114 22 May 2007 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture Document Responsibility: Process & Control Systems Department

Previous Issue: New Page 1 of 63 Primary contact Almadi, Soloman Musa on 966-3- 8739939 Copyright©Saudi Aramco 2007. All rights reserved.

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

Table of Contents 1

Executive Summary.................................................................................................... 4

Part I: Wireline Network Architecture 1 2 3

4

5

Introduction................................................................................................................. 7 Background................................................................................................................. 7 Existing SCADA Network Design............................................................................. 10 3.1 SCADA Over Dedicated Transmission.......................................................... 10 3.2 SCADA Over Dedicated SDH Network…….................................................. 12 3.3 SCADA Over IT WAN Packet Network.......................................................... 15 3.4 SCADA Over Radio Network……………………………………...…............... 17 Industry Trends......................................................................................................... 17 4.1 Purdue Enterprise Reference Architecture (PERA)....................................... 17 4.2 Instrumentation, Systems and Automation Society....................................... 18 4.3 Industry Trends Summary............................................................................. 19 Future I-Field/SCADA/Remotes Communication Network Architecture................... 21 5.1 Existing Performance & Reliability Issues.................................................... 21 5.2 Analysis & Results......................................................................................... 22 5.3 Remote Process Automation Application Network Architecture.................... 24 5.3.1 Future Process Automation (SCADA, I-Field, Remote sites) Network Design Architecture……….................................................. 25 5.3.2 Next Steps......................................................................................... 32

Part II: Process Automation Wireless Access 1 2 3 4 5

Background............................................................................................................... 33 Introduction................................................................................................................ 33 Wireless for Process Networks................................................................................. 33 Industrial Wireless vs. Non-Industrial Wireless......................................................... 34 Industrial Wireless Classifications............................................................................. 35 5.1 Wireless Personal Area Networks (WPAN)................................................... 36 5.2 Wireless Local Area Networks (WLAN)......................................................... 38 5.3 Broadband Wireless Connectivity.................................................................. 42 5.4 Wireless Wide Area Networks (WWAN)........................................................ 43

Page 2 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

Table of Contents (Cont'd) 6 7

Wireless for Process Network Conclusion................................................................ 45 Next Steps................................................................................................................ 45

Part III: Network Timing & Synchronization Architecture 1 2

Synchronization Architecture..................................................................................... 46 Next Steps................................................................................................................ 48

APPENDICES.................................................................................................................... 49 Appendix 1: Design Performance Criteria Analysis................................................. 50 Appendix 2: Bandwidth Calculation for Wireline Network........................................ 53 Appendix 3: Wireless Technology Comparison........................................................ 56 Appendix 4: Wireless Technology Classifications.................................................... 57 Appendix 5: ISM Frequency Band............................................................................ 58 Appendix 6: IEEE 802.11 WLAN Standards Model.................................................. 59 Appendix 7: Support Structure………………………………………………………..… 60 GLOSSARY....................................................................................................................... 61

Page 3 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

1

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

Executive Summary This report summarizes the findings of a detailed assessment of existing Saudi Aramco implementations, industry trends, and recommends network design architecture for data acquisition, control, and monitoring for remote Process Automation applications; which includes I-Field and SCADA. This report also covers other non-real time process control applications such as Cathodic monitoring, Vibration monitoring, etc. The network infrastructure covered in this document, includes Wireline Networking, Wireless Networking, and Network Synchronization. Recommendations outlined in this document will be mapped to the different Process Automation standards and procedures (e.g., SAES-Z-010, SAES-Z-001, 23-SAMS-020, draft SAES-Z-004, etc...). In the interim, this report will serve as a guide for any future SCADA, I-Field, and remote site network connectivity until all related standards are updated with these recommendations. These recommendations are based on benchmarking of international organization standards, industry trends, existing Saudi Aramco different implementations, and lessons learned. The outcome of this effort shall provide higher level of end-to-end network reliability, flexibility, and optimal design configurations. In summary, the recommendations call for the following: I. Wireline Network Architecture: The remote Process Automation applications network architecture shall be based on either one of the following configurations: 1.

Industrial Ethernet using dedicated fiber in a self healing ring topology for grass root projects such as Khurias, Manifa, etc.

2.

For existing implementations, utilize Synchronous Digital Hierarchy (SDH) dedicated bandwidth based on self healing ring architecture.

These two network toloplogies implemenation shall be based on the following general guidelines: •

A standalone fiber optics cable shall be utilized to connect remote sites (i.e., well sites) to a centralized Process Automation Network (PAN). Alternatively, Radio links, or Broadband wireless network solution could be used to connect dispersed remote sites where fiber optics cable installation proves economically not feasible.

•

Self healing ring topology is a ring network topology based on a uni/bidirectional links between a set of stations providing optimum protection against fiber cable cut and equipment failure.

Page 4 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

•

Utilize point-to-point ring system for scattered sites and/or where a true closed loop physical ring topology is economically not justifiable.

•

A PAN comprising of multiple scattered (PANs), shall interface with the Corporate Network via a centralized firewall(s). These scattered PANs shall be connected together to form a unified PAN utilizing the corporate SDH transmission infrastructure with a dedicated bandwidth of 10 Mbps. Alternatively, dark fiber can be utilized if available.

•

SCADA over IP/ATM is a unique implementation for East/West Pipelines. It was recommended to PMT to move SCADA traffic from ATM and have it directly transported over a dedicated SDH transport network to fulfill the physical separation requirements as per SAES-Z010. This would be part of the ongoing BI (BI-10-00179) that will replace the existing Microwave system to SDH. Any future implementation similar to the East/West Pipelines such as Water Injection Plants, scattered GOSPs will be based on either one of the two design options stated above. This is mainly because remote GOSPs or WIPs operation is tightly coupled with centralized process control systems.

Detailed network analysis and recommended guidelines for the Wireline Network Architecture are included in Part I, section 5.3 of this report. II. Wireless Network Architecture: Wireless network solution provides cost effective and efficient connectivity for different Process Automation systems. The following are the recommended wireless connectivity solution mapped to their associated applications. 1.

Zigbee (in conjunction with IEEE 802.15.4): wireless solution considered as a potential emerging short distance wireless technology for instrumentations networking, subsystems networking of network latency tolerant applications.

2.

Wireless LAN (based on standards IEEE 802.11a, b, and g technologies): considered a potential mature and proven wireless technologies for mid range (100’s meters up to few kilometers) applications.

3.

Wireless Wide Areas Network (e.g., GSM phone based service such as Short Messaging Service, General Packet Radio Systems): can be utilized to provide efficient and cost effective process monitoring solution for semi-real time noncritical applications.

4.

Broadband Fixed Wireless connectivity (WiMAX or Propriety solution) •

Provides coverage up to 50km+ radius and data rates up to 75+ Mbps

Page 5 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

•

Provides a cost effective and flexible point-to-point connection

•

WiMax requires frequency approval from local authority Communications and Information Technology Commission (CITC) and should be requested through a 3rd party local service provider

Hence, the strategy is to use propriety Broadband Fixed Wireless Solution to avoid licensing cost and dependence on 3rd party service providers. P

P

Detailed network analysis, benchmarking, and recommended next steps for Wireless Network Architecture for Process Automation are included in Part II of this report. III. Network Synchronizations: Network synchronizations will be based on Global Positioning System (GPS) coupled with Network Timing Protocol (NTP) server within a plant to provide precise time stamping for systems and network infrastructure. To minimize cost, existing plant’s GPS/NTP servers will be extended to its remote sites. Detailed analysis and recommended next steps for the Network Synchronizations are included in Part III of this report. P&CSD will utilize and benchmark ongoing technology trials in both Wireless and Wireline networks to develop Saudi Aramco engineering deployment guidelines such as material specifications, standards, and best practices. As part of the overall recommendation, future Capital Projects (i.e., Budget Items) for Process Automation networks will mainly be referenced under the Z standards index (Process Control). Telecommunication T-index will be sub referenced for overlap areas.

Page 6 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

Part I: Wireline Network Architecture 1

Introduction This part of the report provides recommended network architecture for SCADA systems, I-Field, and other remote site Process Automation applications. The recommended architecture is based on a comprehensive network evaluation for existing Process Automation applications that include SCADA systems, ongoing I-Field initiative, and other remote sites Process Automation requirements. This evaluation includes network topologies, protocols, reliability and recommended network configuration options. Selected Saudi Aramco SCADA and I-Field implementations have been used as a baseline in this assessment. Further, industry trends and international organization bodies (e.g., ISA, API, etc.) have been used to provide an optimal recommendation that would meet existing and future Process Automation applications requirements.

2

Background In April 2006, an assessment to existing Saudi Aramco Standards and Material specifications revealed the need for comprehensive detailed requirements for Process Automation Network (PAN) to address SCADA and other remote site Process Automation applications’ communication network. In addition, different network implementations types in Saudi Aramco have emerged over time based on different site topologies and the available technologies at that time. These can be described as follows: 1.

Dedicated Transmission network was used for some of these implementations; while others were based on transmitting Process Automation traffic utilizing shared transmission infrastructure.

2.

These different network implementation either using self healing rings topology, or point to point redundant links, or single threaded links.

3.

Propriety protocols were used as a defacto of the applications. However, there are some implementations that are based on open standard communication protocols.

These different configurations have caused a non streamlined network design implementations and created a gap in Process Automation Standards. Here is a summary of what the existing standards, as of April 2006, are calling for a long with high level analysis of these standard requirements.

Page 7 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

●

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

SAES-Z-010: ○ Scope: This standard establishes the requirements for design, installation, configuration and commissioning of Process Automation Networks (PANs). “Geographically spread Remote Terminal Units (RTU's) are not in the scope of this standard” ○ This standard calls for: 5.2

“The network design shall provide physical and logical separation between PAN and Corporate Network below firewall. Logical separation, at minimum, is mandatory for network connections above firewall”

○ Issue: The standards scope does not address remote areas (RTU’s) connectivity. ●

SAES-J-902: ○ Scope: This Standard establishes the design criteria for the installation of power and wiring systems for electrical instrumentation. ○ Standard calls for: 12.3.2

“Data Links: Data links, including fiber optic cables, shall be specified and installed per system manufacturers' recommendations. When redundant data links are provided, the primary cable shall follow a different route from the back up cable. Primary and backup data link cables shall preferentially enter cabinets or consoles from opposite sides. Data link cables shall not be routed in the same conduit, duct, or tray with other instrument cables.”

○ Issue: The scope does not address SCADA or other remote sites’ Process Automation application related traffic. It is focused on the instrumentation. ●

SAES-Z-001: ○ Scope: This Standard prescribes the minimum mandatory requirements and guidelines governing the engineering, design and installation of Process Control Systems (PCS) in Saudi Aramco plants (i.e., Oil/Gas Plants, Refineries, and Pump Stations).

Page 8 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

○ This standard calls for: 14.3

“Interface to Corporate Wide Area Networks (CWAN): The control system communication to Corporate Wide Area Network and other non-control computer systems shall be designed to ensure that no failure, no request for information, or network loading problem will impact the performance or availability of the PCS. Use of standard software and hardware protocols for interfaces, such as TCP/IP and OPC, are preferred.”

○ Issue: The standards calls for generic requirements (e.g., no failure, no loading problem will impact the performance, etc…). The standards does not address communication medium (e.g., physical separation vs. logical separation). ●

23-SAMSS-010: ○ Scope: all DCS equipment and associated software required to monitor and control a process plant. ○ This standard calls for: 11.2

“Communication at the control network level shall have redundant or fault tolerant paths. Communications from the controller to the I/O subsystem shall have redundant paths.”

11.3

“DCS internal communication shall be designed such that no single failure will degrade the performance of the system. This requirement applies to all communication between DCS modules, including communication between controllers to their I/O modules.”

○ Issue: The standards scope does not address SCADA or remote areas (RTU’s). Further, the standard calls for fault tolerant network but does not address the communication in detail.

Page 9 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

●

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

23-SAMSS-020: ○ Scope: This specification defines the minimum mandatory requirements for Supervisory Control and Data Acquisition (SCADA) systems for oil & gas and utility applications. ○ This SAMSS calls for: 13.1.4

“The system LAN shall be fault tolerant utilizing a network configuration that prevents a single point of failure.”

○ Issue: The standard calls for fault tolerant network but does not address the communication in detail. 3

Existing Remote Network Connectivity Design There are three (3) different existing remote network connectivity designs in Saudi Aramco. These network design topologies have evolved over the past many years. These topologies are: 1.

The earliest is SCADA network based on dedicated point-to-point digital circuits over digital transmission network and then evolved to be over Synchronous Digital Hierarchy (SDH) transmission network. OSPAS and PDD are the primary users for this network topology at this time.

2.

SCADA using dedicated bandwidth on shared, transmission systems such as Synchronous Digital Hierarchy (SDH). This is typically implemented within oil and gas fields (e.g., Qatif, Haradh, etc.).

3.

The third network design option is based on carrying SCADA traffic over IP then over ATM and then over SDH. This is limited to East /West Pipelines network design implementation and Shaybah’s inter GOSPs connectivity.

Detailed design overview for these different existing options is outlined in the subsequent sections. 3.1

SCADA Over Dedicated Transmission The earliest SCADA network implementation was based on dedicated point-topoint digital circuits over Digital transmission transport network or Synchronous Digital Hierarchy (SDH) networks. This implementation is based on using dedicated point-to-point digital circuits with modems at both ends of the circuit or a radio link. The circuit provides connection between the Remote Terminal Unit (RTU) at the remote site to the SCADA Terminal Sever at the control room. Figure 4.1 depicts the network topology for this access method. Page 10 of 63

Document Responsibility: Process & Control Systems Dept.

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

Issue Date: 22 May 2007

Terminal Server 1

2

3

4

5

6

7

8

9

1

OMNI SW ITCH 9W X

O m n i S w i tc h

2

3

4

5

6

7

8

9

OSPAS Center

VACANT S LOT

Fast Ethernet

V ACANT SLOT

E SX

V ACANT SLOT

E SX

ASX -155 RFS

32 UTP 10/100 Mb ps - E ther net

1 1

2

V ACANT SLOT

VACANT S LOT

V ACANT SLOT

E SX

V ACANT SLOT

S

S

32 UTP 10/100 Mb ps - E ther net

MP X P

P

2 PORTS OC-3 S MF - ATM

E SX

ASX -155 RFS 1 MP X

32 UTP 10/100 Mb ps - E ther net

2 PORTS OC-3 S MF - ATM

V ACANT SLOT

32 UTP 10/100 Mb ps - E ther net

2

2

OMNI SW ITCH 9W X

O m n i S w i tc h

5 0 0 P P

S

1

P

S

2

5 0 0 P

Ethernet Switch 1

2

3

4

5

6

7

8

9

m

n i

S

w

itc h

1 MP X

E SX

E SX

2 P ORT S OC-3 S MF - AT M

VC A ANT S L OT

32 U T P 1 0 /1 0 0 Mb p s - E th e rn e t 0

V ACANT S L OT

32 U T P 1 0 /1 0 0 Mb p s - E th e rn e t 0

V ACANT S L OT

AS X - 51 5 RF S

2

OMNI S WIT CH 9 WX

O

5

V ACANT S L OT

Fiber Optics MUX/Modems connecting digital Point to Point Circuits

P

S

1

P

S

2

P

IT SDH Transport Network

Fiber Optics MUX/Modems connecting digital Point to Point Circuits RTU

UHF Radio

Fiber Optics Mux

RTU RTU

Figure 3.1: Dedicated Point-to-Point Circuits Typically, a RTU is either connected by a digital circuit (2 wire or 4 Wire E/M) or a radio link from remote sites to the nearest transport network node Digital Access Cross Connect System that is part of the Saudi Aramco SDH network. The majority of the dedicated lines operate at low speed, i.e., 1200 to 9600 bauds. Further, many of these dedicated lines utilize twisted telephone line pairs party line features; which can have up to six (6) RTUs connected. Remote sites supported by leased lines circuits from Saudi Telecom Company (STC) are connected to the nearest Saudi Aramco SDH point of presence to STC (i.e., Jizan Bulk Plant, SSSP, etc...). The circuit is then backhauled to the SCADA terminal server at the control room. The implementation of this design configuration is widely used in support of Oil Supply Planning & Scheduling Department (OSPAS), Power Distribution Department (PDD), and some of the pipelines sites.

Page 11 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

OSPAS alone has over 400 point-to-point circuits companywide covering key oil and electric distribution systems offshore and onshore utilizing IT SDH transport network. Some of these connections are backhauled to Dhahran Core areas providing a centralized SCADA management system. The primary user of this architecture is OSPAS, where they monitor and control oil and gas flow from different wells and GOSPs. Also, electric power flow from SCECO to Saudi Aramco substations and distribution lines within Saudi Aramco facilities are managed by PDD through this design configuration. Others are based on a distributed SCADA architecture where the remote sites are connected to a local control room site within the same serving area. An example of this is offshore to onshore SCADA connections. 3.2

SCADA Over Dedicated SDH Network It must be noted in this document, the term Optical Transport Network (OTN) is used interchangeably with SDH as OTN is a vendor product solution deployed in several Saudi Aramco plants. There are two approaches for using Synchronous Digital Hierarchy (SDH) transport network in carrying SCADA traffic. The first is based on circuit emulation for a point- to-point serial connection using the inherent multiplexing hierarchy of SDH systems. This approach is a proven concept that has been used in the industry for real time and non real time applications. Figure 3.2.1 depicts the network topology for this design configuration. Ring network topology is the normal approach. For large fields with many scattered oil wells, multiple rings are implemented to provide robust network; while serving dispersed Remote Terminal Units.

Page 12 of 63

Document Responsibility: Process & Control Systems Dept.

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

Issue Date: 22 May 2007

Enterprise PI Server (s)

Firewall

Work Station

1

2

3

4

5

6

7

8

9

m

n i S w

i tc h

V AC A NT SL OT

V A CA N T SL OT

E SX

E SX

A SX -155 RF S 1 MP X

32 UT P 10/ 10 0 Mbp s - E th ern et

2 POR TS OC -3 S MF - A TM

32 UT P 10/ 10 0 Mbp s - E th ern et

2

OMNI SWI T CH 9WX

O

V AC A NT SL OT

Ethernet Switch

V AC A NT SL OT

Plant Network SCADA Server

P

S

1

P

S

2

5 0 0 P

Fast Ethernet

CCR

Enterprise IT Wide Area Network

Operator Control Console

ENG & OPR Console

SCADA PAN Backbone Transmission Office Ring using SDH nodes User

Enterprise Office User

Oil & Gas Fields/Remote GOSP

Ethernet Switch 2

3

4

5

6

7

8

9

O

m

n iS

w

i tc h

0

VACA NTSLO T

ESX

ESX 0

VAC ANTS LOT

MPX

5

32 UTP 10/1 00 M bps - Et hernet

2 PO RTSO C-3 SMF - ATM

32 UTP 10/1 00 M bps - Et hernet

1

ASX 1- 55 RF S

2

O MNI SW ITC H9W X

1

VACA NTSLO T

PAN Access Transmission Ring Using SDH Nodes

VACA NTSLO T

Radio

P

S

1

P

S

2

GOSP: Eng & OPR Console/ Data Historian

P

RTU

WHESD/ESP/PDHM, etc..

Figure 3.2.1: Dedicated Bandwidth over SDH Transmission Network The configuration is typically deployed in a self healing SDH ring with a recovery time of close to 50 milliseconds. This is a powerful capability as long as the SCADA application’s communication time delay is compatible with distance between the RTU and the SCADA Master Sever on the two sides of the ring.

Page 13 of 63

Document Responsibility: Process & Control Systems Dept.

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

Issue Date: 22 May 2007

The second Synchronous Digital Hierarchy (SDH) transmission implementation approach is based on establishing a dedicated bandwidth on the SDH system, providing logical connection on shared SDH transport system. This eliminates the need for serial point to point connection emulation on the SDH and provides higher flexibility in allocating dedicated bandwidth based on actual bandwidth requirements. This option requires the introduction of Ethernet switches at the Remote Terminal Unit (RTU) site. Figure 3.2.2 depicts a conceptual network design using SDH ring based on dedicated bandwidth allocating.

Oil Well Work

PDHM

ESP MOV SWC= 19.4Kbps/Buffer = 4 Mbps CMS=2.4Kbps/0 CP=2.4Kbps/ 2M Buffer

RTU=409Kbps/ 8M Buffer

MPFM=38.4kbps/ 4M Buffer Phone Line=64kbps/ 0 Buffer Ethernet Access Switch

SDH Load= Number of Oil Well Work Areas Interface Capacity Max= 10Mbps to 1 GE Single Mode Fiber Distant 2KM to >10 KM Ethernet Aggregator Switch

SDH

Transmission Network SDH or OTN Ring

SDH

SDH Access Node / Ethernet Aggregator Effective Bandwidth Capacity = Dedicated Bandwidth Capacity Provisioned on the SDH Ring Ranges from 0 to 100 Mbps

SDH

CCR Room

Ethernet Switch Server

Figure 3.2.2: End-to-End Network Topology for a Typical Oil Well Work Site

Page 14 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

Access traffic, SCADA and others, is aggregated to each serving SDH transport node and then logically mapped to the SDH node hosting the SCADA servers at the control room. The term OTN (Optical Transport Network) is used interchangeably with SDH as OTN is a vendor product solution for SDH deployed in several Saudi Aramco plants. There are over 600 OTN nodes serving well heads, RTU’s and others in different Saudi Aramco facilities (e.g., Qatif, Haradh, etc.). In this example, the peak load for each RTU is 516Kbps. This is the composite traffic resident at the oil well work area automation systems: RTU, MPFM, SWC, CMS, CP, Voice, etc. All of these services are connected to an Ethernet switch; an Access switch. Each access switch with a Giga Ethernet interface is connected to another Ethernet switch at the SDH node; an Aggregator switch. The Aggregator switch is then connected to the SDH network using Fast Ethernet connection. A dedicated bandwidth channel is provisioned from the SDH node serving the Aggregator switch to the SDH node located at the CCR room. Therefore, the effective bandwidth allocation from the Aggregator switch to the CCR room is the dedicated bandwidth provisioned between the SDH node severing multiple oil well working areas and the SDH node at the CCR room. Careful considerations shall be taken when designing such a network from a bandwidth and redundancy perspectives. Sections 5.3 will outline engineering guidelines when design this type of network topology. 3.3

SCADA Over IT WAN Packet Network The IT Wide Area Network (WAN) uses Asynchronous Transport Mode (ATM) cell switching technology as a backbone for Enterprise applications. There are some existing scenarios where the ATM backbone is used to support Process Automation application (i.e., SCADA) for linking remote sites to a central site. This is design was implemented for East/West Pipelines and Shaybah’s interGOSPs connectivity. The SCADA traffic is interfaced via IP over Ethernet to IP over ATM WAN node in the local site (e.g., Pump station). The traffic is then routed to the Central SCADA server at the Control Room (CR) (e.g., Yanbu Crude Oil Terminal –YCOT). The process of carting the SCADA from the local site to the central site is supported by the IT ATM network based on best effort utilizing Local Area Network (LAN) ATM emulation capability. Both local SCADA and central sites are considered virtually on the same LAN using shared infrastructure.

Page 15 of 63

Document Responsibility: Process & Control Systems Dept.

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

Issue Date: 22 May 2007

Each pump station is managed locally by its local SCADA Operation group. However, the YCOT SCADA centers can control the sectional valves in between pump stations and provide 2nd level control support for the local SCADA operation support groups. P

P

Figure 3.3 depicts a conceptual network design using IP over ATM network.

YCOT-Yanbu PI Servers Router

Ethernet Switch

IP Router Network

SD SD

SY ST EM S

A SX - 10 0 0

SY ST EM S

A S X - 10 0 0

D

D

D

B

D

C

B

A

B

C

A

SER

RE SE T

E TH

TX C RX L

N E XT S ELE CT

RX L

NE X T S EL ECT

C

TX C

E TH

A

RE S E T

S ER

C

A

C

A

A

A

A

C

C

C

ATM Switches

P WR

P WR B

D

D

D

D

B

B

B

B

IT ATM Backbone Network

Fiber Optics with SDH Nodes

IT SDH Transmission Network

Operator Control Console

Fiber Optics

1

Pump Station #1

2

3

4

5

6

7

8

9

2

3

4

5

6

7

8

m

n iS w

itc h

OM NI SWI TCH W 9 X

O

1

9

m

n iS w

itc h

MPX

ESX

ESX

32 UTP 01 / 100 Mbps - Et hern et

2 POR TSOC- 3 SM F - ATM

32 UTP 01 / 100 Mbps - Et hern et

VACAN T SLOT

VAC ANT SLOT

1 0

VACAN T SLOT

ESX

ESX

ASX- 155 RFS 1

2 POR TSOC- 3 SM F - ATM

32 UTP 01 / 100 Mbps - Et hern et

32 UTP 01 / 100 Mbps - Et hern et

MPX

Digital Transmission

0

Local Pump Station SCADA Network

VACAN T SLOT

VAC ANT SLOT

VACAN T SLOT

VACAN T SLOT

5

5

VACAN T SLOT

2

ASX- 155 RFS

2

OM NI SWI TCH W 9 X

O

P

S

1

P

S

2

0

0

P

S

1

P

S

2

P

P

Ethernet Switch with ATM Uplink

Pump Stations #2 thru 11 Ethernet Switch with ATM Uplink Digital Transmission

Figure 3.3: SCADA over IP using ATM Transport Network

Page 16 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

3.4

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

SCADA Over Radio Network SCADA over Radio based on UHF band, 460MHZ, is used to establish point to point network connectivity between the RTU’s and the SCADA host. This technology approach was implemented in Zuluf, Marjan, and Safaniya offshore, Haradh III and several other remote sites. Radio RTU’s are depicted into the previous figures 3.2.2 and 3.3. One of the key disadvantages of Radio is the shortage of available frequencies in the VHF/UHF (Very High Frequency / Ultra High Frequency) bands. Besides, the disadvantage of the Radio frequency is that the transmission distant is limited to a line of sight path between antennas. Radio is subject to atmospheric attenuation, distortion, and interference.

4

Industry Trends There are two major organizations working on guidelines for plant and enterprise interconnectivity: Purdue Enterprise Reference Architecture (PERA) and the Instrumentation, Systems and Automation Society (ISA). Both organizations strive to provide a fundamental basis for good design and operational procedures; though their approaches are from different perspectives. 4.1

Purdue Enterprise Reference Architecture (PERA) The PERA group looks at system / plant / infrastructure from a cradle to grave approach ensuring that the full life cycle is identified and implemented in the most efficient manner. PERA model defines the need to separate critical plant functions such as plant communications and control away from the enterprise. In addition to this logical and physical separation, PERA provides a conceptual (design philosophy) means of interconnecting the enterprise and plant communications at locations conducive operational requirements while maintaining security. The Physical System Architectures are best designed using certain rules. These rules are based on assessment of the 4R's of system design, specifically: Response; Reliability; Repairability; and Resolution. Each industry is likely to have different functional requirements, as well as different 4R's, which will result in different system architectures. Hence, it is left to the user of this model to define their network outlook based on the above criteria.

Page 17 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

4.2

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

Instrumentation, Systems and Automation (ISA) Society The ISA’s assessment is based on many industry experts that have long contribution to the success of ISA standards development activities. ISA has an extensive history of collaboration with the following organizations for securing control systems communications and forming best practices guidelines: US National Laboratories, the NIST Process Control Security Requirements forum, The Chemical industry Data Exchange, the North American Electric reliability Council and various others in order to stay ahead of the growing threats. The two main ISA standards of relevance to this assessment are: ●

ISA IS-95 Enterprise-Control System Integration

●

ISA IS-99 Security Technologies for Manufacturing and Control Systems

ISA research generated numerous documents and guidelines discussing the benefits and concerns with the legacy designs. Five major themes were stressed in most of the material found during the research and they are as follows: 1.

Process Control System Communication Network Security: Historically process control systems (PCS) have been essentially proprietary and isolated. Over the past decade and with the rapid advances in Information Technology, these systems have become more open and powerful. DCS systems are increasingly based on standard technologies and communications protocols. Most DCS systems in Saudi Aramco are now connected to the local area network (LAN) either directly or via data historian or advanced process control applications. Hence, process control system communication network security becomes vital to overall business operation. This network security theme is inline with the current Saudi Aramco practices. The practice in Saudi Aramco is to install a Firewall between the Corporate and the Process Automation network; refer to SAES-Z-010 for more details.

2.

Network and Communication System Attacks: Network and communication system shall adopt a plan, design and implement a system that ensures a high level of separation or implement extensive firewall/ DMZ setups to prevent an authorized access to the control network. This theme is also inline with Saudi Aramco; refer to SAES-Z-010 for more details.

Page 18 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

4.3

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

3.

Cyber Security: System and network attack analysis has shown that the current threats are coming from outside hackers trying to break into systems, just for the sake of doing it or for malicious intent. Numerous incidents reports show electrical power plants shutting down or experience temporary loss of control. There have been cases of nuclear power generation stations also experiencing such issues. This over emphasize the need for designing secure systems and keep them updated to minimize the security venerability. This theme is also inline with Saudi Aramco; refer to SAES-Z-010 for more details.

4.

Control System Operation and Availability: The primary requirement for the plant control system is to stay operational and in control. Systems must be designed and implement with that point in mind. If a shared communications system is employed, all facility and cost savings will vanish if a control system / plant communications becomes compromised either by a failure due to poor design or exposed vulnerability due to lack of isolation / security.

5.

Communication Infrastructure Weaknesses: Primary concerns fall into two main categories; communication system redundancy (hardware, fiber, copper, etc.), and communication systems operational control. As with any critical communication connection, all possible single points of failure must be eliminated by providing infrastructure redundancy and or alternate paths to ensure operational status. The other point of concern is control/ownership of communication infrastructure that your critical plant controls links are connected through.

Industry Trends Summary Here are the key recommendations of the industry trends. These recommendations were evaluated as part of our final analysis and recommended network architecture for remote Process Automation applications. ●

The plant control system with its dedicated, isolated, private control LAN / communications provides the best way to ensure security and control. This is the practice within Saudi Aramco as outlined in SAES-Z-010.

Page 19 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

●

In today’s demanding business environment, it has become common to utilize a tightly controlled firewall / multi-zone DMZ secured communication connection that allows the plant control system to operate securely while providing a safe means of sharing required plant information with the enterprise proponents. This is the practice within Saudi Aramco as outlined in SAES-Z-010.

●

Shared communications infrastructure is used to transport the remote plant control information and mainline enterprise (voice, data and video) communications as long as designs provide congestion free, accurate, priority QoS based delivery of plant control data in a consistent manner. Utilize separate communications systems if the facilities and funds are available or tightly control and secure a shared communications system to handle both plant process controls and enterprise requirements. This will be part of our recommendation since fiber optics and communication infrastructure is invested in the new fields (i.e.,Khurais, Khursaniyah, etc.).

●

Plan to eliminate or minimize all single points of failure and if at all possible procure, install and control your own communication infrastructure. This will be part of our recommendation since fiber optics and communication infrastructure is invested in the new fields (i.e.,Khurias, Khursaniyah, etc.).

Page 20 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

5

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

Future I-Field/SCADA/Remote Sites Network Architecture An evaluation criterion was developed based on different performance issues outlined in section 5.1. The criteria a long with the impact was mapped and tabulated in section 5.2. A recommendation is provided based on this assessment in section 5.3. 5.1

Existing SCADA Performance & Reliability Issues The table below summarizes the different performance issues for the SCADA systems in different areas within Saudi Aramco. The most common performance issues are enumerated below: Network Design

Performance issues

Dedicated Lines

Low speed/Daisy chained RTUs. Failure of one RTU will impact subsequent RTU’s

Primary: IP/ATM/SDH Backup: IP/ATM/Radio

Performance issues when failover from SDH to the back up radio links. Also, delay is noticeable when other bandwidth demanding applications (e.g., Video streaming) are pushed through the network on the E/W link Route

Dedicated Bandwidth/OTN

System freeze due to sever failover from primary to secondary. This issue is attributed to the point to point connection switch over at the Terminal server located in the control room. Self healing ring is not working when an intermediate transport node fails. This is attributed to the node technology specs where the node uses a shared CPU for both primary and backup links.

Dedicated lines over SDH

Synchronization and network timing issue in the transport communication network. Fiber cuts due to lack of coordination between plant project groups and plant operation. Also, fiber cables wiring are environmentally prone to attacks by rats.

Daisy chained Design

A failure of one RTU impact the performance of some RTU in the daisy chain link. A failure of one of the intermediate instruments impacts the subsequent instruments that are daisy chained on the same link.

Self healing transport network ring architecture

A node failure does not properly trigger the self healing ring capability. So, all nodes that is below the failed node become out of service.

System Servers: Primary to Backup

Intermittent access failure between the application servers (Primary and back up) and the Network Attached Storage (NAS). In addition, frequent failures of the primary and the back up servers. Intermittent failure in the switching between the Primary and the back up server.

Table 5.1: Existing Process Automation Performance Issues

Page 21 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

5.2

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

Analysis & Results There are different design options that can be used to support Remote Process Automation Application traffic. These are: 1.

Process Automation traffic over SDH Transport network using dedicated bandwidth or serial point-to-point circuit emulation utilizing shared infrastructure. This network design can be based on allocating a dedicated bandwidth for each specified service. For example, a dedicated bandwidth for SCADA, a second for CCTV, etc. The second option is where all of the services are subscribed and queued to one dedicated bandwidth.

2.

Process Automation traffic over IP/ATM based on using ATM best effort LAN emulation capability.

3.

Process Automation traffic has a dedicated fiber optics strand where SCADA will have its own SDH/OTN network.

4.

Process Automation traffic utilizing dedicated Giga Ethernet over dedicated fiber optics infrastructure (dedicated strands).

Five (5) performance criteria were used in a comparative analysis format for the stated above design options. These are: Traffic impacts, Security vulnerabilities, Cost, Support & Management operation, and System availability. Each category for each network design was qualitatively mapped to either a Yes (for yes there is a negative impact) or a No (there are no negative impacts). The degree of impact was not reflected as this is table was used to qualitatively screen the most positive design option. Table 5.2 outlines the results.

Page 22 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

Network Design

SCADA over SDH using Shared Infrastructure

Option # 1

Option #2

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

SCADA Over ATM Shared Infrastructure Option #3

Dedicated over Dedicated Fiber Strands Option# 4

Option # 5

Single SDH Node

Dual SDH Nodes

SCADA over GE Ethernet Dedicated Strands Option # 6

Dedicated Bandwidth using Shared Infrastructure

Subscription over shared Infrastructure

Traffic Impacts

No (1)

Yes (7)

Yes (13)

No (19)

No (25)

No (31)

Security

Yes (2)

Yes (8)

Yes (14)

No (20)

No (26)

No (32)

Cost Support & Management System Availability

No (3)

Yes (9)

No (15)

Yes (21)

Yes (4)

Yes (10)

Yes (16)

Yes (5)

Yes (11)

2 No's (6)

0 No's (12)

Impact Criteria

Results (# of No’s)

Yes (22)

Yes ++ (27) No (28)

No (34)

Yes (17)

No (23)

No (29)

No (35)

1 No (18)

3 No's (24)

4 No's, but very costly (30)

5 No's (36)

No(33)

+ = costly ++ = Very Costly No: Signify no impacts; positive. Yes: Signify an impact and a concern; negative. (#): Note number with details in Appendix 1.

Table 5.2: Comparative Analysis- Refer to Appendix 1 for details The following is a summary of the alternatives that would provide sufficient reliability at an optimal cost. These are: 1.

Dedicated bandwidth, option #4; using dedicated fiber optics stands based on single node SDH ring architecture.

2.

Industrial Ethernet over dedicated fiber optics strands, option #6. This option eliminates the need for an SDH network layer thus improve cost and reduce network traffic encapsulations delays. This option is being trial tested by Saudi Aramco Project Management at this time.

3.

Option #1 will be suitable for remote process automation application where dedicated infrastructure does not exist (e.g., East/West Pipelines, Water Injection Plants); since the company has already invested in a self healing SDH network.

Page 23 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

Implementing this solution for East/West pipelines for example, will substantiate the existing ATM performance issues and position the implementation of an efficient secure solution: implementing one (1) firewall at the Yanbu YCOT CCR Room rather than implementing a Firewall at each Pump station. The following options are not recommended due to either manageability, availability, security concerns or they are at a higher cost. These are:

5.3

1.

Option 3 (Utilizing ATM over SDH) is currently used in East/West Pipelines SCADA Network and will be replaced by option #1; upon BI10-00179 implementation.

2.

Option 2 (Subscription over shared) requires higher level of switching intelligence to support necessary quality of service settings. The complexity in managing and associated cost does not warrant this option.

3.

Option 5 (Dual SDH nodes) provides higher level of redundancy but economically unjustifiable.

Remote Process Automation Application Network Architecture The different network architectures outlined in this document and associated analysis has lead to two acceptable network technology and topology designs for future Process Automation networks. These are: 1.

Industrial Ethernet using dedicated dark fiber in a self healing ring topology.

2.

For existing implementations, utilize dedicated bandwidth based on Synchronous Digital Hierarchy (SDH) transport network self healing ring architecture supported by a dedicated fiber optics strands.

These two network toloplogies implemenation shall be based on the following general guidelines: •

A standalone fiber optics cable shall be utilized to connect remote sites (i.e., well sites) to a centralized Process Automation Network (PAN). Alternatively, Radio links, or Broadband wireless network solutions could be used to connect dispersed remote sites where fiber optics cable prove economically not feasible.

Page 24 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

•

Self healing ring: is a ring topology based on a uni/bidirectional links between a set of stations providing optimum protection against fiber cable cut and equipment failure.

•

Utilize point to point ring system for scattered sites and/or where a true closed loop physical ring topology is economically not justifiable.

•

A PAN comprising of multiple scattered (PANs), shall interface with the Corporate Network via a centralized firewall(s). These scattered PANs shall be connected together to form a unified PAN utilizing the corporate SDH transmission infrastructure with a dedicated bandwidth of 10 Mbps. Alternatively, dark fiber can be utilized if available.

•

SCADA over IP/ATM is a unique implementation for East/West Pipelines. It was recommended to PMT to move SCADA traffic from ATM and have it directly transported over a dedicated SDH transport network to fulfill the physical separation requirements as per SAES-Z010. This would be part of the ongoing BI (BI-10-00179) that will replace the existing Microwave system to SDH. Any future implementation similar to the East/West Pipelines such as Water Injection Plants, scattered GOSPs will be based on either one of the two design options stated above. This is mainly because remote GOSPs or WIPs operation is tightly coupled with centralized process control systems.

Detail design requirements for these two wireleine design options are reflected in section 5.3.1. These two design options will provide the required physical and logical separation between Plant Automation Networks (PAN) and other traffic. The outcomes of each recommendation will be mapped to the appropriate Process Automation Standard to be included as part of the next revision. In the interim this report will serve as the guide for any future SCADA, I-Field, and remote site network connectivity until all the standards are updated with these recommendations. 5.3.1

Future Process Automation (SCADA, I-Field, Remote sites) Network Design Architecture The architecture for I-Field, SCADA application, and Remote sites Process Automation Application communication network shall comply with the following guidelines: Page 25 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

1.

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

The remote Process Automation applications network architecture shall be based on either one of the following configurations: i.

Industrial Ethernet using dedicated dark fiber in a self healing ring topology for grass root projects such ,Khurias, Manifa, etc, Figure 5.3.1.1 depicts the network topology.

ii.

For existing implementations, utilize Synchronous Digital Hierarchy (SDH) dedicated bandwidth based on self healing ring architecture. Figure 5.3.1.2 depicts the network topology.

iii.

iv.

Radio links, or Broadband wireless network solution could be used to connect dispersed remote sites where fiber optics cable prove economically not feasible. Number of nodes in the Ring topology shall be carefully calculated from a time delay, bandwidth utilization, and ring switch over perspective so that application performance is not hindered.

v. Self healing ring topology is based on optical ring with physical route diversity that provides protection against fiber cable cut and equipment failures. Also, the optical ring can be a linear system without a physical routed diversity. This system would protect against equipment failure for sites that are limited/scattered or where a true physically ring topology is not economically justifiable. vi.

Industrial Ethernet will be used as a migration path for the existing Optical Transport Node (OTN). Expansion for existing OTN fields’ networks are recommended to be supported by Industrial Ethernet; thus eliminating the need for communication shelters and the associated cooling requirements.

Page 26 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

2. The network design shall provide physical and logical separation between Plant Automation Networks and Corporate Network. However, logical separation utilizing the company’s provided transmission network (i.e., dedicated SDH bandwidth) is permitted for scattered and remote sites with a minimum SDH bandwidth of 10 Mbps. 3. Real-time control communications must be supported on plant owned and dedicated infrastructure. Supervisory control and remote engineering communications can utilize the company’s provided transmission network (i.e., dedicated SDH bandwidth, or dark fiber). However, a prior approval from P&CSD manager for SCADA applications that are directly linked to the Emergency Shutdown (ESD) systems. 4. Monitoring plants applications from the corporate network shall be allowed via only proxy servers. 5. Logical bandwidth separation between each remote site and the Computer Control Room shall be achieved by provisioning dedicated bandwidth (i.e., trunk). The following shall be considered when supporting this network topology model: i. Logical separation (i.e., dedicated SDH link) must be utilized to separate supervisory control and data acquisition traffic from other traffic such as voice, CCTV, etc. Refer to Appendix B for Bandwidth Calculation of this report. 6. Industrial Ethernet over dark fiber network design ring topology depicted in figure 5.3.1.1, is the direction for future remote Process Automation application network deployment as stated previosuly. This should eliminate the need for an SDH network layer; thus significantly minimize cost and network traffic encapsulations delays. This network design shall adhere to the following: i.

Hardened industrial type communication equipment shall be used for all outdoor Process Automation networks. Industrial switches have higher availability, require minimal software revision upgrades, and have longer life cycle. Page 27 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

Moreover, these switches can interface directly to Ethernet enabled RTUs, PLCs, instruments, and services stations. ii.

Industrial Ethernet Access switches with 1 Gbps uplinks shall be installed at the remote sites in a ring network topology connecting to the backbone Giga Ethernet switches.

iii. Industrial Ethernet Access switches shall have VLAN capabilities to support traffic segregation. iv. Industrial Ethernet Access interface ports at the remote site shall be physically and logically interoperable with the Remote Terminal Unit and I-Field Surface Units (IFSU; i.e., PDHM, MPFM, etc) instrumentations. This includes but not limited to port auto-sensing, Duplex/simplex, cable pin-out automatic configuration, etc. v.

Ethernet backbone switches shall be redundant and provide minimum speed of 1 Gbps utilizing ring network topology architecture.

vi. Ethernet backbone switches shall be layer 3 non blocking switches; (carrier class core switches). vii. Ethernet backbone switches shall perform the routing functionality for whole network. viii. The uplink switch-over for for both access and backbone Ethernet switches shall not exceed applications maximum time out parameters. ix. All TCP/IP addresses shall be obtained from Saudi Aramco network management group. x. Provide integrated Domain Name Service (DNS) and Windows Intergated Name Service (WINS) to address the name IP address resolution. xi. Tight integration (e.g., using OPC Tunneling) between Network Management System and the Process Automation application (e.g., SCADA) is recommended for the ease of overall PAN network manageability.

Page 28 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

Enterprise PI Server Enterprise PI User

IT Wide Area Network

IT Wide Area Network Firewall

MIS User

PI Server (etc..)

CCR Ethernet

Redundant GE Ethernet Switches

10

10 Oil & Gas Fields /Remote GOSP

Remote Sites Ring Size Dependent on # of RTU’s 10/100/1 GGE

Operator Control Console

10 10 Giga Ethernet Switches

Giga Ethernet Switches 10/100/10G

ENG & OPR Console

Remote GOSP

Host Radio

RTU=1, thru n

RTU=1, thru n

Remote Radio

Figure 5.3.1.1: Future Network - Giga Ethernet over Dark Fiber; Eliminating the Need for SDH/OTN

Page 29 of 63

Document Responsibility: Process & Control Systems Dept.

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

Issue Date: 22 May 2007

Enterprise PI Server Firewall 1

2

3

4

5

6

7

8

9

m n i S w

it c h

VA CA N T SL OT

ES X

ES X

V AC A NT SL OT

1 MP X

32 UT P 10/ 10 0 Mb ps - E h t erne t

32 UT P 10/ 10 0 Mb ps - E h t erne t

2 P OR TS OC -3 SMF - AT M

V AC A NT SL OT

A SX 1- 55 R FS

2

OMN I S WIT C H 9WX

O

5 0

V AC A NT SL OT

Process Automation Data Acquestion Network Hostorian Server

P

S

1

P

S

2

0 P

Ethernet Switch

MIS Work Station CCR

IT Wide Area Network

Giga Ethernet Operator Control Console

ENG & OPR Console

Enterprise PI Server

Fiber Optics Backbone Transmission Ring

Oil & Gas Fields

Enterprise Office User

Ethernet Switch Wireless Fiber optics Transmission Access Ring

REMOTE GOSP

WHESD/ESP/PDHM, etc..

Figure 5.3.1.2: Future Networks-Dedicated Bandwidth over SDH Transmission Network Using Dedicated Fiber Strands

Page 30 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

8. Data acquisition frequency (e.g., polling cycle, deadband) shall be identeifed by the proponnet in the Funcational Specification Dcoment for each installation. Comments: For I-Fields pressure transient analysis application data acquisition frequency rate shall be set to 1 second. If the communication protocol is based on report by exception, then the deadband shall be set to 0.01 for the Permanent Down Hole System Pressure and Temperature readings. 9. Redundant RTU and network interface requirements (Single vs. Dual) to the Process Automation servers will be determined at the early design stage for each application. For example, high pressure gas wells may require complete end-to-end system, RTU, and network redundancy. 10. Each RTU shall have its own dedicated network connectivity. A failure of one RTU shall not impact the data acquisition from other RTUs. It shall also be equipped with local storage capacity that can support local data acquisition for 96 hours. 11. The design criteria for the installation of electrical instrumentations are covered in SAES-J-902. For I-Field Surface Units (IFSU; i.e., PDHM, MPFM, etc) network connectivity shall adhere to the following guidelines: i.

Communication interface between IFSU and the RTU/Ethernet switch shall be interoperable. Ethernet is preferred.

ii.

The connectivity design between IFSU and the RTU/Etherenet switch shall be based on a dedicated point-topoint connection. Daisy chained IFSUs wiring design is not permitted to minimize failure impacts to the overall business operation.

iii.

IFSUs shall be able to support standard Ethernet and/ or Serial interfaces. Ethernet is preferred.

iv. IFSU shall be equipped with local storage capacity that can support local data acquisition for 96 hours.

Page 31 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

12. Process Automation application servers shall be deployed as follow: i. SCADA and other critical process applications such as I-Field computer system hardware (e.g., servers) shall adhere to 23SAMSS-20. ii. Non mission critical applications (e.g., Cathodic monitoring, Power System Automation, etc.) shall be supported by single servers. Comments: It is recommended at the design stage for new projects to minimize the number of servers at the Process Automation control room by combining applications on the same set of servers; if feasible. 15. A Business Continuty Plan (BCP) shall be established to remedy possible network design deficiencies if any; but may help reduce the impacts and improve the response time upon a network failure.

5.3.2 Next Steps Following are the next steps to be completed as part of Phase II of this report. 1. P&CSD will evaluate different industrial Ethernet vendor solutions, complete performance testing and test procedures. 2. The outcome of this effort will be used to develop the required material specifications for Industrial Ethernet that would accommodate Saudi Aramco Process Automation requirements for I-Field, SCADA, and Remote Site network connectivity.

Page 32 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

Part II: Process Automation Wireless Access 1

Background This wireless Process Automation outlook was carried out as part of plant networks optimization efforts. The plan is to enable and promote efficient and cost effective and proven industrial wireless technologies for various process related applications such as I-Field, SCADA and other process remote monitoring. Also, this effort would help to streamline the different ongoing Process Automation wireless trials into uniform implementation for future projects. In this effort, we emphasized on the importance of considering and complying with the requirements of critical and time-sensitive process information applications as well as considering wireless reliability, security and interoperability. Moreover, we avoided discussing proprietary wireless solution since it hinders capability, interoperability and flexibility.

2

Introduction Wireless technology is becoming increasingly popular solution for a wide range of applications. After evaluating the technology, many users are convinced of its reliability, satisfied with its performance and are ready to use it for different applications in a large-scale and more complex wireless networks. Wireless LANs (WLANs) differ from regular “WIRED” LANs in the means they use to transport data. WLANs use air instead of a network wired cable as a medium for data transport. During the last few years, WLANs have many applications in warehousing, manufacturing, airports, hotels and health-care. Recently, wireless technology is becoming among the most significant cutting-edge technologies as a viable network connectivity option for Process Automation networks (PAN) as well as instrumentations networking. Different Wireless technologies are benchmarked in this assessment and were mapped to the suitable Process Automation application (s).

3

Wireless for Process Networks In nearly all industrial plants, communication links carry vital information between field devices (instruments, sensors, actuators, etc.) and the process control center. From periodic updates to ongoing process and manufacturing management, reliable and secure data flow is critical operations.

Page 33 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

Control message and status information transferred in industrial settings—actuator position, temperature, or liquid levels, for example—is carried in short “bursts” which require relatively little bandwidth and connection speed. At the other extreme, large file transmission, such as activity logs from a production run, requires moving a lot of data very efficiently. Whatever the specifics of the data being moved, all industrial communications share common critical requirements: Secure, Reliable and Timely data delivery. The following applications have been identified as potential industrial applications for wireless access:

4

1.

Exploration and Producing operation (Wireless-I-Field, data connectivity)

2.

Remote Data Acquisition (temperature, pressure, flow, level)

3.

Energy management & sub-metering

4.

Process monitoring wellhead,…etc.)

5.

Waste water management

and

control

(Condition

based,

Power,

Vibration,

Industrial Wireless vs. Non-Industrial Wireless The Process Automation applications are considered mission critical applications and conventional wireless solution won’t be able to serve these applications in an active environment. Hence, industrial wireless is considered to be the alternate solution that would ensure security, reliably and real time delivery for the Process Automation data. The Industrial Wireless solution should meet the required process networks reliability, availability and security as shown in Table 4.1.

Page 34 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

No

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

Industrial requirements for Industrial Wireless solution

Office / consumer requirements for non-industrial wireless

1

Reservation of bandwidth for selected critical clients (e.g. PLC), Uguaranteed accessU to radio channel

Radio channel is a shared medium; everybody can access it (best efforts)

2

The communication cycles are predictable for UdeterministicU data traffic

The radio traffic is prioritized, but cannot be predicted

3

Redundant radio communication on two separate frequency bands for increased availability (URedundancyU)

Usually one radio channel

4

Metal casing, vibration-resistant connectors and cables

Low-cost plastic casing

5

IP 65 degree of UprotectionU, +60°C and moistureresistant for harsh industrial and outdoor use

For home and office use

6

IU ndustrial approvalsU: ATEX (EX area), Area Classification, EMC, UL, FM

No demand for increased requirements (CE, shock protection)

7

UFlexibleU use through Batteries, 18 - 57 VDC, 100 – 240 VAC, Power-over-Ethernet

Voltage supply 100 VAC – 240 VAC, PoE

8

Ethernet, antennae cables, casing Uresistant to chemicalsU, halogen-free, silicon-free, flame-retardant

No demand for approvals with increased requirements

Table 4.1: Industrial Wireless vs. Non-Industrial Wireless

5

Industrial Wireless Classifications Industrial wireless technologies can be classified into four main categories based on their coverage area as shown in Figure 5.1. These are: 1.

Wireless Personal Area Network (WPAN) – few meters

2.

Wireless Local Area Network (WLAN) – few 100’s meters up to few Km

3.

Wireless Metropolitan Area Networks (WMAN) – few Km

4.

Wireless Wide Area Networks (WWAN) – 10’s up to 100’s of Km

Page 35 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

20km+ 2008/9

MWBA 802.20

2010

2006

10km

3.5G

3G

2.5G

WWAN

4G

2006

5km

Distance

M WiMAX

FWBA

802.16e

100m

WLAN

WLAN 1999

802.11g

WLAN

2003

WLAN 802.11a

2006

802.11n

802.11b

ZIGBEE

WMAN

802.16

2km

WLAN

1999

2004

WPAN

802.15.4

10m

802.15.4a

BlueTooth

BlueTooth

802.15.1/1a

2.0

2005

1Mb/s

UWB (802.15.3a?)

2005

50Mb/s

2005

100Mb/s

Bandwidth Figure 5.0: Industrial Wireless Classifications Bandwidth vs. Distance 5.1

Wireless Personal Area Networks (WPAN) The Wireless Personal Area Network (WPAN) technology uses a short-range radio link that has been optimized for power conscious, battery operated, small size, lightweight devices. There are two dominant WPAN technologies namely Bluetooth (IEEE 802.15) and Zigbee technologies. The Bluetooth Technology operates in the 2.4 GHz band and used in several computing devices such as notebook computers, phones, PDAs to exchange information. Bluetooth is not considered as potential industrial wireless solution for Process Automation due to the excessive latency and limited networking capabilities due to synchronization handshaking process. U

U

Zigbee Technology addresses the unique needs of low-cost, low-power, wireless instrumentation/sensor networks for remote monitoring & control and building automation network applications in the industrial plants at short distances (few 10s of meters). The zigbee standard was ratified in 2006.. U

Page 36 of 63

Document Responsibility: Process & Control Systems Dept. Issue Date: 22 May 2007

SAER-6114 Process Automation (I-Field/SCADA/Remote Sites) Communication Network Architecture

The general specs of Zigbee along with IEEE 802.15.4 solution could be summarized as follows: ●

Dual PHY (2.4GHz ISM and 868/915 MHz)

●

Data rates of 250 kbps (@2.4 GHz), 40 kbps (@ 915 MHz), and 20 kbps (@868 MHz)

●

Optimized for low duty-cycle applications (