Mesh Control

B0700AZ REV L

I/A I/A Seri Series es®System The MESH Control Network Architecture Guide October 15, 2009

Invensys, Foxboro, Foxboro, FoxView FoxView,, I/A Series, and the IPS logo are trademarks of Invensys Invensys plc, its subsidiaries and affiliates.  All other brand names may be trademarks of their respective respective owners. Copyright Copyright 2004-200 2004-2009 9 Invensys Invensys Systems, Systems, Inc.  All rights reserved

SOFTWARE SOFTWARE LICENSE AND COPYRIGHT INFORMATION INFORMATION Before using using the Invensys Invensys Systems, Inc. supplied software supported supported by this documentation, documentation, you should read and understand the following information concerning concerning copyrighted software. 1. The license provisions provisions in the the software license for your system govern your obligations obligations and usage rights to the software described in this documentation. If any portion of those license provisions provisions is violated, Invensys Invensys Systems, Inc. will no longer provide provide you  with support services and assumes assumes no further responsibilities responsibilities for your your system or its operation. 2. All softwa software re issued issued by by Invensy Invensyss Systems, Systems, Inc. and copies copies of of the softwa software re that that you are specifically permitted to make, are protected in accordance with Federal copyright laws. It is illegal to make copies of any software media provided to you by Invensys Systems, Inc. for any purpose purpose other than those purposes mentioned in the software license.

Contents  Figures........................... Figures........................................ .......................... .......................... .......................... .......................... .......................... .......................... ......................... .............. vii Tables......................... Tables...................................... .......................... .......................... ......................... ......................... .......................... .......................... .......................... .................. ..... ix  Preface........................ Preface..................................... .......................... ........................... ........................... .......................... .......................... .......................... ......................... ................ .... xi Purpose ........................................................ ............................................................... ............. xi  Audience ............................................................ ................................................................ ...... xi Revision Information ....................................................... ........................................................ xi Reference Documents ............................................................. ................................................ xii Terms and Definitions .................................................... ....................................................... xiii 1. Introduction...................... Introduction................................... .......................... .......................... .......................... .......................... .......................... .......................... .................. ..... 1 Overview of The MESH Control Network Architecture ........................................................... Switched Ethernet Characteristics ........................................................ ................................ The MESH Control Network Features ................................... ............................................. Standard Configuration Features ............................................................ ......................... Security Enhanced Configuration Features .......................................................... ............ Loop Detection Detection Policy Policy (LDP) (LDP) Deployed Deployed on The MESH MESH Control Network Network ..................... Virtual Local Local Area Networks (VLANs) (VLANs) on The MESH Control Network Network ........................ The MESH Control Network .............................................................................................. The MESH Control Network Topologies .......................................................... .............

1 1 2 2 2 3 3 4 4

The MESH Control Network .......................................................... ....................................... Network Example ....................................................... ....................................................... The MESH Control Network Specifications ...................................................................... The MESH Control Network Workstations ...................................................................... ............................ ..........................................

12 12 15 15

The MESH Control Network Ethernet Switches ................................................. ................... 16  Advantages of Invensys-Supplied Invensys-Supplied Switches Switches .......................................................................... 16 Control Network Cabling ................................................................ ....................................... Category 5 Cabling .......................................................... .................................................. Fiber Optic Cabling ..................................................... ...................................................... Single Mode Cable ............................................................... ......................................... Multimode Cable ................................................................. .........................................

17 18 18 19 19

The MESH Control Network Management Software Tool, NetSight® Con Consol solee ...... .......... ....... ...... ....... .... 20 Netsight Policy Manager ......................................................... ........................................... 20 Obtaining Network Management Software ........................................................................ 22 2. Site Site Planning Planning............. .......................... .......................... .......................... ........................... ........................... .......................... .......................... ....................... .......... 23 Site Planning Overview ........................................................... ................................................ 23 Network Considerations ........................................................... .......................................... 23 iii

B0700AZ – Rev L

Contents

Traffic Considerations .................................................... .................................................... Network Loading ......................................................... ...................................................... Equipment Considerations ................................................................. ................................ Hardware Requirements .............................................................. ....................................... Switch Utilization in the Standard Configurations ........................................................ Switch Utilization in the Security Enhanced Configurations ......................................... Firmware Considerations .......................................................... ..........................................

26 26 27 28 28 28 29

The MESH Control Network Design Rules .................................................................. .......... Standard Configuration Design Rules ....................................................... ......................... Non I/A Series Equipment ............................................................................................ Security Enhanced Configuration Design Rules .................................................................  Additional Guidelines for Planning Security Enhanced Configuration ............................... Security Enhanced Configuration Design Considerations ..............................................

29 29 30 30 34 34

The MESH Control Network Design ..................................................................................... I/O Network Design Rules ................................................................... .............................. The MESH Control Network Topologies .......................................................................... Standard Configurations .................................................................. .............................. Security Enhanced Configurations ............................................................ ..................... I/O Network Topology Configurations ................................................................. ........ Switch and Fiber Cable Budget and Loss ............................................................................ Fiber Cable Budget Cable and Loss ..................................................................... .......... Fiber Optic Budgets .............................................................. ........................................ The MESH Control Network Cabling ......................................................... ...................... Switch and Converter Fiber Optic Cabling ...................................................... .............. FCP270, ZCP270, FCM100Et and FCM100E Fiber Signal Cabling ............................ Twisted-Pair Cabling ................................................................... ...................................... Null Hub ............................................................ ..........................................................

36 36 36 37 52 58 59 59 59 59 59 63 66 66

3. Installation (Cabling) ...................................................................................................... 67 Connecting The MESH Control Network Components ......................................................... Fiber Optic Cabling Guidelines .......................................................... ................................ Interconnecting Ethernet Switches .............................................................. ....................... Uplink Port to Uplink Port ....................................... .................................................... RJ-45 Port to RJ-45 Port ...............................................................................................

67 67 67 68 70

Switch Configuration ......................................................... ..................................................... 70 4. Maintenance.................................................................................................................... 71 The MESH Control Network Addresses ................................................................................. 71 General Troubleshooting Guidelines ................................................................. ...................... Characterize the Problem ........................................................... ........................................ Determine Which Devices are Affected ............................................................. ................. Troubleshoot the Affected Devices ............................................................... ......................

72 72 73 73

System Management Displays ...................................................... ...........................................  Accessing SMDH Switch Network Displays ....................................................................... Switched Network Display ...................................................... ........................................... Switch Equipment Change Display ...............................................................................

74 74 76 77

iv

 Contents

B0700AZ – Rev L

Switch Equipment Information Display ........................................................................ Switch Configuration Information Display ..................................................... ............... Switch Domain Display ..................................................................................................... Switch Ports Display ................................................................. ......................................... Switch Port Equipment Change Display ................................................................. ...... Switch Port Equipment Information Display ................................................................

78 80 81 82 83 84

Indicators ................................................... ............................................................. ................ 85 Ethernet Switches ....................................................... ........................................................ 85 Media Converter ....................................................... ......................................................... 85 Fiber Optic Cable Handling and Cleaning ............................................................ .................. Handling Fiber Optic Cable ................................................................ ............................... Cleaning Fiber Optic Cable ................................................................. ............................... Contamination of Fiber Optic Connectors and Sockets ................................................. Contamination Prevention ............................................................. ............................... Contamination Removal .............................................................. ................................. Examples of Fiber Optic Connector Cleaning Products .................................................

85 85 87 87 88 88 89

5. Combining Two or More MESH Control Networks ...................................................... 91 Overview ................................................................. ............................................................ .... Planning Stage ............................................................ ........................................................ Station Addressing ....................................................... ....................................................... Bridge Switch Functionality ................................................................. .............................. Topology Constants ................................................................. ..........................................

91 91 92 92 92

Combining Star Network Topologies ..................................................................................... 93 Combining Star with Tree Network Topology .............................................................. .......... 95 Combining Tree Network Topologies ........................................ ............................................ 96 Combining Ring Network Topologies .................................................................................... 97  Appendix A. COMEX Fault Handling on The MESH Control Network............................ 99 COMEX Layers ........................................................ .......................................................... .... 99 COMEX Applications Layer ......................................................... ..................................... 99  Application Layer Protocol and Timers ................................................................ ......... 99 Transport Layer .......................................................... ...................................................... 100 Transport Layer Protocol and Timers .......................................................................... 102 Network Layer ........................................................... ...................................................... 103 Operation .................................................... ............................................................. ... 103 Logical Link Control Layer ..................................... ......................................................... 103 Transmit Operation ........................................................ ............................................ 103 Receive Operation ......................................................... .............................................. 103 MAC Layer .................................................................. .................................................... 104  Appendix B. The MESH Network Fault Handling ........................................................... 105 LINK ............................................................ .............................................................. .......... 105 “PORT TEST” Packets ........................................................... .............................................. 105 v

B0700AZ – Rev L

Contents

“LLC_PING” Test ............................................................. ................................................... 106 ZCP-FCM Communications ................................................................. ............................... 107 DIAGNOSTIC Information .................................................... ............................................. 108 Index .................................................................................................................................. 111

vi

Figures  1-1. 1-2. 1-3. 1-4. 1-5. 1-6. 1-7. 2-1. 2-2. 2-3. 2-4. 2-5. 2-6. 2-7. 2-8. 2-9. 2-10. 2-11. 2-12. 2-13. 2-14. 2-15. 2-16. 2-17. 2-18. 2-19. 2-20. 2-21. 3-1. 3-2. 3-3. 4-1. 4-2. 4-3. 4-4. 4-5. 4-6. 4-7. 4-8. 4-9. 4-10. 4-11. 4-12. 4-13.

Linear Topology ............................................................................................................ 5 Ring Topology (Standard Configuration Only) ............................................................ 6 Star Topology ............................................................................................................... 8 Inverted Tree Topology .............................................................................................. 10 Modified Inverted Tree Topology ............................................................................... 11 The MESH Control Network (Star Topology Shown) ................................................ 13 The MESH Control Network with an I/O Network ................................................... 14 Large Network (Security Enhanced Configuration) ..................................................... 33 Small Network (Standard Configuration) ................................................................... 38 Medium Network (Ring) (Standard Configuration) .................................................... 39 Medium or Large Network (Star) (Standard Configuration) ....................................... 40 Medium or Large Network (Double Star) (Standard Configuration) .......................... 41 Large Network - All Blades (Standard Configuration) ................................................. 42 Large Network - Inverted Stepped Tier Tree - All Blades (Standard Configuration) .... 44 Large Network - Blades and Low-Cost Switches (Standard Configuration) ................. 45 The MESH Control Network Tiers ............................................................................ 47 Root Switches Connected ........................................................................................... 48 Second Tier Connections ............................................................................................ 49 Third Tier Connections .............................................................................................. 50 Fourth Tier Connections ............................................................................................ 50 Labeling the Switches .................................................................................................. 51 Small Network (Security Enhanced Configuration) .................................................... 52 Star Topology (Security Enhanced Configuration) ...................................................... 53 Double Star Topology (Security Enhanced Configuration) ......................................... 54 Inverted Tree Topology (Security Enhanced Configuration) ....................................... 56 Modified Inverted Tree Topology (Security Enhanced Configuration) ....................... 58 Switch to Switch and Switch to Patch Panel Connections ........................................... 63 FCP270, ZCP270, FCM100Et and FCM100E Signal Cabling .................................. 65 Switch-to-Switch Fiber via Uplink Port ....................................................................... 69 Port-to-Port Connection via Fiber Optic Ports ........................................................... 69 Connecting Switches via RJ-45 Ports .......................................................................... 70 Accessing SMDH Switched Network Displays ............................................................ 75 SMDH Switched Network Display ............................................................................. 76 Switch Equipment Change Display ............................................................................. 77 Switch Equipment Information Display ...................................................................... 78 Switch Configuration Information Display ................................................................. 80 Switch Domain Display .............................................................................................. 81 Switch Ports Display - Typical .................................................................................... 82 Switch Equipment Change Display ............................................................................. 83 Switch Port Equipment Information Display - Typical ............................................... 84 SC Connector, Typical ............................................................................................... 86 Multimode MT-RJ Connector .................................................................................... 87 Multimode Duplex LC Connector .............................................................................. 87 Lint in Fiber Optic LC Socket .................................................................................... 88 vii

B0700AZ – Rev L

5-1. 5-2. 5-3. 5-4. 5-5. 5-6. 5-7. 5-8.

viii

Figures

Combining Two Star Network Topologies Into One Tree Network Topology (Before) .................................................... ............................................................. ...... 93 Combining Two Star Network Topologies Into One Tree Network Topology (After) 93 Combining a Star Network Topology with a Tree Network Topology Into One Tree Network Topology (Before) ........................................................................................ 95 Combining a Star Network Topology with a Tree Network Topology Into One Tree Network Topology (After) .......................................................................................... 95 Combining Two Tree Network Topologies Into One Tree Network Topology (Before) .................................................... ............................................................. ...... 96 Combining Two Tree Network Topologies Into One Tree Network Topology (After) .................................................... ............................................................ ......... 96 Combining Two (4) Tier Tree Network Topologies Into One (4) Tier Tree Network Topology (After) ......................................................................................................... 97 Combining Ring Network Topologies ........................................................................ 98

Tables  1-1. 1-2. 1-3. 1-4. 1-5. 1-6. 1-7. 1-8. 2-1. 2-2. 2-3. 2-4. 2-5. 2-6. 2-7. 2-8. 2-9. 2-10. 2-11. 3-1. 4-1. 4-2. 4-3. 4-4. 4-5. 4-6.  A-1.

The MESH Control Network Specifications ............................................................... 15 NetSight Console Policies ........................................................................................... 20 NetSight Client Policy ................................................................................................ 21 NetSight Policy Bundle ............................................................................................... 21 NetSight Advanced Bundle ......................................................................................... 21 NetSight Single User Policy ........................................................................................ 22 NetSight A-La-Carte Plug-Ins ..................................................................................... 22 NetSight Appliance Policy ........................................................................................... 22 Fiber Cable Power Losses ............................................................................................ 59 Multimode Fiber Cables with LC Connectors ............................................................. 60 Multimode Fiber Cables with MT-RJ to ST Connectors ............................................ 60 Multimode Fiber Cable with LC and SC Connectors ................................................. 61 Copper Cables with RJ-45 Connectors ....................................................................... 61 Single Mode Fiber Optic Cable - Maximum Transmission ......................................... 61 Single Mode Fiber Optic Jumper Cables ..................................................................... 62 Fiber Optic Cables ...................................................................................................... 64 CAT5 Cable - Maximum Transmission Distance ........................................................ 66 Prefabricated CAT5 STP Cables with RJ-45 Connectors ............................................ 66 Null Hub .................................................................................................................... 66 Methods of Connecting Ethernet Switches ................................................................. 68 IP Address Assignments ............................................................................................... 71 Switch Equipment Change Display Actions ................................................................ 77 Switch Equipment Information Display Fields ............................................................ 78 Switch Configuration Information Display Fields ....................................................... 80 Switch Equipment Change Actions ............................................................................. 83 Switch Port Equipment Information Display Fields .................................................... 84 Out of Sequence DT and Ack Packets ........................................................... ........... 101

ix

B0700AZ – Rev L

x

Tables

Preface  Purpose This document provides overview guidelines and requirements for designing, installing, and maintaining The MESH control network. Topics include: ♦ ♦

Site Planning  Installation

Maintenance.  Additional documentation provides the information specific to the hardware for The MESH control network. These books are listed in “Reference Documents” below. ♦

For detailed and specific information on the Ethernet equipment, refer to the documentation supplied by the switch vendor. These documents may also be available on the IPS Global Client Support web site at http://support.ips.invensys.com. The MESH control network documents for I/A Series® systems are available on The MESH Network Configuration Tool CD-ROM (K0173ZU).

Audience This book is intended for use by process engineering, operations, installation, and maintenance personnel. They are expected to have a working knowledge of Ethernet LANs and I/A Series configurations.

Revision Information For this revision of the document (B0700AZ-L), the following changes were made: Chapter 1 “Introduction” ♦

Significantly expanded “Network Example” on page 12.

♦

Updated references to Invensys in “Advantages of Invensys-Supplied Switches” on page 16.

Updated Figure 1-6 “The MESH Control Network (Star Topology Shown)” on page 13. Updated “The MESH Control Network Management Software Tool, NetSight® ♦ Console” on page 20 to reflect the NetSight® products currently offered. Chapter 2 “Site Planning” ♦

♦ ♦ ♦

Minor edits to “Switch Utilization in the Standard Configurations” on page 28.  Added “I/O Network Design Rules” on page 36.  Added “I/O Network Topology Configurations” on page 58.

xi

B0700AZ – Rev L

Preface

Reference Documents The following documents provide additional or related information to the hardware used in The MESH control network:

 A-Series (P0973BH/P0973BJ/P0973BK) Switches, Hardware and Software Configura-  tion Instructions   (B0700CH) The MESH Control Network Hardware Instructions for C-Series Switches ♦ (P0973BL/HA)  (B0700CJ) The MESH Control Network Hardware Instructions for N-Series Switches ♦ (P0973AR/P0973AS/P0972YE)   (B0700CK) V-Series (P0972WP/P0972YC) Switches, Hardware and Software Configuration Instruc-  ♦ tions   (B0700CL) ♦ E7 Chassis and 16-port Fiber (P0972MK/P0972MJ) Switches, Hardware and Software Configuration Instructions (B0700CM) ♦ I-Series (P0973GB) Industrial Switch, Hardware and Software Configuration Instruc-  tions  (B0700CN) Media Converter Installation and Configuration Guide for Control Networks  (B0700CP) ♦ The following documents provide additional or related information to The MESH control network concepts: ♦ The MESH Control Network Operation, and Switch Installation and Configuration Guide   (B0700CA) The MESH Control Network Architecture  (PSS 21H-7C2 B3) ♦ The MESH Control Network Ethernet Equipment  (PSS 21H-7C3 B4) ♦ I/A Series System Definition: A Step-by-Step Procedure  (B0193WQ) ♦ I/A Series Configuration Component (IACC) User’s Guide   (B0400BP). ♦ ♦

xii

 Preface

B0700AZ – Rev L

Terms and Definitions 10Base-T

10 Mb twisted-pair Ethernet

100Base-TX

100 Mb twisted-pair Fast Ethernet

100Base-FX

100 Mb fiber optic Fast Ethernet

1000Base-LX

IEEE 802.3z specification for Gigabit Ethernet over two strands of 50/125 or 62.5/125 micron core MMF or 9/125 micron core SMF fiber cable using long wavelength optical transmission.

1000Base-SX

IEEE 802.3z specification for Gigabit Ethernet over two strands of 50/125 or 62.5/125 micron core MMF fiber cable using short wavelength optical transmission.

1000Base-ZX

IEEE 802.3z specification for Gigabit Ethernet over two strands of 9/125 micron core SMF fiber cable using 1550 nm wavelength optical transmission.

1000Base-T

IEEE 802.3ab specification for Gigabit Ethernet using CAT5 copper Ethernet cable.

 ANSI

American National Standards Institute

 Auto-Negotiation

Signalling method allowing each node to select its optimum operational mode (e.g., speed and duplex mode) based on the capabilities of the node to which it is connected.

Backbone

Another term for bus - refers to the main link that connects network nodes. The term is often used to describe the main network connections composing the network.

Beacon

The packet type and packet on the network upon which the port disabling is enacted.

BPP

Beacon Priority Policy - A role/service that allows for the Beacon packet to have the highest priority when propagating though the network. This ensures the Beacon packet will be transmitted back to the PBQ in a flooded switch.

BootP

Bootstrap Protocol

Bridge Priority Value

The range of priority values used to determine which device is selected as the Spanning Tree root. This value can range from 0- 65535 for bridge priority mode 802.1d (decrement by 1) or from 0-61440 for bridge priority mode 802.1t (decrement by 4096).

CAT5

Category 5 Twisted Pair Cable - such as 10Base-T, 100Base-TX and 1000Base-T.

xiii

B0700AZ – Rev L

xiv

Preface

CBP

(Circuit Breaker Policy) a role/service that disables a port when a Beacon packet is received from an edge switch or from the tier below.

Circuit Breaker

A policy rule that will disable a port that receives an incoming packet of an outgoing Beacon packet “Loop”.

Circuit Breaker PBQ/SBQ (CBPBQ/CBSBQ)

This is used to refer to policy rule that will disable an uplink port that interfaces two root switches that receives an incoming packet of an outgoing Beacon packet “Loop”. This function is a subset of the Circuit Breaker.

CLI

Command Line Interface

Core Switch

Refers to the main body of switches that provide the network with its backbone connections. A core switch can also be considered an “edge switch” in reference to the root; however the outer most edge switches  within the network are normally not considered to be core switches.

CRC

Cyclic Redundancy Check 

CSMA/CD

Carrier Sense Multiple Access/Collision Detection

Data Loop or Loop Path

Refers to a condition where data traverses a redundant path with no termination point.

DCE

Data Communications Equipment (modem)

DSR

Data Set Ready 

DTE

Data Terminal Equipment

DTR

Data Terminal Ready  

Edge Switch

Refers to an outer switch in a network topology that is linked to the primary root or backup root bridge switch directly in one to two tier configurations, and indirectly in three to four tier configurations.

ESD

Electrostatic Discharge

FCS

Frame Check Sequence

Fast Ethernet (FE)

Set of Ethernet standards that carry traffic at the nominal rate of 100 Mbit per second.

FTM

Frame Transfer Matrix  

Full Duplex

Transmission method that allows two network devices to transmit and receive concurrently, effectively doubling the bandwidth of that link.

GBIC

Gigabit Interface Converter

HTTP

Hypertext Transfer Protocol

 Preface

B0700AZ – Rev L

ICMP

Internet Control Message Protocol

IEEE

Institute of Electrical and Electronics Engineers

IEEE 802.3

Defines carrier sense multiple access with collision detection (CSMA/CD) access method and physical layer specifications.

IEEE 802.3ab

Defines a media access method and physical layer specifications for 1000Base-T Gigabit Ethernet.

IEEE 802.3u

Defines a media access method and physical layer specifications for 100Base-TX Fast Ethernet over CAT5 cable.

IEEE 802.3x

Defines Ethernet frame start/stop requests and timers used for paused flow control on full-duplex links.

IEEE 802.3z

Defines a media access method and physical layer specifications for 1000Base Gigabit Ethernet.

IGMP

Internet Group Management Protocol, used to establish host memberships in particular multicast groups on a single network.

IOC

Input/Output Controller (part of the Z-Module Control Processor (ZCP270))

IOM

Input/Output Module

IP

Internet Protocol

LAN

Local Area Network  

LDP

Loop Detection Policy (Refer to The MESH Control Network Operation, and Switch Installation and Configuration Guide (B0700CA) for more information.)

LED

Light Emitting Diode

MAC

Media Access Control

MDI

Media Dependent Interface or Media Device Interface

MIB

Management Information Base

MMF

Multi-mode Fiber cable

NEM

Network Expansion Module

PBQ

Primary Beacon Queryer - The switch with the lowest IP address and with the IGMP “Beacon” enabled.

Policy

A group of rules which a network device uses to make forwarding, blocking or port-disable decisions.

xv

B0700AZ – Rev L

xvi

Preface

RFC

Request for Comment

RMON

Remote Monitoring  

Role

A collection of services

RSTP

Rapid Spanning-Tree Protocol (IEEE 802.1w standard)

Rule Hit

An action when a packet classifier finds the packet.

Rules

Packet classifiers that are used to identify packet types on the network.

RXD

Receive Data  

SBQ

Secondary Beacon Query - The switch with the second lowest IP address and with the IGMP “Beacon” enabled.

Service

A collection of Rules

SFP

Small Form Factor Pluggable (Type of Mini-GBIC)

SMF

Single-mode Fiber cable

SNMP

Simple Network Management Protocol

STP

Spanning-Tree Protocol (IEEE 802.1d standard)

TCP/IP

Transmission Control Protocol/Internet Protocol

TDR

Transient Data Recorder

TFTP

Trivial File Transfer Protocol

TXD

Transmit Data  

UTP

Unshielded Twisted Pair

VLAN

Virtual Local Area Network  

1. Introduction  This chapter provides an introduction to the concepts and equipment used in The MESH control network.

Overview of The MESH Control Network Architecture The MESH control network is a switched Fast Ethernet network based on IEEE 802.3u (Fast Ethernet) and IEEE 802.3z (gigabit Ethernet) standards. The MESH control network consists of a number of Ethernet switches connected in a MESH configuration. The MESH control network configuration allows high availability by providing redundant data paths and eliminating single points of failure caused by component link failures. The flexibility of the architecture allows you to design a network configuration that fits the needs of the control system. Configurations can be as simple as a workstation and controller connected with a single pair of switches, or as complex as a multi-switch, fully meshed control network, communicating at speeds up to 1 gigabit per second. The MESH control network architecture integrates powerful control stations and workstations in a 100 Mb/1 Gb Ethernet network. These control stations, workstations and networks comprise scalable systems for process monitoring, process control and integration with industrial information management systems. High speed, coupled with redundancy and peer-to-peer characteristics, provides high performance and superior security. Station interfaces to redundant Ethernet switches ensure secure communications between the stations. Station interfaces can use single paths but this compromises the security of The MESH control network. NOTE

 All graphics of switches and media converters in this document are intended as generic illustrations of networking concepts and do not necessarily reflect the currently offered products.

Switched Ethernet Characteristics Standard Fast Ethernet switches and fiber optic/copper cabling provide versatile solutions for building MESH networks using industry standard protocols. The 16-port or larger managed Ethernet switches used in The MESH control network allow connection of multiple control stations, workstations and other Ethernet switches. Unmanaged switches are not supported by The MESH control network because they offer no redundancy and you cannot run diagnostics if the switch should fail.

1

B0700AZ – Rev L

1. Introduction

The MESH Control Network Features Two types of network configuration methods are available in The MESH control network Standard Configuration and Security Enhanced Configuration.

Standard Configuration Features  The Standard Configuration of The MESH control network provides the following features. ♦

System scalability by interconnecting Ethernet switches with 16-ports or more in a linear, ring, star, inverted tree or modified inverted tree network topology (configuration). The topology is dependent on the network site requirements.

♦

Ethernet switches connected in a MESH configuration with up to 1920 I/A Series stations Support for Fast Ethernet (100 Mb) and Gigabit (uplink only) Ethernet (1000 Mb)

♦ ♦

Modular uplinks to high-speed backbones using 1 Gb 1000Base-T, 1000Base-SX, 1000Base-LX and 1000Base-ZX standards

♦

Full-duplex operation based on the IEEE 802.3 standards

♦

Rapid Spanning Tree Protocol (RSTP - IEEE 802.1w) which manages redundant paths, prevents loops, and provides high speed convergence time for a network  Network management and configuration via local port or Web access for various switches

♦

♦

♦

♦

System Management software for monitoring the health of the control system and managing equipment in the system Software in every station that manages redundant Ethernet ports in response to net work faults High speed response to network and station faults to provide a highly reliable redundant network.

Security Enhanced Configuration Features  The Security Enhanced Configuration is now available to provide additional features. Due to the recent advances in switch network technologies, the Security Enhanced Configuration of The MESH control network offers improvements in network security, Loop Detection and many additional features not offered in the Standard Configuration. The Security Enhanced Configuration deploys specific network topologies and switch configurations that allow for advanced network loop detection in the event of a RSTP failure. This advanced network loop detection minimizes the potential for a single point of failure that will degrade communications between devices in the network. Deploying the advanced network loop detection is accomplished by carefully designing the net work and correctly deploying the Loop Detection Policy (LDP) algorithms required for the net work design, and by following the network configuration requirements provided in this document.

2

1. Introduction

B0700AZ – Rev L

NOTE

 When designing a Security Enhanced Configuration network, each device/switch  within the network is required to be connected to two different switches within the network. If the network is constructed with less than two connections between devices/switches, data traffic could be disrupted if any single device fails or there is a fault that causes the loop detection algorithm Rule Hit.

Loop Detection Policy (LDP) Deployed on The MESH Control Network  Due to the design of the Standard or Security Enhanced Configurations of The MESH control network, redundant links form physical loops in the network and are controlled (Blocked) by Rapid Spanning Tree Protocol (RSTP), creating a logical loop-free network. In a Security Enhanced Configuration in addition to RSTP, Loop Detection (LDP) is deployed to block redundant loops that could occur in the event of an RSTP or Data Loop (storm) failure.  A loop is determined by establishing a well-known data path and its source. To establish a known path, the concept known as the “Beacon” is developed. The Beacon routinely sends out an IGMP data packet. When the data packet is seen at an unexpected source port, the assumption is that a loop occurred and an action (Rule) needs to occur. A switch’s port deployed with “Circuit Breaker” will disable the first port on which the incorrectly sourced packet is received. Recent chassis switches, such as the N-Series Platinum (DFE) switches (P0973BQ, P0973BR, P0973BT, and P0973BS), offer advanced packet switching services that can scope data packets beyond the source and destination MAC-address. By looking at other data points in the packets, the switch can make decisions on which of these data points to mark a packet on. Once a particular packet is identified, the switch can take action on it. The action of interest is disabling a looped port. Disabling this looping port maintains a loop-free network. The switches alert the network administrator with SNMP traps and syslog messages. These should be acted upon to 'fix' the net work loop in a timely manner. When disabled by the LDP, a disabled port can be monitored by SMDH via a link down trap. Other methods of monitoring and management of ports can be accomplished by utilizing the switch's CLI port or NetSight Policy Manager.

Virtual Local Area Networks (VLANs) on The MESH Control Network   A VLAN acts like an ordinary Local Area Network (LAN), but in a VLAN, connected devices do not have to be physically connected to the same segment. The VLAN allows devices located in separate areas or connected to separate ports to belong to a single VLAN group. Devices that are assigned to such a group will send and receive broadcast and multicast traffic as though they were all connected to a common network. VLAN-aware switches isolate broadcast, multicast, and unknown traffic received from VLAN groups, so that traffic from stations in a VLAN are confined to that VLAN.  Additional details about VLANs are provided in The MESH Control Network Operation, and Switch Installation and Configuration Guide  (B0700CA revision F or later), in Chapter 10 “VLANs Usage on The MESH Control Networks” and Appendix D “Understanding Virtual Local Area Networks (VLANs)”.

3

B0700AZ – Rev L

1. Introduction

The MESH Control Network The MESH control network is designed to provide multiple communication paths between any two devices or stations connected to the network. This network architecture provides very high availability, while reducing network complexity, cost, and maintenance requirements.

The MESH Control Network Topologies  There are several basic MESH control network topologies supported by the I/A Seriessystem in each of the Standard  and Security Enhanced Configurations. These are: ♦

Standard Configuration ♦

Linear Ring 

♦

Star

♦

Inverted tree

♦

Modified inverted tree

♦

♦

Security Enhanced Configuration ♦

Linear Star

♦

Inverted tree

♦

Modified inverted tree

♦

NOTE

 When deploying the Security Enhanced Configuration, The MESH control net work should be constructed using one of the four enhanced topologies listed above. The ring topology should never be used when deploying this configuration. Each configuration/topology listed has unique features and the one chosen for a particular net work depends on the specific requirements of the site or installation. The following diagrams provide examples of the different topologies as well as recommendations on where they might be used.

4

1. Introduction

B0700AZ – Rev L

Linear Configuration The linear configuration is appropriate for a small, two switch network, as shown in Figure 1-1. In this example a failure of any one component in The MESH control network does not affect the operation of the remaining components. The linear configuration does not require a root or backup switch configuration. Large chassis switches with hundreds of ports can be used in a linear configuration to create a large control system. A multiple switch linear configuration (more than two) is not supported due to its lack of network redundancy, which can result in loss of communication between two devices within the network.

Figure 1-1. Linear Topology

5

B0700AZ – Rev L

1. Introduction

Ring Topology This topology is suited to networks containing three to seven switches. As shown in Figure 1-2, each switch has connections to the two adjacent switches. In the event of a failed switch, the ring is broken and the network assumes the characteristics of the linear topology shown in Figure 1-1.

There is a limit of seven (7) switches between any two devices on the network. This limit is imposed by the I/A Series system and cannot be exceeded. Figure 1-2 illustrates a network composed of six managed switches configured in a ring. A net work in this configuration is able to handle a single component failure and still maintain its integrity. The ring configuration does not require a root or backup root switch configuration. The ring topology cannot be configured with the Security Enhanced Configuration. When LDP is deployed in a ring topology, the loop detection algorithm will run before RSTP can block the port resulting in an uplink port being disabled. The end result will produce a multiple switch linear topology which is not a supported configuration. When designing Security Enhanced Configurations, use one of the four supported topologies (on page 4).

Figure 1-2. Ring Topology (Standard Configuration Only)

6

1. Introduction

B0700AZ – Rev L

Star Topology The star topology is the preferred topology for all networks, provided that geographical constraints allow this topology. In a star topology, the switches at the outside edge of the network have connections to each of the two root switches. The root switches are connected to each other and the edge switches. Redundant data paths allow the network to continue to operate if any one component fails.

Figure 1-3 illustrates a star network containing ten edge switches and two root switches. In a Standard Configuration star topology utilizing Gold Series blades, as many as 40 edge switches can be connected to the Chassis switch using 1 Gb uplinks. When utilizing Platinum Series blades, as many as 166 edge switches can be utilized. For a Security Enhanced Configuration star topology (Policy enabled switches only) a maximum of 166 edge switches can be connected to the Chassis switch using 1 Gb uplinks. An inverted tree topology can be considered if there is a larger number of edge switches required (up to 250 switches). Refer to the table “Qualified Switch Standard/ Security Enhanced Configuration Compatibilities Matrix” in The MESH Control Network Operation, and Switch Installation and Configuration Guide  (B0700CA) for a list of switches capabilities.

7

B0700AZ – Rev L

1. Introduction

Figure 1-3. Star Topology

8

1. Introduction

B0700AZ – Rev L

Inverted Tree Topology The inverted tree topology is suited to very large networks with specific physical constraints. In this topology the switches are arranged in tiers, with the root switches in the top tier and up to three tiers below them. The root switches (Tier 1) are the only switches in the network that have connections between switches on the same tier; all other switches have two connections to switches in the tiers above them. This topology is supported in both the Standard and Security  Enhanced Configurations.

 When deployed in a Security Enhanced Configuration, all switches within the network must be switch types that support Loop Detection (LDP). Refer to the table “Qualified Switch Standard/Security Enhanced Configuration Compatibilities Matrix” in The MESH Control Network Operation, and Switch Installation and Configuration Guide (B0700CA) for a list of the switches applicable to either of these configurations.  An inverted tree network topology is illustrated in Figure 1-4.There can be no more than fourtiers of switches (including the root) in order to comply with the I/A Series system requirement limiting the number of switches between devices to seven.

9

B0700AZ – Rev L

1. Introduction

Note: Primary and Backup Root Switches have two connections.

Figure 1-4. Inverted Tree Topology

Modified Inverted Tree Topology The modified inverted tree topology is suited for very large enhanced networks with specific physical constraints and requirements. The modified inverted tree topology allows for standard qualified (lower cost) switches to be utilized at the outer edge, which allows for larger networks to be deployed at a lower cost. In this topology, the switches are arranged in tiers, with the root switches in the top tier with up to three tiers below them. The root switches (Tier 1) are the only switches in the network that have connections between switches on the same tier; all other switches have two connections to switches in the tiers above them. All outer edge switches are interfaced to the network on different tiers (e.g. tier 3 and tier 4). By doing this, all end devices with redundant 10

1. Introduction

B0700AZ – Rev L

connections interface to the network on different tiers. A modified inverted tree network topology is illustrated in Figure 1-5. There can be no more than four-tiers of switches (including the root) in order to comply with the I/A Series system requirement limiting the number of switches between devices to seven.

Figure 1-5. Modified Inverted Tree Topology

11

B0700AZ – Rev L

1. Introduction

The MESH Control Network Network Example The MESH control network utilizes qualified Fast Ethernet switches which are configured to form a highly robust redundant network. Figure 1-6 on page 13 shows an example with several I/A Series stations and Control Processors connected redundantly to The MESH control network.  Workstations are also redundantly connected to the Fast Ethernet switches. The Z-module Control Processor 270 (ZCP270) main processor’s Ethernet ports connect to The MESH control network, where the ZCP270 I/O controller (IOC) ports have t he option of connecting directly to The MESH control network or a dedicated I/O network. The Fieldbus Communications Module (FCM100Et or FCM100E) provides the interface between the ZCP270 and the FBMs and connects to The MESH control network or a dedicated I/O network.  When designing The MESH control network architecture, the following concerns should be addressed: For critical I/O communications, which in this context can be defined as I/O applica♦ tions that cannot allow for a disruption of fieldbus communications over a five second span, it is recommended that the ZCP270 IOC and FCM I/O be placed on a separate network. For non-critical I/O communications, the FCM100s and ZCP IOC ports can con♦ nect directly to The MESH. Whenever possible, it is recommended that the ZCP IOC ports and FCMs be attached to the same switches within the network, keeping the ZCP IOC to FCM communications local to the switch. (See Figure 1-6.) If a dedicated I/O network is to be employed, a simple linear topology is used and switch selection should be per the standard guidelines for The MESH, where the ZCP270 I/O controller (IOC) ports A/B and FCM I/O ports A/B (Fieldbus A and Fieldbus B) A side and B side are separated on independent networks. It is also possible to have multiple I/O networks, such as having an I/O network for each ZCP IOC and its FCMs. ! WARNING Switches on a dedicated I/O network cannot be seen in SMDH or System Manager. However, Fieldbus A and Fieldbus B errors are indicated in a normal manner via the “Sys” Key and are visible in SMDH or System Manager.  Also, a dedicated I/O network cannot be used for FBMs using the GPS SOE/TDR time sync package (see Time Synchronization User's Guide  (B0700AQ)).

For more information on configuring a dedicated I/O network, refer to “I/O Network Design Rules” on page 36 and “I/O Network Topology Configurations” on page 58.

12

1. Introduction

B0700AZ – Rev L

ZCP270 INFORMATION NETWORK WORKSTATIONS

SPLITTER/  COMBINERS

ETHERNET SWITCHES

The MESH CONTROL NETWORK

SPLITTER/COMBINER (NOT USED WITH F CM100E) DIN RAIL BASEPLATE FCP270 FBM

FBM

FCM100Et or FCM100E

DIN RAIL BASEPLATE TO/FROM PROCESS

TO/FROM PROCESS

Figure 1-6. The MESH Control Network Network (Star Topology Topology Shown)

13

B 0 7 0 0 A Z – Re v L

1. Introduction

The MESH CONTROL NETWORK

ZCP270

SPLITTER/  COMBINERS

A

I/O NETWORK

B

SPLITTER/COMBINER (NOT USED WITH FCM100E) DIN RAIL BASEPLATE

FBM

FCM100Et or FCM100E

TO/FROM PROCESS

Figure 1-7. The MESH Control Control Network with an I/O Network  Network 

14

1. Introduction

B0700AZ – Rev L

The MESH Control Network Specifications Table 1-1. The MESH Control Network Specifications Specifications Number of I/A Series stations connected connected to The MESH control network

1,920 I/A Series stations including switches, switches, maximum (FCM100Ets (FCM100Ets and FCM100Es are not included in count). Up to 250 switches.

Number of Ethernet switches between any two stations

Seven maximum

Number of IP addresses

10,000 maximum includes switches, controllers, workstations and FCM100Ets and FCM100Es.

VLAN VLANs s on The The MES MESH H cont contro roll netw network ork

Six Six (6) (6) conf config igur urab able le VLA VLANs Ns are are supp support orted ed on on The The MESH MESH Cont Contro roll Network

Standards Suppor ted

100 Mb Full-duplex operation for fiber optic and copper cable. Modular uplinks using 1 Gb 1000Base-T, 1000Base-T, 1000Base-SX and 1000Base-LX standards

Speeds Suppor ted

Fast Et E thernet (100 Mb) and uplink Gigabit Ether net (1000 Mb)

Protocols Used

Rapid Spanning Tree Protocol (RSTP - IEEE 80 8 02.1w), 802.3, 802.3ad

Cable Lengths - Interconnecting stations or Ethernet switches

CAT5:100Base-TX CAT5:100Base-TX or 1000Base-T; 100 m (328 ft) maximum Fiber optic: 100Base-FX; 2 km (6,560 ft) maximum (MMF) 1000Base-SX; 275 m (900 ft) maximum (MMF) 1000Base-LX/LH; 2 km (6,560 ft) maximum (MMF) 1000Base-LX; 10 km (6.2 mi) maximum (SMF) 1000Base-ZX; SMF, 80 km (49.7 miles) maximum

Cable Lengths - Total connection length allowed between switches

Single mode fiber (SMF), 80 km (49.7 mi) maximum Multimode fiber (MMF), 2 km (6,560 ft) maximum

MESH Con Conttrol rol Net Netw work ork Di Dista stance nce

It is pos possi sib ble to ext exte end The The MES MESH H con conttrol rol net netw work ork dis disttance ance using sing various third-party network equipment and extenders. It should be noted that the total network delays between two end devices should not exceed 100ms roundtrip.

For more information, refer to the following Product Specifications Sheets: The MESH Control Network Architecture  (PSS  (PSS 21H-7C2 B3) ♦ ♦

The MESH Control Network Ethernet Equipment (PSS 21H-7C3 B4).

The MESH Control Network Workstations I/A Series workstations can can be connected to The MESH control network. network. These workstations provide host services to fault-tolerant control processors. In addition, the workstation provides the operator interface for the display of graphic and textual information. Each workstation connects to the switches in The MESH control network by way of copper or fiber interface cards in the workstation. Two Network Interface Cards (NIC) are offered: ♦

100Base-FX

♦

10/100Base-TX 

The workstation can be directly connected to Ethernet switches or through the use of a media converter, described in Media Converter Installation and Configuration Guide for Control Networks  (B0700CP). In addition, the built-in Ethernet interface on the workstation’s motherboard can be used to interface to a plant information network.

15

B0700AZ – Rev L

1. Introduction

The MESH Control Network Ethernet Switches  A switch is an active multiport network and bridge device that provides a separate collision domain for each port, and uses Media Access Control (MAC) layer to direct network packets to the appropriate station or switch. This allows multiple simultaneous communications among net work devices connected to the switch. The MESH control network utilizes standard commercial off-the-shelf Ethernet switches to allow you to configure your system to meet your functional, performance and plant requirements. Switches listed in “Reference Documents” on page xii have been tested and qualified by Invensys for use with I/A Series products. Other Fast Ethernet switches from other vendors may be allowed to be on The MESH control network. Using non-qualified switches may cause unpredictable failures or responses. The list of vendors and their qualified switches can be obtained from the Global Client Support Center (Global CSC) web site at http://support.ips.invensys.com . Refer to the documentation included with your Invensys qualified Ethernet switches for details of their capabilities. NOTE

The MESH control network was designed and tested for operation with the Ethernet switches listed in “Reference Documents” on page xii. The network may operate  with similar, off-the-shelf equipment, but Invensys Systems, Inc. is not responsible for any system malfunctions that may occur if such equipment is used. If you use your own network: 1. You must meet the bandwidth requirements for the I/A Series equipment you have chosen (1 Gb for uplinks and 100 Mb for ports). 2. There can be no Layer 3 inter-network devices (for example, routers) between any I/A Series equipment. 3. Typically a failover time of less than 1 second is achievable using Fast Ethernet switches qualified and supplied by Invensys and configured in accordance with I/A Series documentation. The network management module for the Fast Ethernet switches, provides a menu-driven or Command Line Interface (CLI) system configuration program with management capability. See the documentation included with your Invensys qualified Ethernet switches and the vendor manuals listed in “Reference Documents” on page xii for further details.

Advantages of Invensys-Supplied Switches The MESH control network requires switches purchased from Invensys. This provides a number of advantages, described below. ♦

Can customers include Ethernet switches from other suppliers for use in The MESH control network? Only Invensys-supplied Ethernet switches have been qualified for use in the I/A Series system. Other switches may or may not work and may not meet the performance specifications required for a secure, reliable high performance in The MESH control network.

16

1. Introduction ♦

♦

B0700AZ – Rev L

 Why won't other switches switches provide the same same performance? The Invensys-supplied Invensys-supplied switch is off the shelf equipment and has been optimized to meet the stringent performance specifications for The MESH network. For example, the Rapid Spanning Tree Protocol algorithm that manages traffic paths is optimized to meet the requirement requirement for network recovery recovery on failure of a root root switch in less than one second. This level of performance is much better than is typically achievable with other vendor's hardware and RSTP implementation. Can customers purchase switches from third parties with the correct certified firm ware to work with with the I/A Series system?  Although switches switches qualified for use with I/A Series systems can can also be purchased purchased from third parties, they will not necessarily have been furnished with the correct firmware version that has been qualified. The I/A Series Switch Configurator  Application Software Software (SCAS), (SCAS), supplied with the Invensys-suppl Invensys-supplied ied switch, is also designed to configure switches qualified by Invensys, making configuration easier, easier, quicker, quicker, more reliable and facilitates troubleshooting and configuration verification.

♦

How could a user get the correct firmware? They would need to t o contact Invensys and either purchase replacement firmware firmware for each switch on a one time charge basis or purchase an Invensys support contract which would cover the cost of replacement firmware.

♦

Is the firmware provided by Invensys for The MESH network switches exclusive to Invensys-supplied Invensys-supplied switches? Changes to the firmware used in I/A Series systems are included included in the standard product. Periodically, changes are made to the firmware; future versions of the firmware  which have not not been qualified by Invensys Invensys may or may not not be compatible with I/A Series Series systems systems..

Control Network Cabling Three different types of cabling of various lengths may be used: ♦

Shielded twisted-pair 100Base-T CAT5 cabling - 100 m (328 ft.) maximum

♦ ♦

Shielded twisted-pair 1000Base-T CAT5 cabling - 100 m (328 ft.) maximum Multimode fiber optic cabling - 2 km (1.25 mi) maximum for 100Base-FX 

♦

Multimode fiber optic cabling - 275 m (900 ft) maximum for 1000Base-SX 

♦

Multimode fiber optic cabling - 550 m (1800 ft) maximum for 1000Base-LX 

♦

Multimode fiber optic cabling - 2 km (1.25 mi) maximum for 1000Base-LX/LH

♦

Single mode fiber optic cabling - 10 km (6.21 mi) maximum for 1000Base-LX.

♦

Single mode fiber optic cabling - 80 km (49.68 mi) maximum for 1000Base-ZX/ELX.

17

B 0 7 0 0 A Z – Re v L

1. Introduction

Category 5 Cabling There are two basic configurations of Category 5 (CAT5) cables: Straight-through Straight-through cables: cables used to connect workstations to Ethernet switches, and ♦ media converters to Ethernet switches. Crossover Crossover cables (also called a null hub): cables used to interconnect Ethernet switches. Some switches have auto MDI/MDI-X ports (for example, 24-Port 24-Port Copper) which provides the crossover and do not require a crossover cable. ♦

NOTE

 All category catego ry 5 Cabling Cablin g must be of shielded s hielded type t ype for optimal o ptimal interference int erference mitigation. m itigation. Normally, when one switch is connected to another, the transmit and receive wires must be crossed over, over, such that the transmit wires from switch #1 connects to the receive wires from switch #2, and vice versa. Crossover cables are used much less frequently than straight-through straight-through cables. A straight through cable can be used as a crossover crossover cable if used in conjunction with a null hub cable adapter (P0971PK). (P0971PK).  A null hub is a very short cable that has a male RJ-45 RJ-45 connector on one end and a female RJ-45 RJ-45 connector on the other. The transmit and receive wires are reversed, reversed, so when it is connected to the end of a straight-through straight-through cable, the resulting cable system can act as a crossover cable. The null hub is used to interconnect switches using straight-through CAT5 CAT5 when neither switch is equipped with auto MDI/MDI-X ports (crossover port).

Fiber Optic Cabling Fiber optic cable is used to connect workstations to Ethernet switches and to make connections between Ethernet switches. The fiber optic cable’s cable’s electrical isolation characteristics provide protection from voltage differentials and ground loops and permit communication installations to pass through areas where intrinsically safe operation is required. The fiber optic cable is unaffected by electrical noise such as EMI and RFI and can be installed even in the following cases: containing rotating machinery machinery,, arc welders, and so forth ♦  Areas containing ♦ ♦ ♦ ♦

Cable trays containing high voltage power lines Outdoor areas exposed to lightning hazards (with appropriately rated cable)  Areas containing containing strong magnetic magnetic fields Longer distances than twisted pair cable.

Two different types of fiber optic cable may be used in The MESH control network:

18

♦

Single mode cable

♦

Multimode Multimode cable (switch-to-device (switch-to-device connections and uplink connections).

1. Introduction

B0700AZ – Rev L

NOTE

Single mode and multimode fiber optical devices are not compatible. Both devices being connected (and the cable) must be of the same type in order to ensure proper operation. In certain limited cases (connecting Ethernet switch uplink ports, for example), a mode conditioning cable may be employed so that multimode fiber cable can be used with a single mode device. Refer to the diagrams in the “Ethernet Switch Interconnection Diagrams” in the documentation included with your Invensys qualified Ethernet switches for specific information.

NOTE

The fiber optic cables mentioned in this document may require additional mechanical protection, particularly particularly when run between different enclosures. enclosures.

Single Mode Cable  In the control network, single mode fiber optic cable can be used to connect Ethernet switches to each other through each switch’s switch’s uplink port. The uplink ports of each of the switches being connected must be designed for single mode operation. Refer to the “Ethernet Switch Interconnection Diagrams” in the documentation included with your Invensys qualified Ethernet switches for information on uplink modules for use with single mode fiber optic cable.

Multimode Cable  Multimode Multimode fiber optic cable is employed in different situations in The MESH control network: Controller to Switch - A controller connects through splitter/combiner(s) to the ♦ Ethernet switch(es) 100B 100Base-FX ase-FX port(s) using a multimode fiber optic cable terminated with MT-RJ connectors. The connection is LC on one end and MT-RJ on the other end between the switch and the splitter. splitter. Ethernet switch(es) without fiber optic port(s), can use media converter(s) between the switch(es) and the splitter/combiner(s) to enable a connection. ♦

Field Communications Module (FCM) to Switch - A redundant FCM connects through splitter/combiners to the Ethernet switches 100Base-FX ports using a multimode fiber optic cable terminated with MT-RJ connectors. The connection is LC on one end and MT-RJ MT-RJ on the other end between the t he switch and the splitter. splitter. Ethernet switch(es) without fiber optic port(s), can use media converter(s) between the switch(es) and the splitter/combiner(s) to enable a connection. A single FCM module does not use a splitter/combiner and connects directly to a switch via an LC/MT-RJ LC/MT-RJ fiber cable.

♦

 Workstation  Workstation to Ethernet Ethernet Switch Switch - A workstation is capable capable of connecting directly to Ethernet switch’s 100Base-FX port using a multimode fiber optic cable terminated  with MT-RJ MT-RJ connectors. An An Ethernet switch switch without a fiber optic port, can use two media converters between the switches and the workstations. Connection is based on NIC type and switch type. Media converters can be used when NICs do not match switch ports.

19

B0700AZ – Rev L ♦

1. Introduction

Ethernet Switch to Ethernet Switch - Ethernet switches that are equipped with multimode fiber uplink ports may be connected directly to each other. The fiber optic cable should be terminated with MT-RJ type or LC type connectors, depending on the switch interface modules.

The MESH Control Network Management Software Tool, NetSight® Console The NetSight® Console enables enterprise-wide management of The MESH control network to provide network-wide monitoring and troubleshooting, such as device discovery, topology mapping, and event management. The enterprise-level command and control console provides: ♦

♦

Multi-element management approach to facilitate the abstraction of complex network policy into everyday business language System-level monitoring, and troubleshooting capabilities such as device discovery, event management, and logging.

NetSight Plug-ins use the advanced features while reducing administrative burden and lowering total cost of ownership. The Plug-in applications include the NetSight Policy Manager.

Netsight Policy Manager  An advanced plug-in application for NetSight®, the NetSight Policy Manager enables simple management of complex network security policies to greatly enhance reliable network connectivity. It provides the following. ♦

Role-based enterprise management ♦

Defines Roles, Rules and Services of the network

♦

Matches the role of the device with available network services

Facilitates a distributed firewall to all edge points in the network  Automated capabilities - offers ease of implementation, administration and troubleshooting ♦

♦

♦

Complex policy management - provides an Audit Trail (event log)

The following tables list the various NetSight Policy Manager software available. Table 1-2. NetSight Console Policies

Part Number NS-CON-50 NS-CON-U NS-CON-U-UG

20

Detailed Description NetSight Console 50 Devices (50 device license for 1 server plus 3 concurrent user licenses) NetSight Console Unrestricted (Unrestricted device license for 1 server plus 25 concurrent user licenses) NetSight Console 50 to Unrestricted Upgrade

1. Introduction

B0700AZ – Rev L

Table 1-3. NetSight Client Policy

Part Number NS-USER

Detailed Description NetSight User (Add 1 concurrent user license to existing NetSight server)

Table 1-4. NetSight Policy Bundle

Part Number NS-PB-50

NS-PB-U

NS-PB-U-UG

Detailed Description NetSight Policy Bundle 50-devices (50 device Console license for 1 server plus 3 concurrent users, Policy Manager and Policy Control Console) NetSight Policy Bundle Unrestricted (Unrestricted device Console license for 1 server plus 25 concurrent users, Policy Manager and Policy Control Console) NetSight Policy Bundle 50 to Unrestricted Upgrade

Table 1-5. NetSight Advanced Bundle

Part Number NS-AB-50

NS-AB-U

NS-AB-U-UG NS-AB-50FT

NS-AB-UFT

NS-AB-UFT-UG

Detailed Description NetSight Advanced Bundle 50-devices (50 device Console license for 1 server plus 3 concurrent users with Policy Manager, Policy Control Console, Automated Security Manager, Inventory Manager, and NAC Manager) NetSight Advanced Bundle Unrestricted (Unrestricted device Console license for 1 server plus 25 concurrent users with Policy Manager, Policy Control Console, Automated Security Manager, Inventory Manager, and NAC Manager) NetSight Advanced Bundle 50 to Unrestricted Upgrade NetSight Advanced Bundle 50-devices FT (50 device Console license for 1 server plus 3 concurrent users, Policy Manager, Policy Control Console, Automated Security Manager, Inventory Manager, NAC Manager, a redundant NetSight license for fault tolerance (manual failover), Includes Lab License) NetSight Advanced Bundle Unrestricted FT (Unrestricted device Console license for 1 server plus 25 concurrent users, Policy Manager, Policy Control Console, Automated Security Manager, Inventory Manager, and NAC Manager, a redundant NetSight license for fault tolerance (manual failover), includes Lab License) NetSight Advanced Bundle 50 to Unrestricted FT Upgrade

21

B0700AZ – Rev L

1. Introduction

Table 1-6. NetSight Single User Policy

Part Number NS-SU-10

Detailed Description NetSight Single User 10-devices (10-device, single client (on same machine as server) license for Console, Policy Manager, and Inventory Manager)

Table 1-7. NetSight A-La-Carte Plug-Ins

Part Number NS-ASM NS-IM NS-NAC NS-PM NS-LAB

Detailed Description NetSight Automated Security Manager (NetSight Automated Security Manager. Requires existing NS-CON-50 or NS-CON-U license) NetSight Inventory Manager (NetSight Inventory Manager. Requires existing NS-CON-50 or NS-CON-U license) NetSight NAC Manager (NetSight NAC Manager. Requires existing NS-CON-50 or NS-CON-U license) NetSight Policy Manager (NetSight Policy Manager. Requires existing NS-CON-50 or NS-CON-U license) NetSight Lab License (Non-production use, one-time fee inclusive of maintenance. 50 device license for 1 server plus 2 concurrent users  with Policy Manager, Automated Security Manager, Inventory Manager, NAC Manager, Policy Control Console. Requires existing NSCON-50 or NS-CON-U license)

Table 1-8. NetSight Appliance Policy

Part Number SNS-NSS-A

Detailed Description NetSight Appliance (Hardware only, requires separate NetSight license)

Obtaining Network Management Software To obtain Invensys pricing discounts, use the following contact information: Holly O'Gara  978-878-4579 (U.S. Number) [email protected] 

OR

Christine Leblanc 978-684-1559 (U.S. Number) [email protected] 

Enterasys Networks Corporate Headquarters 50 Minuteman Road  Andover, MA 01810 U.S.A   A free evaluation copy of the software can be downloaded at: http://www.enterasys.com/products/management/NSA-LIC/  22

2. Site Planning  This chapter describes the steps that should be taken and the options that should be considered when planning The MESH control network. The design of each instance of The MESH control network is different, and depends on the needs and requirements of the individual site. The following paragraphs provide the information necessary to help network designers to plan The MESH control network that meets the needs of their specific site.

Site Planning Overview The MESH control network allows communication between the control stations, workstations, and Field Communications Modules (FCMs) connected to the network. This network is formed by interconnecting the control stations and workstations through the use of fiber optic or copper cable, Ethernet switches and, if necessary, media converters. Although copper cable can be used, fiber cable is recommended for industrial networks. In a properly constructed network, all the stations on the network are able to communicate with each other. For a high degree of reliability, The MESH control network should be constructed so that there are redundant signal paths between each device on The MESH control network.  Although the design of The MESH control network is driven largely by the site’s physical environment, traffic and organizational requirements, there are rules that must be followed when connecting devices to form The MESH control network. The following two sections explain the factors affecting the physical and the traffic considerations of The MESH control network.

Network Considerations The physical location of equipment at the site influences The MESH control network design. To choose equipment and cabling appropriate to the requirements for the specific site, use the answers to the following questions, along with The MESH control network design rules and device/cable specifications in this chapter and Appendix A “COMEX Fault Handling on The MESH Control Network”. ♦  What is the maximum end-to-end distance between devices in The MESH control network? The distance between the ends of The MESH control network may determine  what kind of Ethernet switches are used and what kind of cabling is used between those switches. If individual cable runs are greater than a hundred meters, it is necessary to use fiber optic cable, due to its capability for transmitting signals over longer distances. All hardware must be considered when the network requires multiple hops; no low end standalone switches should be used in the network core. It should be noted that the total network delays between two end devices should not exceed 100 ms. ♦

 Will The MESH Control Network have VLANs?

23

B0700AZ – Rev L

2. Site Planning

NOTE

If the answer to this question is No, it is recommended that when setting up a first time installation of The MESH control network that VLAN 2 “I/A Control Ports” is deployed on all ports used for communications between I/A Series devices. (See the warnings below.) If a system is up and functional without VLAN 2 deployed across the network, a system shutdown is required to deploy VLANs. If VLAN 2 is deployed prior to system startup, additional VLANs can be added at a later date without system interference (shutdown). ♦

If yes, how many VLANs will The MESH control network require? The MESH control network will support up to six VLANs, one (1) of which must be reserved for I/A Series system devices (VLAN 2 “I/A Control Ports”), and all I/A Series control devices shall be attached to VLAN 2 FE ports. Only one I/A Series control system can be supported on The MESH control network. There can be no duplicate MAC addresses across The MESH control network VLANs. If VLANs are deployed, all switches in the network must have VLAN 2 enabled on the switch’s uplink ports.

! WARNING  When VLANs are added to an existing installation of The MESH control network, if the I/A Series devices are on VLAN 1, they must be moved to VLAN 2, at which time communications between the I/A Series devices will be broken.

! WARNING  All I/A Series devices must be connected to Device ports which have been assigned to VLAN 2 “I/A Control Ports”. If this is not done, the I/A Series devices on the network will not communicate correctly with each other. Refer to the “Flowchart for Assignment of Uplinks and Ports to VLAN” figure in the “Switch Configuration Parameters Dialog Box” section in The MESH Control Network Operation, and Switch Installation and Configuration Guide  (B0700CA).

If VLANs are enabled, all switches in the network must have VLAN 2 “I/A Control Ports” set to ENABLED. ! WARNING If VLANs are to be utilized in The MESH control network, all switches within the network are required to be configured for all utilized VLANs. If a VLAN is configured on an outer edge switch and a core switch has not been configured for that VLAN (in the case where no port assignment is required), data from the outer edge switch VLAN will not propagate through the core switch.

24

2. Site Planning

B0700AZ – Rev L

♦

 What will the protocols be on the non-I/A Series VLANs?

♦

VLANs other than VLAN 1 and 2 shall not utilize protocols other than TCP/IP and/or UDP/IP.  What will be the loading on the FE device ports? No VLAN Port shall exceed 50% loading.

♦

♦

 What is the non-I/A Series VLAN end device speed usage?

No end devices with transfer (Tx) and receive (Rx) speeds greater than 100Mb are supported.  Will the switch be installed in an adverse environment? For the requirements for industrial switches in The MESH control network, refer to I-Series (P0973GB) Industrial Switch, Hardware and Software Configuration Instructions  (B0700CN) for the I-series Industrial switch specifications.

♦

How many control stations and workstations will be connected to The MESH control network? The number of control stations and workstations influences how many Ethernet switches and separate cables are included in The MESH control network. It may be wise to plan for expansion and growth when arriving at this number.

♦

♦

How are the control stations and workstations distributed? The physical distribution of control stations and workstations can have a bearing on the type and number of Ethernet switches, as well as the type of cabling used. If the control stations and workstations are clustered together, two larger capacity switches may be able to accommodate them all. If they are dispersed, several smaller switches may be a better solution. For large distances and noise protection, you should use fiber optic cable between switches, or between switches and control stations, workstations, or FCMs. Cable routing between devices must also be considered.  Will The MESH control network have redundant signal paths?  Yes, a control network with redundant paths is recommended for all control systems.

♦

Through what kind of physical environment will the cabling pass? Fiber optic cable is immune to magnetic fields and electrical noise, so it can be used in places where copper cable would be unreliable. If cabling must be routed through areas subject to high EMI or RFI, fiber optic cable is recommended. Due to its immunity to lightning, fiber optic cable should also be used for outdoor cable runs (rated for outdoor use).

♦

 Where will the equipment be located?  Will switches be mounted in racks, in cabinets, or placed on shelves or tabletops? Take into consideration thermal requirements, especially if the equipment will be mounted in an enclosed cabinet or area. High or moderate electromagnetic noise sources, for example, machinery, switchgear, high-voltage lines, and so forth, in close proximity must be avoided to ensure reliable operation.

♦

Have plans for future expansion been considered?

25

B0700AZ – Rev L

♦

2. Site Planning

Consider installing Ethernet switches with additional ports so that control stations or workstations can be easily added in the future. Running extra copper or fiber optic cable will allow for network expansion later. Have maintenance and troubleshooting provisions been made? Leaving at least one unused port on each Ethernet switch allows for maintenance and troubleshooting of managed switches. A system network management package (SNMP) must be available that runs on a separate PC, not an I/A Series workstation. Patch panels at the end of cable segments facilitate troubleshooting and network re-configuration.

♦

Newer switches use 1 Gb uplinks. Can a new MESH network be built including legacy 100 Mb uplinks? ♦

♦

 While a MESH network may be built using 100 Mb uplinks, this is not recommended due to loading concerns. Note that Security Enhanced Configurations require the use of 1 Gb uplink ports.

 Are ample ac power outlets available at the proper location: For the Chassis switch? ♦ ♦

For switches?

♦

For redundant power supplies? For media converters and ac to dc power adapters for media converters?

♦

Traffic Considerations Specific traffic requirements should be considered when planning The MESH control network. ♦

Should certain control stations or workstations be connected to the same switch? To reduce traffic through the root switches, it may be advantageous to group control stations or workstations according to department, process, or other criteria that is important to the site or organization.

♦

Should certain control stations or workstations not  be connected to the same switch? It is recommended that each control station or workstation from a particular group be connected to two separate switches. If one switch were to fail, these control stations or workstations can access the network through the redundant switch. Without a redundant network, none of the workstations from that group  would have network access. Workstations can have two Ethernet ports to access the network.

Network Loading Understanding the details of the network traffic flow is an important part of planning and implementing The MESH control network for an I/A Series system. This provides insight on how to verify that there is sufficient network bandwidth available between network devices. This is difficult to measure, since available bandwidth is a dynamic quantity where the amount of traffic that can be transmitted over a link can change given its current traffic conditions and what applications are running on those devices. The process of designing industrial networks is a very challenging task due to its inherent complexity. A load prediction can be achieved by approximating the environment, modeling the network components and analyzing the interrelations. This pro-

26

2. Site Planning

B0700AZ – Rev L

cess works well for industrial networks since they tend to be more static and are typically slow to evolve. It is recommended that The MESH control network steady state load not exceed a maximum load of 50% of any given link. The MESH control network is a full-duplex, point-to-point system. This network does not have a collision domain. Unlike half-duplex links, full-duplex links do not deteriorate as the network load increases. All end-devices can transmit at will. However, if The MESH network is designed  with 100 Mb uplinks communication from these end-devices, this could cause a bottleneck due to insufficient available network bandwidth1. Since The MESH network runs in full duplex mode, both transmit and receive channels can run at 100 Mb simultaneously without degradation on the adjacent channel. The MESH network traffic rates are also affected by third-party applications or user applications that generate high packet rates. Workstation to workstation operations on The MESH network, such as copying extremely large files, can also result in a temporary high bandwidth usage up to 50% of the network: Network loading will be represented by the percent of time that the network is in use over a given period. By definition, individual Ethernet segments can only transmit one packet at a time. For any given moment, the Ethernet segment is either at 100% loaded (transmitting a packet), or at 0% utilization (idle). The network loading percentage shows the percentage of time the network is in use over a set period.  When calculating the network loading, you need to know how many bytes of network traffic are being handled by the network over a set period. This involves totaling the input (or output) byte counts for a set period, and dividing by the total capacity of the device interface for that period. To determine the total number of bits received on the interfaces, each of the packet byte rates is multiplied by 8.  Where:  = total number of bytes sent  = maximum connection speed  = duration of time required for transmissions For example, for a system having 6 x 1500 byte sends on 100Mb connection at maximum of 2Mb (2 * 1,000,000 bps) every 500 milliseconds:

Equipment Considerations The MESH control network was designed and tested for operation with the Ethernet switches listed in “Reference Documents” on page xii. The network may operate with similar, off-the-shelf

1.

Flow control and rate limiting are disabled, and full-duplex cables are used, allowing the controllers transmitted packet to egress though the network without delaying or filtering. However, by doing this, a packet on a heavily loaded network can cause traffic to slow. When using these features, the switch acts as a traffic cop controlling the data to maximize the traffic flow. Disabling these features allows the traffic to flow faster but can cause issues when traffic loads are high. 27

B0700AZ – Rev L

2. Site Planning

equipment, but Invensys Systems, Inc. is not responsible for any system malfunctions that may occur if such equipment is used. If you use your own network equipment: 1.  You must meet the bandwidth requirements (1 Gb for uplink ports and 100 Mb for ports) for the I/A Series equipment you have chosen. 2. There can be no Layer 3 inter-network devices (for example, routers) between any I/A Series equipment.

Hardware Requirements  While switches with 100 Mb uplink ports are allowed on networks with standard configurations, Security Enhanced Configurations require the use of 1 Gb uplink ports. However, 1 Gb uplink connections should be used to interconnect switches in all configurations, if available.

Switch Utilization in the Standard Configurations   Any switches defined in the table “Invensys-Supplied Ethernet Switches” in the “Introduction” of The MESH Control Network Operation, and Switch Installation and Configuration Guide  (B0700CA) can be used in Standard Configurations. Note that C-Series switches (P0973BL/HA) are used only in a Standard Configuration and cannot be used in the Security Enhanced Configuration.

Switch Utilization in the Security Enhanced Configurations  Refer to the Appendix table “Qualified Switch Standard/Security Enhanced Configuration Compatibilities Matrix” in The MESH Control Network Operation, and Switch Installation and Config-  uration Guide  (B0700CA) for a list of allowed switch types in Security Enhanced Configurations. Since the Security Enhanced Configurations deploy Loop Detection Policy algorithm (LDP) methods to monitor and protect the network from RSTP or Data Loop (storm) failures, specific switch types must be used in a variety of required locations within the network. The table listed above identifies the switch types which fall into the categories “Security Enhanced” and “LDP Deployable”. ♦

♦ ♦

♦

♦

28

In a linear topology in a Security Enhanced Configuration, both switches must be from the “LDP Deployable” category.  A ring topology cannot be supported as Security Enhanced Configuration. In a star topology in a Security Enhanced Configuration, both root and backup root switches must be from the “LDP Deployable” category. All other switches within the star network can be from the “Security Enhanced” category. In an inverted tree topology in a Security Enhanced Configuration, all switches  within the network must be from the “LDP Deployable” category. The modified inverted tree topology in a Security Enhanced Configuration requires that all switches within the network core are “LDP Deployable”. All outer edge switches within the tree network can be any switch type; however, the redundant switches (A/B switches) must be deployed on different tiers.

2. Site Planning

B0700AZ – Rev L

Firmware Considerations  When planning which Invensys-supplied Ethernet switches to use when building The MESH control network for your system, be aware that certain versions of these switch’s firmware may be incompatible with one another. Refer to the Appendix “Qualified Switch Firmware Compatibilities Matrix” in The MESH Con-  trol Network Operation, and Switch Installation and Configuration Guide  (B0700CA) for the I/A Series qualified and compatible firmware for each switch.

The MESH Control Network Design Rules  When designing The MESH control network, the following rules/guidelines should be kept in mind.

Standard Configuration Design Rules The following rules apply to the standard design of The MESH control network: 1. No more than 1920 logical stations where: ♦ ♦ ♦ ♦

Fault-tolerant stations count as one logical station  Workstations count as one logical station Ethernet switches count as one logical station. FCMs are not counted.

2. No more than 10,000 IP addresses where: ♦  A fault-tolerant FCP270 counts as one address ♦

 A fault-tolerant ZCP270 counts as two addresses

♦

 An ATS station counts as two addresses MESH workstations count as two addresses prior to I/A Series system version 8.2; MESH workstations with I/A Series system V8.2 or later count as one address. Nodebus workstations count as one address

♦

♦ ♦

Ethernet switches count as one address

FCMs count as one address. 3. No more than 250 managed switches are allowed. ♦

4. There should be no more than seven switches in the path between any two devices in The MESH control network. 5. Routers are not allowed in The MESH control network. 6. Switch-to-Switch connections (uplink ports) should be made using 1 Gb uplink ports to allow enough bandwidth for network traffic of I/A Series equipment 7. Only two configured root bridges are allowed in the network. 8. Horizontal switch interlinks (links between switches of the same tier) are not allowed, except between the root and backup root switches. (See Figure 2-1 on page 33). 9. The MESH control network must have redundant uplink connections between the root and backup root switches.

29

B0700AZ – Rev L

2. Site Planning

10. The MESH control network will support up to six VLANs, one (1) of which must be reserved for I/A Series system devices (VLAN 2 “I/A Control Ports”). a. Only one I/A Series control system can be supported on The MESH control net work. b. If VLANs are deployed, all I/A Series control devices shall be attached to VLAN 2 FE ports. c. If VLANs are deployed, all switches in the network must have VL AN 2 enabled on the switch’s uplink ports. d. VLANs other than VLAN 1 and 2 shall not utilize protocols other than TCP/IP and/or UDP/IP. e. No VLAN port shall exceed 50% loading  f. No end devices with transfer (Tx) and receive (Rx) speeds greater than 100Mb are supported. NOTE

It is recommended that one port on each managed Ethernet switch be reserved for testing and diagnostic purposes. No device should be connected to this port.

Non I/A Series Equipment  The following rules apply to non I/A Series equipment: 1. Non I/A Series equipment (for example, routers, hubs and unmanaged switches) are not allowed to be connected to The MESH control network. 2.  A corporate or plant enterprise network should not be connected directly to The MESH control network. The recommended way to communicate from a corporate or plant enterprise network is by using an additional NIC on an I/A Series workstation.

Security Enhanced Configuration Design Rules  When designing the Security Enhanced Configuration, the following rules/guidelines must be followed. 1. No more than 1920 logical stations where: ♦ ♦ ♦ ♦

Fault-tolerant stations count as one logical station  Workstations count as one logical station Ethernet switches count as one logical station. FCMs are not counted.

2. No more than 10,000 IP addresses where: ♦  A fault-tolerant FCP270 counts as one address ♦

 A fault-tolerant ZCP270 counts as two addresses

♦

 An ATS station counts as two addresses MESH workstations count as two addresses prior to I/A Series system version 8.2; MESH workstations with I/A Series system V8.2 or later count as one address.

♦

30

2. Site Planning

B0700AZ – Rev L

♦

Nodebus workstations count as one address

♦

Ethernet switches count as one address FCMs count as one address.

♦

3. No more than 250 managed switches are allowed. 4. There should be no more than seven switches in the path between any two devices in The MESH control network. 5. Routers are not allowed in The MESH control network. 6. Switch-to-Switch connections (uplink ports) must be made using 1 Gb uplink ports to allow enough bandwidth for network traffic of I/A Series equipment. 7. Core switches must be only LDP-supported switches, as described in the Appendix table “Qualified Switch Standard/Security Enhanced Configuration Compatibilities Matrix” in B0700CA. 8. The root switch must have the lowest IP address in the network, or at least an address lower than the backup root switch. 9. Only two configured root bridges are allowed in the network. 10. Do not use an application on the network which requires IGMP, such as IP video. 11. Horizontal switch interlinks (links between switches of the same tier) are not allowed, except between the root and backup root switches. (See Figure 2-1). 12. The Circuit Breaker Policy (CBP) should be deployed at all uplinks ports facing the outer edge (away from the root), ports that are not and will not be blocked by spanning tree. 13. The Beacon Priority Policy (BPP) should be deployed at all uplink ports facing towards the root. 14. The Backplane Circuit Breaker Policy (BPCB) (used only for E-series bridge cards P0973BS) should be used to detect RSTP failures on the E-series second and third generation blades only (defined in the Appendix table “Qualified Switch Standard/Security Enhanced Configuration Compatibilities Matrix” in B0700CA). It is not recommended that any other switches be interfaced to the second and third generation blades via uplink or data ports. Blades protected with the BPCB policy (i.e. Eseries second and third generation blades) should reside at the outer most edge of the network. A detailed description of BPCB and other policies can be found in the appendix section “Deploying Loop Detection Policies” in the B0700CA document. 15.  All edge switch uplink ports facing the root require the spanning tree admin path cost to be increased (≥200000), this ensures efficient port blocking at the edge. 16. Methods for active network monitoring (e.g. NetSight Policy Manager) are not required but are recommended for medium to large networks. 17. The MESH control network must have redundant uplink connections between the root and backup root switches. 18. The MESH control network will support up to six VLANs, one (1) of which must be reserved for I/A Series system devices (VLAN 2 “I/A Control Ports”). a. Only one I/A Series control system can be supported on The MESH control net work.

31

B0700AZ – Rev L

2. Site Planning

b. If VLANs are deployed, all I/A Series control devices shall be attached to VLAN 2 FE ports. c. If VLANs are deployed, all switches in the network must have VL AN 2 enabled on the switch’s uplink ports. d. VLANs other than VLAN 1 and 2 shall not utilize protocols other than TCP/IP and/or UDP/IP. e. No VLAN port shall exceed 50% loading  f. No end devices with transfer (Tx) and receive (Rx) speeds greater than 100Mb are supported.

32

2. Site Planning

B0700AZ – Rev L

Interlink flows across the network, i.e. on the same tier, this is only recommended at the root tier with Circuit Breaker PBQ and SBQ deployed.

Uplink paths should be gigabit paths

Interlinks should flow towards or away from root, i.e. from one tier to the other.

Note: On N-series switches, the two interlinks between the root and backup should be on separate blades. Figure 2-1. Large Network (Security Enhanced Configuration)

33

B0700AZ – Rev L

2. Site Planning

Additional Guidelines for Planning Security Enhanced Configuration The design of each instance of The MESH control network is different, and depends on the needs and requirements of the individual site. This section provides the information necessary to help network designers to plan The MESH control network that meets the needs of their specific site. The MESH Control Network with LDP (also referred to as the Security Enhanced Configuration) isolates storming switches, and allows communications to continue on The MESH. In a properly constructed network, all switches on the network are able to communicate with each other with a high degree of reliability and redundancy. The MESH control network is constructed so that there are redundant links that form physical loops in the network which are controlled (Blocked) by Rapid Spanning Tree Protocol (RSTP), creating a logical loop-free network. In the event of a failure of this function the loop detection algorithm provides a backup to this type of failure. There are rules that must be followed when connecting switches to form The MESH control net work. The following section explains the dynamics affecting the physical topology and the traffic considerations of The MESH control network.

Security Enhanced Configuration Design Considerations  Once the general plan for the Security Enhanced Configuration has been outlined, the physical design of the network can be defined. Refer to B0700CA for the rules concerning network configuration/topology and list the specifications for the network devices (switches, converters, interface modules) and cabling, and to choose the equipment that is needed to implement the network plan. NOTE

For a variety of questions to help you define both the Standard and Security Enhanced Configurations, refer to “Network Considerations” on page 23.  When the physical design has been finalized, make a drawing or map of the network topology and save it. The map should be updated whenever a physical change is made to The MESH control network. Spanning Tree Behavior  A Spanning Tree design must be implemented correctly to provide a loop-free network during normal operations. When LDP is deployed, improper configuration can cause “false” triggers of the policy which will disable active ports/switches.

Users should ensure that the edge switches keep their redundant connections blocked. The path cost typically involves the sum of the path costs, that is, of the links that are traversed. This cost can be adjusted by altering the default assigned path costs for individual links. An edge switch may not block its ports in an expected order, though other switches may do so. Edge switches must have their path cost administratively increased to get their local ports to block. This keeps the “Beacon” packet from being forwarded to a port with the “Circuit Breaker” policy. Increasing a path cost on a switch port, increases its likeliness to block all local ports on edge devices in a MESH topology. This path cost is dependent on the switch interconnections between the switch tiers. The edge switch uplink ports path cost must be increased to ≥200000. Ports designated as the “Beacon” ports, must have their path cost set from default value of 0 to a value ≥400000. 34

2. Site Planning

B0700AZ – Rev L

The admin path cost is automatically adjusted when utilizing the I/A Series Switch Configurator  Application Software (Rev 1.1.4 or greater) discussed in The MESH Control Network Operation, and Switch Installation and Configuration Guide  (B0700CA). However, if other methods are used, refer to the switch configuration documentation provided with the switch to perform this function. IGMP “Beacon” Behavior LDP functions utilize IGMP packets which act as a beacon. The IGMP “Beacon” is enabled only on the root and backup root switches in the network when deploying LDP. Only the designated root switch with the lowest IP address will send out a “Beacon” query packet. The Secondary “Beacon” queryer (SBQ) acts as a backup and only sends out a “Beacon” query packet if the root fails. The root switch must be configured with the lower IP address of the root and backup root pair in order for LDP to operate properly. The switch with the lowest IP address and with its IGMP “Beacon” enabled will be chosen as the Primary Beacon Queryer (PBQ). When configuring the network in SysDef or IACC System Editor, the root switch should be designated with the lowest switch IP address of the root/backup root pair.

In the event of a root switch failure the SBQ on the backup root will take over. Once the root switch has been placed back online PBQ from the root will send out an IGMP “Beacon” packet stopping the SBQ on the backup root from transmitting its Beacon packet. ! CAUTION  When using Netsight Policy Manager to manage a Security Enhanced Configuration switch, you must have Policy Manager 2.2.0 build 17 or greater installed. In the event of a root switch failure, the LDP Beacon “PBQ” will become disabled and the functions of the root switch will move to the backup root switch along with the LDP Beacon “SBQ”. Once the root switch failure has been resolved and the switch has been placed back on the network, the LDP Beacon “PBQ” will become enabled, disabling the “SBQ”. Due to this event, the redundant root switch links between the root and backup root will be viewed by LDP as a loop within the network,  which causes LDP to disable one of the links (the blocking port). To recover from this rule hit, the root switch (switch with the lowest IP address) needs to regain all root functions. Netsight Policy Manager 2.2.0 build 17 or greater has the correct software required to recover from this rule hit. This event can be prevented if VLAN 2 is deployed on the network, by moving the root switch host ports to a secure VLAN “VLAN 2” (host port moved to VLAN 2 is required for an I/A Series system) will resolve the false port hits between the two root switches.

The default timing for the IGMP protocol causes one packet to be sent every 125 seconds. In order to provide a “Beacon” packet that will allow a loop to be detected within one second the IGMP timer is set to one second for LDP. Since IGMP packets are multicast packets they are for warded out from the root to the edge switches of the network. In the event of a loop, the packet  will be forwarded back towards the root where a port with a circuit breaker policy will detect it and disable the port shutting down the loop. Normally IGMP packets are used to sustain and prune multicast flows in a network and are used for IP video. Any application which uses IGMP, such as IP video, cannot be used on a Security Enhanced MESH Control Network since it will disrupt LDP operation.

35

B0700AZ – Rev L

2. Site Planning

The MESH Control Network Design Once the general plan for The MESH control network has been outlined, the physical design of the network can be defined. The following two sections describe the rules concerning network configuration/topology and list the specifications for the network devices (switches, converters, interface modules) and cabling. Use these sections to choose the equipment that is needed to implement the plan that was derived from the answers to the questions at the beginning of this chapter. When the physical design has been finalized, make a drawing or map of the network topology and save it. The map should be updated whenever a physical change is made to The MESH control network.

I/O Network Design Rules The following rules apply to the I/O network design of The MESH control network: 1. Only Invensys-qualified switches supported for The MESH control network are permitted on the I/O network. 2. The I/O network does not support redundant ISL links between switches. Only single links between switches are allowed. 3. Routers are not allowed in the I/O network. 4. Switch-to-Switch connections (ISL uplink ports) must be made using 1 Gb uplink ports to allow enough bandwidth for network traffic of the I/A Series equipment. 100 Mbps uplinks are not supported on the I/O network. 5. Configured root bridges are allowed in the I/O network, but not required. 6. The “A” Fieldbus switches must not be interlinked to the “B” Fieldbus switches (see Figure 1-7 on page 14). The two Fieldbuses must maintain separation. 7. Only the linear topology is supported on the I/O network, with no more than seven switches in series. It is recommended to use the fewest number of switches with which you can configure the I/O network. 8. The same number of switches must be maintained on both the “A” side and the “B” side of the I/O network. NOTE

 A dedicated I/O network switch can be configured with the Switch Configurator  Application Software (SCAS), as discussed in The MESH Control Network Opera-  tion, and Switch Installation and Configuration Guide   (B0700CA).

NOTE

It is recommended that one port on each managed Ethernet switch be reserved for testing and diagnostic purposes. No device should be connected to this port.

The MESH Control Network Topologies  A key feature of The MESH control network is that single points of failure will not prevent communications between all the devices in the network. This is accomplished by using a MESH net work design in which each I/A Series station is ideally connected to two different Ethernet 36

2. Site Planning

B0700AZ – Rev L

switches. Each of the Ethernet switches is then connected to two other Ethernet switches. This design provides redundant data paths so that the failure of a single device doesn’t cause operational network problems. Security Enhanced Configurations provide the loop detection algorithm known as LDP to protect against RSTP and/or data loop (storm) failures. When deployed on a Security Enhanced Configuration, LDP allows for the isolation of a defective switch or port (depending on the type of failure) during a network storm when caused by a spanning tree failure or a data looping failure. This function will prune the defective loops, maintaining network communications between devices during a network storm. NOTE

If the network is constructed with less than two connections between devices, data traffic could be disrupted if any single device fails. The MESH control network can be constructed using two network configuration/topology methods. The first is the Standard Configuration as described in this section under “Standard Configurations” below and the second is the Security Enhanced Configuration as described in “Security Enhanced Configurations” on page 52. The topology chosen depends largely on the size, complexity, site requirements, and network specifications of The MESH control network. The following paragraphs provide some guidelines for choosing a network configuration (topology). NOTE

 All graphics of switches and media converters below are intended as generic illustrations of networking concepts and do not necessarily reflect the currently offered products.

Standard Configurations  There are five basic standard configurations that are supported by The MESH control network.  As well, the variations of these topologies listed below can be made as long as all design rules are met. Be aware that some restrictions to these variations listed below may be warranted. The basic standard configurations and their approved variations are as follows: ♦

Linear

♦ ♦

Ring  Star (Variation: Double Star)

♦

Inverted tree

♦

Modified inverted tree (Variation: Inverted Stepped Tier Tree)

Standard Small Networks  A small network, consisting of two switches, could be configured as in Figure 2-2.

37

B0700AZ – Rev L

2. Site Planning

Gigabit Ethernet over Fiber Uplink Ports P92

Fiber Managed Switches

FCP270 Figure 2-2. Small Network (Standard Configuration)

In this example, two fiber optic Ethernet switches are used and the workstation and controller have a 100 Mb connection to each of the switches. The connections between the switches can use the switch’s high speed uplink ports (1 Gb). NOTE

Two connections between switches are required for proper redundancy. Larger Chassis switches can be used instead of the non-Chassis Fiber switches. Larger Chassis switches will allow hundreds of workstations, controllers and FCMs to be connected to The MESH control network. This provides a small network but quite a large control system. Multiple FCP270/FCM100s/ZCP270s, ATS modules and workstations can be connected to a redundant switch.

38

2. Site Planning

B0700AZ – Rev L

Standard Medium Networks  A medium-sized network of three to seven switches can be configured as a ring. Figure 2-3 shows an example of a ring topology containing six switches. In this example, all the switches are fiber optic and each switch is connected to the two switches adjacent to it. If one of the switches should fail, the ring is broken but RSTP (802.1w) reroutes data around the break and the network remains operable. Typically a failover time of less than 1 second is achievable using switches qualified and supplied by Invensys and configured in accordance with I/A Series documentation. NOTE

The I/A Series system imposes a seven switch limit on the length of any data path. This means that there can be no more than seven switches between any two devices in the network.

Primary Root

Backup Root

Fiber Managed Switches Gigabit Ethernet over Fiber Uplink Ports

Figure 2-3. Medium Network (Ring) (Standard Configuration)

 A medium and large-size network can also be configured in a star topology as shown in Figure 2-4. The star topology is the preferred topology for control systems. In star topology, two switches make up the backbone of the network and should be configured as the root and backup root switches. The other switches “edge switches” in the network are connected to both root switches. If desired, other devices (workstations or controllers) can be connected directly to the root switches.  A medium network Star Configuration has a limitation of 40 switches on the edges when Gold Series blades are used and 166 when Platinum Series Blades are used. This limit is derived from the maximum number of Gb ports on the Chassis switch. Two ports are used for interconnection between the primary and backup root.

39

B0700AZ – Rev L

2. Site Planning

Expandable Chassis Switch Fiber Managed Switches

Primary Root

Fiber Managed Switches

Backup Root Gigabit Ethernet over Fiber Uplink Ports Figure 2-4. Medium or Large Network (Star) (Standard Configuration)

The Double Star Configuration (a variation of the Star Configuration, shown in Figure 2-5) allows the user to benefit from the Star Configuration advantages while allowing the user to deploy a much larger sized network. Bandwidth considerations between the Star topology switches must be observed, to minimize bandwidth consumption. The majority of peer-to-peer end device communications should reside local to the individual Star topologies.

40

2. Site Planning

B0700AZ – Rev L

Expandable Chassis Switches N-Series Primary Root

N-Series Secondary Root

Gigabit Ethernet over Fiber Uplink Ports

Tier 1

A-, I- or V-Series Switches

A-, I- or V-Series Switches Tier 2 C-Series Switches

A-, I- or V-Series Switches

A-, I- or V-Series Switches

A-, I- or V-Series Switches

Tier 3

Figure 2-5. Medium or Large Network (Double Star) (Standard Configuration)

Standard Large Networks Large networks should be designed to use an inverted tree topology or a modified inverted tree topology.

For the inverted tree topology, Ethernet switches are arranged in tiers, with primary and backup root switches in the top tier. Each switch is connected to two different switches in the tier above it and end stations may be connected to any of the switches. A large network containing 12 switches in four tiers is illustrated in Figure 2-6. NOTE

There is a limit of four tiers on both inverted tree topology or a modified inverted tree topology. This is to prevent the path between any two devices from containing more than seven switches.

41

B0700AZ – Rev L

2. Site Planning

Uplink paths should be gigabit paths

Figure 2-6. Large Network - All Blades (Standard Configuration)

42

2. Site Planning

B0700AZ – Rev L

The Inverted Stepped Tier tree topology, shown in Figure 2-7, is an example which demonstrates that users can utilize a non-symmetrical tree topology. In this topology, switches connected between every other tier cannot be separated by more than 100 meters due to the distance restrictions on the copper uplinks. Every other tier separation can be greater than the 100 meter restriction, by using fiber Mini-GBICs. Separation greater than the 100 meter restriction can be between any Tier, but EMI noise should be taken in consideration  when using copper uplink ports. In the example in Figure 2-7, the uplinks between tiers 2 and 3 are not required to be interfaced to the switch directly above each. However, all switches in the network must be interfaced with two (2) uplinks to the tier above them.  When using I-series switches at the out edge, fiber Mini-GBICs must be used to interface the switches into The MESH control network. If 100Mb uplinks are used instead of 1Gb uplinks, bandwidth consumption must be minimized between the two tiers. To minimize bandwidth consumption as much as possible, the end device peer-to-peer communications should reside local to the switch to which they are connected.

43

B0700AZ – Rev L

2. Site Planning

Expandable Chassis Switches N-Series Primary Root

N-Series Secondary Root

Gigabit Ethernet over Fiber Uplink Ports

Tier 1

A-Series Switches

A-, I- or V-Series Switches Tier 2 A-Series Switches

Tier 3 A-Series Switches

A-, I- or V-Series Switches

Tier 4 A-, I- or V-Series Switches Figure 2-7. Large Network - Inverted Stepped Tier Tree - All Blades (Standard Configuration)

! CAUTION In this topology, switches connected between every other tier cannot be separated by more than 100 meters due to the distance restrictions on the copper uplinks. Every other tier separation can be greater than the 100 meter restriction by using fiber Mini-GBICs. Separation greater than the 100 meter restriction can be between any tier, but EMI noise should be taken in consideration when using copper uplink ports.

44

2. Site Planning

B0700AZ – Rev L

The modified inverted tree topology is similar to the inverted tree topology but with considerations to cost restraints. This topology utilizes low end, low cost switches as edge devices. In this topology, the switches are arranged in tiers, with the root switches in the top tier and up to three tiers below them (maximum of four tiers). The two root switches are connected to each other and the other switches each have a connection to two of the switches in the tiers above and below them. Low-end low-cost switches must be added to the outer edge at different tiers. Redundant data paths allow the network to continue to operate if any one component fails.  A large network containing 10 switches in four tiers is illustrated in Figure 2-8. There can be no more than four-tiers of switches (including the root) in order to comply with the I/A Series system requirement, limiting the number of switches between devices to seven.

Figure 2-8. Large Network - Blades and Low-Cost Switches (Standard Configuration)

45

B0700AZ – Rev L

2. Site Planning

 When designing the physical layout of a large network, the following guidelines apply: ♦

♦

Switch-to-Switch connections (uplink ports) should be made using 1 Gb uplink ports to allow enough bandwidth for network traffic of I/A Series equipment There should be a primary and backup root switch on the network 

♦

There should be no more than four tiers of switches, including the primary and backup root switches

♦

There should be no horizontal connections between switches on the same tier except the root and backup. ♦

This minimizes the number of switch interconnections in order to facilitate faster network respawning of the network tree.

♦

This also reduces the likelihood of a loop occurring.

♦

Each tier should contain an even number of switches

♦

Each switch should be connected to two different switches in the tier above it

There should be two connections between the primary root and the backup root. Example: ♦

If it has been determined that the number of workstations, controllers and FCMs that must be connected requires a network containing ten switches, the total number of switches needed will be 12 (ten switches plus primary and backup root switches). Refer to Figure 2-6 for a diagram of the finished network.

46

2. Site Planning

B0700AZ – Rev L

Step 1: Determine Physical Structure The primary and backup root switches will be in the top tier, and the remaining ten switches will be divided between the other three tiers. To determine the number of switches in the three tiers, divide the ten switches by three. The result is three (with one left over), which suggests that there should be approximately three switches per tier. Keeping in mind that there should be an even number of switches in each tier, switches are added or subtracted until this guideline is met. In this case we arrive at a solution that has two switches in the second tier, and four switches in each of the remaining two tiers.

Primary

Backup

Root Switches

Switch

Switch

2nd Tier

Switch

Switch

3rd Tier

Switch

Switch

Switch

Switch

4th Tier

Switch

Switch

Switch

Switch

Figure 2-9. The MESH Control Network Tiers

47

B0700AZ – Rev L

2. Site Planning

Step 2: Connect Root Switches Two connections are made between the primary and backup root switches. These connections should be from gigabit uplink port to gigabit uplink port. Switch-to-Switch connections (uplink ports) should be made using 1 Gb uplink ports to allow enough bandwidth for network traffic of I/A Series equipment. Connecting a gigabit uplink port to a gigabit uplink port increases the speed of the response. Primary

Backup

Root Switches

Switch

Switch

2nd Tier

Switch

Switch

3rd Tier

Switch

Switch

Switch

Switch

4th Tier

Switch

Switch

Switch

Switch

Figure 2-10. Root Switches Connected

48

2. Site Planning

B0700AZ – Rev L

Step 3: Connect Second Tier  Once the primary and backup root switches have been connected, separate connections should be made from each second tier switch to both the primary and backup root switches.

Primary

Backup

Root Switches

Switch

Switch

2nd Tier

Switch

Switch

3rd Tier

Switch

Switch

Switch

Switch

4th Tier

Switch

Switch

Switch

Switch

Figure 2-11. Second Tier Connections

49

B0700AZ – Rev L

2. Site Planning

Step 4: Connect Third Tier  Each switch on the third tier should have separate connections to two different switches in the tier above it. In this case there are only two switches in the tier above, so a connection is made to each of them. Primary

Backup

Root Switches

Switch

Switch

2nd Tier

Switch

Switch

3rd Tier

Switch

Switch

Switch

Switch

4th Tier

Switch

Switch

Switch

Switch

Figure 2-12. Third Tier Connections

Step 5: Connect Fourth Tier  The final step is to connect each switch in the forth tier to two different switches in the tier above it. Connections may be made to any two switches in the tier, but consideration should be made  with respect to keeping a manageable traffic load on each switch. Primary

Backup

Root Switches

Switch

Switch

2nd Tier

Switch

Switch

3rd Tier

Switch

Switch

Switch

Switch

4th Tier

Switch

Switch

Switch

Switch

Figure 2-13. Fourth Tier Connections

50

2. Site Planning

B0700AZ – Rev L

Step 6: Label the Switches Switches and cables should be labeled to aid in organizing the network connections. The strategy illustrated here is one method. This example uses the ‘A’ and ‘B’ labels as an aid in migrating from two separate nodebus networks to The MESH control network. Numeric superscripts are added to designate the tier and the switch number within the tier. Any labeling strategy which helps to keep the network connections organized may be used. Primary Root Switches

2nd Tier

A11

Backup

B11

Switch

Switch

21 Switch A

21 Switch B

3rd Tier

31 Switch A

31 Switch B

32 Switch A

32 Switch B

4th Tier

41 Switch A

41 Switch B

42 Switch A

42 Switch B

Figure 2-14. Labeling the Switches

51

B0700AZ – Rev L

2. Site Planning

Security Enhanced Configurations  There are four basic Security Enhanced Configurations that are supported by The MESH control network. These are: Linear ♦ ♦

Star (Variation: Double Star)

♦

Inverted tree

Modified inverted tree The following diagrams provide examples of the different topologies deploying the loop detection algorithm (LDP) as well as recommendations on where they might be used. All Security Enhanced Configurations require specific switch models as the root or backup root. The inverted tree requires switches which have LDP. The modified inverted tree requires all switches except the outer edge switches to have LDP. ♦

Refer to the “LDP Deployable” switches in the Appendix table “Qualified Switch Standard/Security Enhanced Configuration Compatibilities Matrix” in The MESH Control Network Operation, and Switch Installation and Configuration Guide  (B0700CA) for the applicable switch models. “Linear” Networks (Security Enhanced)  A small network, consisting of two switches, could be configured as shown in Figure 2-15.

Gigabit Ethernet over Fiber Uplink Ports P92

DFE Platinum Switches

FCP270 Figure 2-15. Small Network (Security Enhanced Configuration)

NOTE

Two connections between switches are required for proper redundancy. Larger Chassis switches can be used instead of the non-Chassis Fiber switches. Larger Chassis switches will allow hundreds of workstations, controllers and FCMs to be connected to The MESH control network. This provides a small network but quite a large control system. Multiple FCP270/FCM100s/ZCP270s, ATS modules and workstations can be connected to a redundant switch.

52

2. Site Planning

B0700AZ – Rev L

Star Topology (Security Enhanced) The star topology is the preferred topology for all networks. It is the easiest to maintain, expand, and deploy. In a star topology, the switches at the outside edge of the network have connections to each of the two root switches. The two root switches are connected to each other and all edge switches. Redundant data paths allow the network to continue to operate if any one component fails.

Figure 2-16 illustrates a star network in the Security Enhanced Configuration. In a Security Enhanced star topology, as many as 166 edge switches can be connected to the Chassis switch using 1 Gb uplinks. In this configuration, if an edge switch spanning tree protocol fails, LDP will remove the loop that was created by the failure. In the event the switch fails in a manner that causes a network flood, LDP will remove the defective switch from the network, allowing its redundant switch to take over. An inverted tree topology or modified inverted tree topology can be considered if there is a larger number of edge switches required.

Figure 2-16. Star Topology (Security Enhanced Configuration)

53

B0700AZ – Rev L

2. Site Planning

The double star topology allows the user to benefit from the star topology advantages while allowing the user to deploy a much larger sized network. Bandwidth considerations between the star topology switches must be observed. To minimize bandwidth consumption, the majority of peer-to-peer end device communications should reside local to the individual star topologies. Expandable Chassis Switches N-Series Primary Root

N-Series Secondary Root

Gigabit Ethernet over Fiber Uplink Ports

Tier 1

A-, I- or V-Series Switches

A-, I- or V-Series Switches

N-Series Chassis Switch

N-Series Chassis Switch

A-, I- or V-Series Switches

Tier 2

A-, I- or V-Series Switches Tier 3 A-, I- or V-Series Switches

Figure 2-17. Double Star Topology (Security Enhanced Configuration)

54

2. Site Planning

B0700AZ – Rev L

Inverted Tree Topology (Security Enhanced) The inverted tree topology is suited for very large networks with specific physical constraints. In this topology the switches are arranged in tiers, with the root switches in the top tier and up to three tiers below them (maximum of four tiers). The two root switches are connected to each other and the other switches each have a connection to two of the switches in the tiers above and below them. Redundant data paths allow the network to continue to operate if any one component fails. The deployment of the loop detection algorithm allows for redundant network condition monitoring by RSTP and LDP. NOTE

 When utilizing an Inverted Tree Topology in the Security Enhanced Configuration, all switches within the network must be switch types that support the loop detection algorithm.  An inverted tree network topology is illustrated in Figure 2-18. There can be no more than four-tiers of switches (including the root) in order to comply with the RSTP requirement, limiting the number of switches between devices to seven. NOTE

The Stepped-Tier Tree variation example of this topology is not shown as a Security Enhanced Configuration. However, this configuration can be utilized with the following rule heeded:  All switches in the network must be interfaced with two (2) uplinks to the tier above it, all uplinks attached to a tier above it must be attached to a switch that is “LDP deployable” and if a switch is attached to a tier below it, it too must be “LDP deployable”. Refer to the “LDP Deployable” switches in the Appendix table “Qualified Switch Standard/Security Enhanced Configuration Compatibilities Matrix” in The MESH Control Network Operation, and Switch Installation and Configuration Guide  (B0700CA) for the applicable switch models.

55

B0700AZ – Rev L

2. Site Planning

Figure 2-18. Inverted Tree Topology (Security Enhanced Configuration)

56

2. Site Planning

B0700AZ – Rev L

Modified Inverted Tree Topology (Security Enhanced) The modified inverted tree topology is similar to the inverted tree topology but with considerations to cost restraints. The modified inverted tree topology is suited for very large networks. However, this topology utilizes low end, low cost switches as edge devices. In this topology, the switches are arranged in tiers, with the root switches in the top tier and up to three tiers below them (maximum of four tiers). The two root switches are connected to ea ch other and the other switches each have a connection to two of the switches in the tiers above and below them. Lowend low-cost switches can be added to the outer edge at different tiers. Redundant data paths allow the network to continue to operate if any one component fails.

By deploying this, the loop detection algorithm allows for redundant network condition monitoring by RSTP and LDP.  An inverted tree network topology is illustrated in Figure 2-19. There can be no more than four-tiers of switches (including the root) in order to comply with the I/A Series system requirement, limiting the number of switches between devices to seven.

57

B0700AZ – Rev L

2. Site Planning

Figure 2-19. Modified Inverted Tree Topology (Security Enhanced Configuration)

I/O Network Topology Configurations  Only linear topologies are supported for use in a dedicated I/O network. The Linear I/O Network The linear I/O network can consist of a single A bus switch and a single B bus switch (two switches total) or up to seven switches interlinked together via 1Gb (ISL) uplinks per bus. Additional switches should only be used when the total FCM and ZCP port count exceeds the switch port count, or when the geographical conditions dictate the requirement for more than one switch. When using Single Mode Fiber ISLs, the distance between two switches can be up to, but not exceed, 80 km. When using Multi-Mode Fiber ISLs, the distance can be up to 2 km. No 58

2. Site Planning

B0700AZ – Rev L

crossover ISL links between the “A” and “B” buses are supported on the linear I/O network. Refer to Figure 1-7 on page 14.

Switch and Fiber Cable Budget and Loss Fiber Cable Budget Cable and Loss  The typical power budget for 100FX system of The MESH control network with 62.5/125 m fiber is 11.0 db and worst case (includes aging) is 10.0 db. Table 2-1 lists some typical and worst case power losses for fiber optic cable, splitter/combiners and patch cords. Table 2-1. Fiber Cable Power Losses

Link Element Splitter/Combiner MT-RJ Patch Cord LC, SC, and other similar patch cords Singlemode fiber cable (SMF) Multimode fiber cable (MMF) Splice loss 50/125 µM mating 62.5/125 µM loss  Aging reserve Patch Panel loss

Value 4.5 db 0.5 db 0.15 db 0.5 db/km 0.275 dB 1.0 db/km 3.5 db/km 0.25 dB 5.25 dB 1.0 dB 2.0 dB

Comments 0.75 db worst case 0.5 db worst case at 1300 nm wavelength at 1550 nm wavelength at 1300 nm wavelength at 850 nm wavelength Mechanical splice -5.0 dB mating loss and a -0.25 dB for mechanical splice 62.5/125 micron cable average

Fiber Optic Budgets  Refer to the documentation included with your Invensys qualified Ethernet switches for details of their fiber optic budgets.

The MESH Control Network Cabling The following sections provide the specifications and transmission distance capabilities for t he fiber optic, and twisted-pair cabling used in The MESH control network.

Switch and Converter Fiber Optic Cabling  Two types of fiber optic cable can be used to connect devices in The MESH control network: multimode fiber optic cable and single mode fiber optic cable. Each cable type is used in different applications and the devices that are connected must support the type of cable being used. The following sections give the supported uses and maximum transmission distances for each type of fiber optic cable.

59

B0700AZ – Rev L

2. Site Planning

Multimode Fiber Optic Cabling Multimode fiber optic cable is used to connect stations to switches, media converters to other devices, and, with the appropriate uplink module, Ethernet switches to other Ethernet switches.

Invensys recommends that the selected multimode fiber optic cabling have the following characteristics: ♦

62.5 micron core/125 micron cladding 

♦

Maximum allowable signal loss = 1 dB/km at a wavelength of 1300 nm

♦

Maximum allowable signal loss = 3.5 dB/km at a wavelength of 850 nm.

Cables with different characteristics can be used, but maximum transmission distance may be reduced. Cable requirements, such as flexibility, rodent protection, fire retardancy, and durability, depend on the particular application. Check with your cable installer/vendor for a list of application-specific cable characteristics. Multimode fiber cable offered by Invensys that meet the recommended specifications are given in Table 2-2, Table 2-3, and Table 2-4. Copper cable offered by Invensys that meet the recommended specifications are given in Table 2-5. Refer to the diagrams in the documentation included with your Invensys qualified Ethernet switches for detailed information on cabling. Table 2-2. Multimode Fiber Cables with LC Connectors

Cable P0972UN P0972VG P0972UJ P0972TN P0972TP P0972TQ P0972WX (50 micron/mode conditioning cable)

Minimum Connector  Bend Type Radius

Length 0.5 m (1.5 ft) (gray) 0.5 m (1.5 ft) (orange) 1 m (3 ft) 3 m (9.8 ft) 15 m (50 ft) 50 m (165 ft) 3 m (10 ft)

LC LC LC LC LC LC LC

2 inches

Table 2-3. Multimode Fiber Cables with MT-RJ to ST Connectors

Cable P0972KV P0972KW P0972KX

60

Length 3 m (9.8 ft) 15 m (50 ft) 50 m (165 ft)

Connector Type MT-RJ MT-RJ MT-RJ

Minimum Bend Radius 2 inches

2. Site Planning

B0700AZ – Rev L

Table 2-4. Multimode Fiber Cable with LC and SC Connectors

Cable

Length

P0972WW

3 m (9.8 ft)

Connector Type Two ceramic-type LC connectors on one end, with SC connectors on the other end

Table 2-5. Copper Cables with RJ-45 Connectors

Cable P0972UB P0971XK P0971XL P0972MR P0971XM P0971XN

Length 0.5 m (1.5 ft) 3 m (9.8 ft) 15 m (50 ft) 30 m (110 ft) 50 m (165 ft) 100 m (330 ft)

Connector Type RJ-45 RJ-45 RJ-45 RJ-45 RJ-45 RJ-45

Single Mode Fiber Optic Cabling Each switch must be equipped with an appropriate uplink module in order to use single mode fiber optic cable. Invensys recommends that the selected single mode fiber optic cabling have the following characteristics: ♦

8-10 micron core/125 micron cladding 

♦

Typical allowable signal loss = 0.5 dB/km at a wavelength of 1300 nm or 0.275 dB at a wavelength of 1550 nm.

Cables with different characteristics can be used, but maximum transmission distance might be reduced. Cable requirements, such as flexibility, rodent protection, fire retardancy, and durability, depend on the particular application. Check with your cable installer/vendor for a list of application-specific cable characteristics. Invensys does not offer single mode fiber optic cables. SMF cables are customer supplied. Transmission distances for single mode fiber optic cable are given in Table 2-6. Table 2-6. Single Mode Fiber Optic Cable - Maximum Transmission

 Application Switch to Switch

Transmission Protocol 1000Base-LX

Max. Distance 10000 m (32808 ft)

SMF cable is supplied by the customer

Invensys does offer single mode fiber optic cables for use with patch panels. The MESH control network uses cables with MT-RJ connectors which have a pair of fibers which crossover within the cable (TX-to-RX). When patch panels are used on site, if they are constructed using MT-RJ connectors, consider the number of panels in series between end devices which need to be built. For instance, if a cable run is terminated with two patch panels and also connects to end devices, and the patch panel was constructed so that the cable is straight-through, then using two crossover cables (one at each end) will not work. 61

B0700AZ – Rev L

2. Site Planning

Table 2-7. Single Mode Fiber Optic Jumper Cables

Description Single Mode Fiber (SMF) fiber optic patch cable w/ LC and SC connectors. Cable to be configured as a null modem (i.e., input/output crossed) Single Mode Fiber (SMF) fiber optic patch cable w/ LC and ST connectors. Single Mode Fiber (SMF) fiber optic patch cable w/ LC and ST connectors. Single Mode Fiber (SMF) fiber optic patch cable w/ LC and ST connectors. Single Mode Fiber (SMF) fiber optic patch cable w/ LC - LC connectors. Single Mode Fiber (SMF) fiber optic patch cable w/ LC - LC connectors. Single Mode Fiber (SMF) fiber optic patch cable w/ LC - LC connectors.

Length

Part Number  

3 m (9.8 ft)

P0973EW  

3 m (9.8 ft)

P0973EX  

15 m (50 ft)

P0973FY  

50 m (165 ft)

P0973FZ

3 m (9.8 ft)

P0973FV  

15 m (50 ft)

P0973FW  

50 m (165 ft)

P0973FX 

Connector Types for Switches Refer to the documentation included with your Invensys qualified Ethernet switches for their connector types and uplink ports.

 When patch panels are used on site, and if they are constructed using MT-RJ connectors, consider the number of panels in series between end devices which need to be built. For instance, if a cable run is terminated with two patch panels and also connects to end devices, and the patch panel was constructed so that the cable is straight-through, then using two crossover cables (one at each end) will not work. Patch Panels The MT-RJ jumper cables go straight through and the connector on the switch serves as the crossover. When patch panels are created using MT-RJ connectors, if the fiber run between two patch panels has a crossover, then the jumpers supplied by Invensys will not work. See Figure 2-20.

62

2. Site Planning

B0700AZ – Rev L

P1

P4

TX RX

RX TX

TX RX

RX TX

TX RX

RX TX

TX RX

RX TX

P4

P1

Switch

Switch Switch to Switch Connection

P1

P4

Switch

TX RX

RX TX

TX RX

RX TX

TX RX

RX TX

TX RX

RX TX

TX RX

RX TX

TX RX

RX TX

TX RX

RX TX

TX RX

RX TX

TX RX

RX TX

TX RX

RX TX

TX RX

RX TX

TX RX

RX TX

Patch

P4

P1

Patch

Switch

Switch to Patch Panel Connection Figure 2-20. Switch to Switch and Switch to Patch Panel Connections

FCP270, ZCP270, FCM100Et and FCM100E Fiber Signal Cabling  The cables listed in Table 2-8 are offered for use in making the 100 Mb MESH control network fiber optic cable connections. Whether these items are shipped to your location depends on the 100 Mb network cabling configuration specified at the time of system purchase. Fiber optic cabling connects to the Ethernet 100 Mb switches using one MT-RJ connector and connects to the splitter/combiner for the FCP270, ZCP270, or FCM100Et using two ceramic type LC fiber optic connectors. These same connectors can connect the switches directly to a FCM100E. The maximum optical insertion loss though each connector must be equal to or less than 0.5 db. For the fiber optic cable lengths, refer to Table 2-8. Figure 2-21 shows the cabling for the FCP270, ZCP270, FCM100Et, FCM100E or patch panels.

63

B0700AZ – Rev L

2. Site Planning

Table 2-8. Fiber Optic Cables

Part Number

Connector

P0972TR P0972TS P0972TT -

LC to MT-RJ

P0972 UJ P0972TN P0972TP P0972TQ P0972ZQ P0972UN

LC to LC

P0972VG

Length 3 m (9.8 ft) 15 m (49.5 ft) 50 m (165 ft) Customer supplied, over 50 m (165 ft) to 2 km (6600 ft) fiber optic cable 1.0 m (3 ft) 3 m (10 ft) 15 m (50 ft) 50 m (165 ft) 0.25 m (1.0 ft) 0.5 m (2.1 ft) 0.5 m (2.1 ft)

P0972VD P0972VE P0972VF

ST to MT-RJ

3 m (9.8 ft) 15 m (49.5 ft) 50 m (165 ft)

P0972WW

LC to SC

3 m (9.8 ft)

P0972WX P0972QP P0973EW

SC to SC LC to SC

3 m (9.8 ft) 3 m (9.8 ft) 3 m (9.8 ft)

P0973EX

LC to ST

3 m (9.8 ft)

64

Color No specified color

Gray color for  A Bus Orange color for B Bus No specified color

Use

Cable Material

Switch to Splitter/Combiner FCP270/ ZCP270/ FCM100Et or directly to FCM100E (without Splitter/Combiner)

MMF 62.5/125 micron

Interface MT-RJ devices to an ST-type patch panel Interface legacy P0972LQ (GPIM-01) to P0972WT (MGBIC-LC01)

MMF 62.5/125 micron

Interface legacy P0972LR (GPIM-09) to P0972WU (MGBIC-LC09) Interface P0972WU (MGBIC-LC09) to an ST patch panel

Mode condition MMF

SMF SMF 10 micron

2. Site Planning

B0700AZ – Rev L

     S  )   (   D    R    A    C    T    E    N    R    E    H    T    E    E   L    G    N  I     S   O     W    T

   X    T   -    D    Y  e   T    E   R    K     s    2     a   N    2   B   R   A    9   0   E   C    0   0   H   I    P   1    /   T   C    0   E   P    1

   X    T   -    D    Y  e   T    E   R    K     s    2     a   N    2   B   R   A    9   0   E   C    0   0   H   I    P   1    /   T   C    0   E   P    1

       S

   D     )    R    n    A    i    o    C    t    a    R    t    s    O    k        S        S

   r    o

   E     W    C     (    O    R    P

     s

   M    B    F

  e    i   r   e

   M    B    F

  e    i   r   e

   A    /    I

   A    /    I

 )    A    Z   2   7   9   0    P   (

   A    /    I

     s   e   *   *    E   0   0   1    M    C    F    i   r

 )    A    Z   2   7   9   0    P   (

  e

       S

   A

   A    /    I

  e    i   r   e

       S

   A    /    I

   t    E    0    0    1    M    C    F    E    O   0    T   0    1    N    H    O   M    A    I    C    6    T   F    2    C  r    9    E  o    0    P    N    N    O    C

     s

  e   *   * t    E   0   0   1    M    C    F    i   r

 )     S    G   6   2   9   0    P   (

   R    I    A    P    T    N    A    D    N    U    D    E    R

  e

       S

   B

     s

   M    B    F

     s

       S

     s   e   *   *    E   0   0   1    M    C    F    i   r

   M    C    F    E    L    G    N    I

  e

       S

   A    B

   A    /    I

     s   e   *   * t    E   0   0   1    M    C    F    i   r

 )     S    G   6   2   9   0    P   (

  e

       S

   A    B

   A    /    I

       S

   A    P    R   U    E   O    P   R    P   G    O    E    C   L    B    A    C

   H    E    C       S    L  ,    T   R    I  ,    U   L   J    O    A    Y   G    W    D    T    P  .   O   2        S    C    U        S    7       3    E    O    X    9   7    K   M    0   9    T    R    N   I   -   N    C   P   0    I   e   N    G    L   B  ,    P   )      s    P   G   U  ,    U      a   O    H    C    H   F    B   C    T    U    W        3    0   5    I    t    W    i    K   2   G    4        3   7    0        S    7    B   N    9      I    9   7   0    1   J    L   0   9    G   r    P    R    /    1   P   P   0   r   e   p   W  ,    P    L   U   T  ,   o  ,   p    A   E    Q   T   o    R   W    N   I    C    2   Y   F    O        3    I   U   7   2    7   7    T   Q   9    9    9    P   E   0    0   0    O    R   P    (    P   P        S

   H       S    C   R    T    I   O    T    A    W    C    P        S    U    X   E    N    O    F   -   N    R   e   O    G      s    H      a   C    C    T    B   J    I    0   R    W    0   T        S    1   r   /   e   M        b   W    i    F

   )    E    )    G    Y   N    A   A    R   R    G    (    (    O

   Q   N   G    Z   U   V    2   2   2    7   7   7    9   9   9    0   0   0    P   P   P

   H   L   K        S    R    E   O    R   O    M    T   W    E   N   T    H   O   E    T   C   N

   H    C       S    T   R    I    O    B    W    T    P        S    C    U    X   E    O    T    R   -   N   e   N    G      s   O    H    C      a    I    B   C    T    0   5    W    0   4        S      1   J   r   R   e   /   p   W   p   o    C

   H       S    C   R    T    I   O    T    B    W    C    P        S    E    U    X   N    O    R    F   -   N    G   e   O    H      s      a   C    C    I    B   J    T    W    0   R        S    0   T    1   M   r   /   e        b   W    i    F

   P    C    F    E    L    G    N    I

  e    i   r   e

   M    B    F

     s

  0   7   2    P    C    F

       S

     S    R    E    T    T  I  L    P   L    A    C  I    T    P    O

   M    B    F

       S

   A    /    I

  e    i   r   e

 )    Y    Z   7   1   9   0    P   (

       S

   A    /    I

   B

     S    R    E    T    T  I  L    P   L    A    C  I    T    P    O

   D    E    T    N    U    O    M   L  I   A    R    N  I   D

     s

   M    B    F

 )    Y    Z   7   1   9   0    P   (

   A

 .      s   e    h   c    t    i   w      s    )   e   e    A  .    i   r   h    t    Z   t      s    2   E     u      q   t    7   0   e   o    9   0   r   n    0   1   t    i   o    t    P    (    M  o   n   c      s   C  o   e   e   F   d   n    l   n      u   f      s   o    d   o   l   e   c   o   e      u   r   o   m  c   d   f      a   o    E   l      s   p    0   m  r    0   i   n   e   n    1    E    0   i    M   d    0       b   e    C     s   1   m    F     u   M  o   c  ,    /   o   e   C  r   e      s       b   F   t    l    t   y   e   l    i    A     a    h   p    *    *   m   T     s

       S

   R    I    A    P    T    N    A    R    E    L    O    T    T    L    U    A    F

   A

  0   7   2    P    C    F

   A

  0   7   2    P    C    F

 )    Y    Z   7   1   9   0    P   (

     S    R    E    T    T  I  L    P   L    A    C  I    T    P    O

     s

  e    i   r   e

       S

   A    /    I

   )    E    )    G    Y   N    A   A    R   R    O    G    (    (

   Q    N   G    Z    V    B    2   U    2    7   2    7    0    9   7    9    0   0   9    0    7    P    P   P    2    P    C    F    H    O    B    A    T    6    2    N    9    I    0    O    P    T    C    E    N    N      S    R    E    T    T  I  L    P    O    C   L    A    C  I    T    P    O

   E    L    B    A    C    T    E    N    R    E    H    T    E    )      u    C  ,   e    i    (    R    E    P    P    O    C    N    E    E    R    G

   E    C   P    I    T   U    P   O    R    O    R   G    E   E    L    B    I   B    F   A    C

   E    L    B    A    C    T    E    N    R    E    H    T    E    )    O    F  ,   e    i    (    R    E    B    I    F    A    T    N    E    G    A    M

 .   y    l   n   o      s   e      s   o   p   r      u   p   g   n    i    l

       b      a

     S    R    E    T    T  I  L    P   L    A    C  I    T    P    O

     S    R    E    T    T  I  L    P   L    A    C  I    T    P    O

   0    7    2    P    C    Z    0    7    2    P    C    Z

   R    E    T    T  I  L    P      S   L    A    C  I    T    P    O

   R    E    T    T  I  L    P      S   L    A    C  I    T    P    O

   E    L    0    B    7    A    2    )    P    C    E    M    )    G    C    Y   N    Z    E    A   A    R   R    O    D   E    T    O    O   L    G   (    (    N   G   N    M   B    Q    O    L   A    Z   U   V    I    2   2   2    T    L   C    7   7   7    C    U   1   :    9   9   9    0   0   0    E    N   1    P    P   P    N   =   =    N    O   *   #    C

   H    A    6    2    9    0    P

  c   r   o    f    d   e    t      a   r    t      s      u    l    l    i   e   r      a      s

  m    k    2   <  ,   m    0    5        S   >    E    L   =    E   B    D    P   A    E    C    I    U    L    C    O    P    R   L    /    P    J    G   m  m    U    R   m      5    E   T       3    0       S    1    L    5   R    B   M  =  =  =   E    A   h    t   R       S   T   M    C   i    T   T   T   O    2   2   T    R  w   2    7   7    E   F   7    9   9   9       S    B    M    I    0   0   0   U    F   M    P   P   P   C

   d   r      a   c    O    /    I   r   o      s      s

  e   c   o   r    P   :    E    T    O    N

   5    A   T   m  m  m    P   A   m  m  m    0   0   0    0    U   C        3   5   1  ,   1       3   5    1    O    X  =  =   =  =  =   =    R   T    G     K   L   R   M    N   e   B    U   X   X   M    X   X    E     s      a   2   1   1   2   1   1    L   B    7   7   7   7   7    B   0   7    9   9   9   9   9   9    A   0   0   0   0   0   0   0    C   1   P   P   P   P   P   P

Figure 2-21. FCP270, ZCP270, FCM100Et and FCM100E Signal Cabling

65

B0700AZ – Rev L

2. Site Planning

Twisted-Pair Cabling Category 5 (CAT5), shielded twisted-pair (STP) copper cable is used in The MESH control net work for connecting switches to RJ-45 ports on other devices. CAT5 cable can be used, with the appropriate uplink module, to connect copper switch ports to other copper switch ports or media converters. Transmission distances for CAT5 copper cable are given in Table 2-9. Table 2-9. CAT5 Cable - Maximum Transmission Distance

 Application

Transmission Protocol

Media converter to RJ-45 port, switch to switch, port to port

100Base-TX and 1000Base-T

Max. Distance 100 m (328 ft) for all applications

Table 2-10 lists the prefabricated CAT5 STP cables with RJ-45 connectors that are offered by Invensys. Table 2-10. Prefabricated CAT5 STP Cables with RJ-45 Connectors

Length

Connector Type

1 m (3.3 ft) 3 m (10 ft) 15 m (50 ft) 30 m (100 ft) 50 m (165 ft) 100 m (330 ft)

RJ-45 RJ-45 RJ-45 RJ-45 RJ-45 RJ-45

CAT5 Cable P0972UB P0971XK P0971XL P0972MR P0971XM P0971XN NOTE

Plenum grade jackets are supplied on cables over three meters in length that are offered by Invensys. This applies to both fiber optic and CAT5 copper cables.

Null Hub   A null hub is a very short cable that has a male RJ-45 connector on one end and a female RJ-45 connector on the other. The transmit and receive wires are reversed, so when it is connected to the end of a straight-through cable, the resulting cable system acts as a crossover cable. The null hub is used to connect switches that do not have an MDI crossover port or an auto MDI/MDI-X port. Table 2-11. Null Hub

66

Null Hub Part Number

Length

P0971PK

0.3 m (1 ft)

3. Installation (Cabling)  This chapter describes the steps necessary to install the cabling that enables The MESH control network equipment to communicate. Installation of The MESH control network consists of locating the individual components and then making connections between them using fiber optic or copper cable. The following paragraphs provide the information necessary for the cabling installation of the high performance system network. NOTE

Refer to the documentation included with your Invensys qualified Ethernet switches to install these switches and the media converters as part of The MESH control network.

Connecting The MESH Control Network Components Fiber Optic Cabling Guidelines The following guidelines should be followed when making fiber optic connections between devices in The MESH control network: ♦

Minimum bend radius - Fiber optic cable can be damaged if it is bent too sharply. Maintain a minimum bend radius of two inches when using fiber optic cable supplied by Invensys. Follow the cable manufacturers guidelines if third-party fiber optic cable is used.

♦

Cable straps - Cable straps, if used, should be designed for use with fiber optic cable. Hard plastic cable ties can damage fiber optic cable and are not recommended. Dust caps - Dust caps should remain in place to protect the polished cable ends over the cable ends and in the switches while cables are being routed. Dust caps should be installed on unused switch ports and device ports.

♦

Interconnecting Ethernet Switches Depending on which switch model has been selected as the root network switch, additional switches may be connected in one of several ways. They are summarized in Table 3-1 below. Refer to the “Ethernet Switch Interconnection Diagrams” in the documentation included with your Invensys qualified Ethernet switches for detailed cabling diagrams.

67

B0700AZ – Rev L

3. Installation (Cabling)

! CAUTION If Fast Ethernet “Device” ports are used as uplink ports between switches on an NSeries chassis switch configuration, it is very important that the switch spanning tree protocol settings be configured correctly. Failure to do so will cause system degradation during switch failover, causing excessive packet flooding possibly resulting in device failures. (Refer to The MESH Control Network Operation, and Switch Instal-  lation and Configuration Guide   (B0700CA).)

! CAUTION Do not make direct interconnection between ports of the same switch (i.e. loopback). Loopbacks can result in network failures upon a loss or failure of RSTP.

Table 3-1. Methods of Connecting Ethernet Switches

Root Switch Port Uplink port (Gigabit Ethernet) copper and fiber Fiber optic port (100Base-FX) Fiber optic port (100Base-FX) RJ-45 (10Base-T/100Base-TX) RJ-45 (10Base-T/100Base-TX)

End Switch Port   Uplink port (Gigabit Ethernet) Fiber optic port (100Base-FX) RJ-45 (100Base-TX) RJ-45 (10Base-T/100Base-TX) Fiber optic port (100Base-FX)

 Also, refer to page 61 for a list of single mode fiber optic cables for use at patch panels. NOTE

 All graphics of switches and media converters below are intended as generic illustrations of networking concepts and do not necessarily reflect the currently offered products.

NOTE

These sections pertain to both standard switches and chassis switches.

Uplink Port to Uplink Port  The Ethernet switches can be connected to each other through the use of an optional uplink port. Depending upon which uplink module is installed, the switch is able to communicate over single mode or multimode fiber optic cable, or CAT5 copper cable. Transmission via uplink ports is made using the Gigabit Ethernet protocol. The fiber optic cable uses LC type connectors and should be designed for the mode (single mode vs. multimode) of uplink module that is installed in the Ethernet switch’s interface slot. Figure 3-1 shows a connection between two Ethernet switches using their fiber optic uplink ports.

68

3. Installation (Cabling)

B0700AZ – Rev L

Fiber Switch

Fiber Switch

1000Base-SX, 1000Base-LX or 1000Base-ZX 62.5/125 micron Single/Multimode Fiber Optic Cable w/LC-type Connectors Figure 3-1. Switch-to-Switch Fiber via Uplink Port

Managed switches can be connected directly to other switches connecting each switch’s fiber optic ports with fiber optic cable. NOTE

Make sure that the transmit (TX) port of one switch is connected to the receive (RX) port of the other switch. The standard Invensys LC to LC cable ensures proper TX to RX connections. Fiber optic cable can be used to connect fiber switches to other fiber switches using each device’s 100Base-FX fiber ports. Multimode fiber cable with MT-RJ connectors is connected between a port on each switch. Figure 3-2 gives an example of a port-to-port connection via fiber optic cable. However, even though this method can be used for uplink ports it is not recommended due to the 100 Mb port's bandwidth capabilities. This method of uplinks must not be used for the Security Enhanced Configuration topology designs. Normally 1 Gb uplink connections should be used to interconnect switches in all standard or enhanced network designs. Fiber Switch

Fiber Switch

100Base-FX 62.5/125 micron Multimode Fiber Optic Cable w/LC Connectors Figure 3-2. Port-to-Port Connection via Fiber Optic Ports

69

B0700AZ – Rev L

3. Installation (Cabling)

! CAUTION If Fast Ethernet “Device” ports are used as uplink ports between switches, it is very important that the switch spanning tree edge-port protocol settings be configured correctly. Failure to do so will cause system degradation during switch failover, causing excessive packet flooding possibly resulting in device failures. (Refer to The MESH Control Network Operation, and Switch Installation and Configuration Guide  (B0700CA).)

RJ-45 Port to RJ-45 Port  Switches can be connected to other switches through an RJ-45 port on each switch. It is necessary that the first switch’s transmit signal is connected to the second switch’s receive line. The same is true for the second switch’s transmit and receive signals. However, even though this method can be used for uplink ports it is not recommended due to the 100 Mb port's bandwidth capabilities and the RJ-45 auto-negotiation characteristics. This method of uplinks is not recommended for the enhanced network topology designs. Normally 1 Gb uplink connections should be used to interconnect switches in all standard or enhanced network designs. Figure 3-3 shows a connection between a copper switches with auto MDI/MDI-X ports. Copper Switch

Copper Switch

Auto MDI/MDI-X

Auto MDI/MDI-X 100Base-TX

Figure 3-3. Connecting Switches via RJ-45 Ports

In the event that neither switch has an MDI/MDI-X port, a separate crossover cable (P0971PK  with a 1 ft cable), called a null hub, must be used between the switches.

Switch Configuration Refer to The MESH Control Network Operation, and Switch Installation and Configuration Guide  (B0700CA) and the documentation included with your Invensys qualified Ethernet switches for instructions on configuring them for The MESH control network.

70

4. Maintenance  This chapter provides information on identifying the cause of problems with The MESH control network.

The MESH Control Network Addresses  All I/A Series equipment IP addresses for The MESH control network are 151.128.81.x (where x is a number between 1 and 254 that identifies the specific Ethernet switch). IP addresses assigned to the primary port of controllers (FCP270, ZCP270), FCM100Ets, FCM100Es, and workstations are 151.128.y.z (where y is a number between 152 and 191, and z is a number between 1 and 254). The IP addresses for the alternate ports of FCM100Ets, FCM100Es or workstations are 151.128.w.z (where w is derived from y above as w = y - 128, and z is the same value as above). The IP addresses for the primary I/O controller ports of ZCP270 are 151.128.v.z (where v is derived from y above as v = y + 64, and z is the same value as above). The IP addresses for the alternate Input/Output Controller (IOC) ports of ZCP270 are 151.128.u.z (where u is derived from y above as u = y - 64, and z is the same value as above). See Table 4-1. Fault tolerant FCP270s and ZCP270s use the same IP addresses for both modules of the fault tolerant pair. Additionally, the ZCP270 requires a third and fourth IP address for its Input/Output Controller (IOC) ports. These also are derived and assigned according to Table 4-1. Table 4-1. IP Address Assignments

I/A Series Release v8.0 or later v8.0 or later v8.0 to v8.1.x v8.0 or later v8.0 or later v8.0 or later Pre-v8.0 Pre-v8.0 Pre-v8.0 1.  At

Port

IP Address

Network Monitoring Devices Primary Port Alternate Port1 Primary IOC Port Alternate IOC Port Switches Single Node Multi-Node Switches

151.128.82.1 through 151.128.82.254 151.128.152.1 through 151.128.191.254 151.128.24.1 through 151.128.63.254 151.128.216.1 through 151.128.255.254 151.128.88.1 through 151.128.127.254 151.128.81.1 through 151.128.81.254 151.128.8.65 through 151.128.8.126 151.128.16.65 through 151.128.23.254 151.128.79.1 through 151.128.79.254 or 151.128.80.1 through 151.128.80.254

v8.2, the intermediate driver was changed so that the Alternate Port IP is never used.

IP addresses for all devices on The MESH control network (workstations, controllers, managed switches, FCM100Ets and FCM100Es) are assigned by the system configurator (SysDef or IACC). For information on using the system configurator, refer to: I/A Series System Definition: A Step-by-Step Procedure (B0193WQ) or ♦ ♦

I/A Series Configuration Component (IACC) User’s Guide   (B0400BP).

71

B0700AZ – Rev L

4. Maintenance

Once the configurator has assigned the primary IP address, each Ethernet switch must be configured with its assigned IP address using the Command Line Interface (CLI) Entries or I/A Series Switch Configurator Application Software (discussed in the switch configuration section of the documentation included with your Invensys qualified Ethernet switches). One IP address is assigned to each non-Chassis Ethernet switch in The MESH control network. For the N1 Chassis switch (P0973AR), the chassis is assigned one IP address. For the N3 Chassis switch (P0973AS), the chassis is assigned one IP address for each of its three blades. For the N7 Chassis switch (P0972YE), the chassis is assigned one IP address for each of its seven blades. The system configurator (SysDef or IACC) assigns a MAC (Media Access Control) address for each FCP270, ZCP270, FCM100Et, and FCM100E on the network. The primary port MAC addresses are assigned from the range, 00006CC00000 to 00006CC03FFF. The alternate port MAC addresses are derived from the primary port MAC addresses by “ORing” the primary net work MAC address with 000000004000, yielding a range of 00006CC04000 to 00006CC07FFF. The MAC addresses for the primary I/O controller ports of ZCP270 are derived from the primary port MAC addresses by “ORing” the primary network MAC address with 000000008000, yielding a range of 00006CC08000 to 00006CC0BFFF. The MAC addresses for the alternate I/O controller ports of ZCP270 are derived from the primary port MAC addresses by “ORing” the primary network MAC address with 00000000C000, yielding a range of 00006CC0C000 to 00006CC0FFFF.  Workstations, FCP270, ZCP270, FCM100Et, FCM100E and FBMs are assigned letterbugs to identify the module by the system configurator. FBMs are not assigned IP or MAC addresses. Letterbugs and IP addresses can be changed using the system configurator

General Troubleshooting Guidelines  When a problem occurs with The MESH control network, it is usually best to take an organized approach to diagnosing the cause. A random or scattered approach generally takes longer and can make it more difficult to track down the cause of the problem by introducing additional unknowns. The following sections will help to determine the cause of a network problem.

Characterize the Problem The initial step in diagnosing network problems is to understand what kind of problem exists. Has the problem always existed or has it just started? ♦ New problems are sometimes caused by a change in The MESH control network configuration. Check to see whether there have been recent changes to The MESH control hardware, software, or configuration files. ♦

Is the problem constant or intermittent? Constant problems are often the result of a component failure or a change to The MESH control network configuration. Intermittent problems can be caused by environmental factors such as excess heat, electrical noise, poor contacts, or high attenuation.

♦

 What kind of error indication is occurring? Device communication failure - inability to communicate with one or more devices on The MESH control network. This happens when the information path has been broken or interrupted. Possible causes are damaged or misconnected cabling, a failure in a network device, or high attenuation.

72

4. Maintenance

B0700AZ – Rev L

Data Loss - incorrect data is arriving at one or more devices. The data path is intact, but the data is being corrupted along the way. This can be caused by failing network devices, environmental factors, or exceeding transmission distance limits. Error message - software has detected a problem. This can be caused by any number of things. The text of the error message indicates what kind of problem has been detected.

Determine Which Devices are Affected Identifying the devices that are affected helps to determine where the cause of the problem lies. For example, if all the unreachable devices are connected to the same Ethernet switch, then you should probably start troubleshooting at that switch.  When troubleshooting a Security Enhanced Configuration with LDP deployed, it is highly recommended that you refer to the section “Loop Detection Monitoring and Maintenance” in The MESH Control Network Operation, and Switch Installation and Configuration Guide (B0700CA) before continuing. Follow the steps below to identify which device or devices are affected. 1.  Are the problems limited to specific devices (stations, switches), or are they general in nature? 2.  Are the problems constant or are they intermittent? 3. Can the devices be reached using the ping command? 4. Does the ping -s command report that packets are being dropped? 5.  What does System Management Displays indicate? Refer to “System Management Displays” on page 74. 6. Use a network topology map to determine the physical location of the affected devices.

Troubleshoot the Affected Devices 1. Check activity and status indicators on the affected devices. 2. Verify that the devices are getting power (check Status LEDs). 3. Check that the devices are correctly cabled and that the cables are not damaged (swap a suspect cable with a known good cable). 4. Verify that the limit on distance between devices has not been exceeded. Distance is dependent on transmission mode and cable type. 5. If fiber optic cable is being used, verify that the optical budget (total signal loss - dB), bandwidth, and dispersion are within specification. 6. Verify that traffic rates to the affected devices are not excessive. 7. Check System Management error counters. Refer to System Management Displays  (B0193JC). 8. Verify that the devices are properly configured. Also check that the adjacent device in the network is correctly configured as well.

73

B0700AZ – Rev L

4. Maintenance

NOTE

Before removing a switch from the network, be sure to record its existing configuration. This information is necessary when the replacement switch is configured.

NOTE

Make sure a replacement switch is correctly configured before adding it to the net work. Any uplink ports should be installed before the switch is configured. Refer to the documentation included with your Invensys qualified Ethernet switches for more information.

System Management Displays The I/A Series System Management Display Handler (SMDH) obtains current and historical information about the system, and displays this information in System Management displays. SMDH uses SNMP to access the switches to obtain status information from each switch. With regard to The MESH control network and associated Ethernet switches, SMDH provides the following displays: ♦

 A Switched Network View of all the configured switches

♦

 A Switch Domain display that shows all The MESH control network stations in the selected Switch’s domain.

♦

 A Switch Ports Display which shows each port of the switch that was selected on the Domains display Detailed equipment information (EQUIP INFO) and equipment change (EQUIP CHG) displays for each port of the selected Ethernet switch.

♦

Accessing SMDH Switch Network Displays System Management Displays are accessed from the FoxView™ application. To access System Management on Windows® based systems or Solaris® based systems with the FoxView application, refer to System Management Displays   (B0193JC). The initial System Management Display, System Monitor Domains, appears. From this display you can select a system monitor and navigate through the Domain Display(s).  All switches are shown in the system monitor domain as stations designated by their letterbug. To access displays for The MESH control network switches, select (click-on) the menu bar soft keys indicated in Figure 4-1.

74

4. Maintenance

B0700AZ – Rev L

System Monitor Domains Domains Click Switched Network Display (Figure 4-2) Click

Click Click Click Click

Switch Domain Display (Figure 4-6) Click

Click Click Click Click

Switch Ports Display (Figure 4-7) Click

Click Click

Equipment Change Display (Figure 4-3) Equipment Information Display (Figure 4-4) Equipment Configuration Display (Figure 4-5) Equipment Change Display (Figure 4-8) Equipment Information Display (Figure 4-9)

Figure 4-1. Accessing SMDH Switched Network Displays

75

B0700AZ – Rev L

4. Maintenance

Switched Network Display  A typical Switched Network display is shown in Figure 4-2. The display is accessed from the System Monitor Domains display by clicking on the  soft key. It displays the letterbug “system name” of every switch in the network. Note that each card (blade) in an E7 Chassis switch is assigned a letterbug, where the N7 Chassis switch is only assigned one letterbug. Media converters and extenders are not assigned letterbugs. From the Switched Network Display you can select a switch (click on the letterbug of a particular switch of interest) and then click on the EQUIP CHG, EQUIP INFO, CONFIG INFO, or NEXT LEVEL. Clicking NEXT LEVEL brings up the Switch Domain Display.

Figure 4-2. SMDH Switched Network Display

76

4. Maintenance

B0700AZ – Rev L

Switch Equipment Change Display   A typical Switch Equipment Change display is shown in Figure 4-3. The display is accessed from the Switched Network display or the Switch Domain display. The Switch Equipment Change display enables/disables station (switch) alarms and reports from being propagated to the I/A Series system. Table 4-2 describes only the available text fields in the order that they appear on the display pages, from left column to right column.

Figure 4-3. Switch Equipment Change Display

Table 4-2. Switch Equipment Change Display Actions

 Action

Description Enables device status to affect the overall system status. Inhibits device status to affect the overall system status. Enables reports between the device and the I/A Series System Management. Disables reports between the device and the I/A Series System Management.

77

B0700AZ – Rev L

4. Maintenance

Switch Equipment Information Display   A typical Switch Equipment Information display is shown in Figure 4-4. The display is accessed from the Switched Network display or the Switch Domain display. It shows the name, type, and status of the switch. Table 4-3 describes only the available text fields in the order that they appear on the display pages, from left column to right column.

Figure 4-4. Switch Equipment Information Display

Table 4-3. Switch Equipment Information Display Fields

Field

Description Name (letterbug) of the selected station. On-line or Off-line (default) is displayed.  Acknowledged (default) or Not Acknowledged is displayed. If the  value changes from Not Failed to Failed, the value changes to Not Acknowledged to indicate this transition, and remains until you acknowledge the switch failure.  Acknowledged (default) or Not Acknowledged is displayed. If any of the switch ports fail and become unacknowledged, Not Acknowledged is displayed. The MAC Address of the station. The number of ports for a particular switch: Sw w/16 ports, Sw w/24 ports, Sw w/48 ports. For the N1, N3, N7 Chassis switches, displays “Ethernet Switch.” The ports for all switches are displayed on the Switch Ports Display.

78

4. Maintenance

B0700AZ – Rev L

Table 4-3. Switch Equipment Information Display Fields (Continued)

Field FAIL STATE ALARMING STATE

MT REPORT STATE

Description Fail or Not Failed is displayed.  Alarming State indicates whether alarming is enabled or inhibited for the switch. When alarming is inhibited, the System Monitor continues to indicate overall system and network health (a green ).  Yes is displayed if one or more switch ports has failed; otherwise, No is displayed. Master Timekeeper Report State. GPS Not Configured is always displayed for the switch equipment information display.

79

B0700AZ – Rev L

4. Maintenance

Switch Configuration Information Display   A typical Switch Configuration Information display is shown in Figure 4-5. The display is accessed from the Domains display. Table 4-4 describes the available text fields in the order that they appear on the display pages, from left column to right column.

Figure 4-5. Switch Configuration Information Display

Table 4-4. Switch Configuration Information Display Fields

Field

SW CONNECTION

SMON AP NAME BOOT HOST

80

Description Name (letterbug) of selected station. System Monitor name for the selected station. IP Address for the selected station. Lists all of the switches to which this switch’s ports are connected such as; SW001B, SW003A, SW004B. The number of ports for a particular switch: Sw w/16 ports, Sw w/24 ports, Sw w/48 ports, up to a Sw w/672 ports. The name (letterbug) of workstation that is hosting SMON. Name (letterbug) of boot host for the switch.

4. Maintenance

B0700AZ – Rev L

Switch Domain Display The Switch Domain display for a switch is shown in Figure 4-6. The display is similar for all switches. From the Switch Domain Display you can select a switch (click on the letterbug of the switch) and then click on the EQUIP CHG, EQUIP INFO, CONFIG INFO, or NEXT LEVEL. The EQUIP CHG, EQUIP INFO, CONFIG INFO displays are the same as those that can be selected from the Switched Network Display. Clicking NEXT LEVEL brings up the Switch Switch Ports Display.

Figure 4-6. Switch Domain Display

The SWITCH DOMAIN DISPLAY indicates the status of The MESH control network Ethernet cables to/from the modules connected to the switch. If a fault in cable A or cable B is detected, the faulty Ethernet cable is identified by the mnemonic RA (Receive A), RB (Receive B), or RAB (Receive A and Receive B) appearing next to the receive cable as shown below:

81

B0700AZ – Rev L

4. Maintenance

Switch Ports Display  A typical Switch Ports Display for a 16-port switch is shown in Figure 4-7. The Switch Ports Display, displays a maximum of 30-ports on one screen. Paging is enabled so that more than 30-ports can be shown on the display. The carrot < next to the port number in Figure 4-7 indicates that the port alarming has been inhibited (see “Switch Port Equipment Change Display” on page 83). All unused ports must be inhibited, disabling switch alarming for that port. This stops false alarms from propagating upward in the I/A Series System Management. The asterisk “*” adjacent to the port number in Figure 4-7 indicates that the port has failed “Lost Link” and should be investigated to determine the cause of the port failure. Refer to“General Troubleshooting Guidelines” on page 72 to aid in the isolation of the failure.

Figure 4-7. Switch Ports Display - Typical

82

4. Maintenance

B0700AZ – Rev L

Switch Port Equipment Change Display  The Equipment Change display for a switch is shown in Figure 4-8. The display is similar for all switch ports. Table 4-5 describes the available text fields in the order that they appear on the display pages, from left column to right column.

Figure 4-8. Switch Equipment Change Display

Table 4-5. Switch Equipment Change Actions

 Action

Description Enables switch alarms to propagate upward in the I/A Series System Management. Inhibits switch alarms from propagating upward in the I/A Series System Management. All non-used Ethernet ports must be inhibited for proper device monitoring.

83

B0700AZ – Rev L

4. Maintenance

Switch Port Equipment Information Display   An Equipment Information display for a port of an Ethernet switch is shown inFigure 4-9. The display is similar for all switches. Table 4-6 describes the available text fields in the order that they appear on the display pages, from left column to right column.

Figure 4-9. Switch Port Equipment Information Display - Typical

Table 4-6. Switch Port Equipment Information Display Fields

Field

NUMBER CURRENT STATE TYPE

84

Description The port number of the switch displayed in SMDH as P01 through P378. On-line or Off-line (default) is displayed.  Acknowledged (default) or Not Acknowledged is displayed. If the  value changes from Not Failed to Failed, the value changes to Not Acknowledged to indicate this transition, and remains until you acknowledge the port failure.  Yes is displayed if the device has a non-fatal error condition; otherwise, No is displayed. The port number 1 through 64. Port running or port failed. The switch port or the uplink of the switch. Failed is displayed if one or more devices connected to the switch has failed; otherwise, Not Failed is displayed.

4. Maintenance

B0700AZ – Rev L

Table 4-6. Switch Port Equipment Information Display Fields (Continued)

Field

Description  Alarming State indicates whether alarming is enabled or inhibited for the device. When alarming is inhibited, the System Monitor continues to indicate overall system and network health (a green ) while equipment is Failed or Off-line. OK or Fail. Port Enabled or Port Disabled. Interface state indicates the communications state of the port. Ethernetcsmacd or software Loopback are normally displayed for The MESH control network. Other states that can be displayed are: other, regular1822, hdh1822, ddn-25, rfc877-x25, iso88023-csmacd, iso88024-tokenbus, iso88025-tokenRing, iso88026-man, starLan, proteon-10 Mb, proteon-80Mb, hyperchannel, fddi, lapb, sdlc, dsl, el, basic ISDN, proppointToPointSerial, ppp, eon, ethernet-3Mb, nsip, slip, ultra, ds3, sip, or fram-relay. These other states can be shown depending on the type of switch and communications employed.

Indicators Ethernet Switches For the location and meaning of the indicators on a particular Ethernet switch, refer to the documentation provided with the switch.

Media Converter For the location and meaning of the indicators on a particular media converter, refer to the documentation provided with the device.

Fiber Optic Cable Handling and Cleaning Fiber optics communication relies on a clear path for its signals. Make every effort to install splices, connectors, and terminations as clean as possible, thus reducing their effects on optical data transmission.

Handling Fiber Optic Cable Consult the cable specifications for the cable you are installing. Mishandling the cable may cause damage that can alter its transmission characteristics requiring replacement of the cable. When handling fiber optic cabling:

85

B0700AZ – Rev L

4. Maintenance

! WARNING Never look directly into the end of a fiber cable or bulkhead adapter. Eye damage may result. Laser light can damage your eyes. Laser light is invisible. Viewing it directly does not cause pain. The iris of the eye will not close involuntarily as when viewing a bright light. Consequently, serious damage to the retina of the eye is possible. Never look into the end of a fiber or at a connector, or a connector in an inspection microscope which may have a laser coupled to it. Should accidental eye exposure to laser light be suspected, arrange for an eye examination immediately. ♦ ♦

♦

Turn off power to the equipment before cleaning or looking at fiber cable ends.  Always use dust covers on the end of the fiber cable connector (see Figure 4-10, Figure 4-11 and Figure 4-12) or any bulkhead (including unused bulkheads) Do not pull or kink the cable as the glass strand in the middle might become damaged or broken

♦

Do not walk-on, step-on, or crush the cable as the glass strand in the middle might become damaged or broken

♦

Keep bend radii no less than two inches

♦

Use specialized optical cable raceways and plenums whenever available Never use tie wraps as you would with electrical cables

♦ ♦

♦

 When using optical connectors, insert or remove the ferrule straight into the sleeve. Minimize wiggling the connection as this may loosen the tight fit For SC connectors, orient the prominent key on the connector body with the slot in the bulkhead adapter. Push the connector until it clicks. To remove, pinch the connector body between your thumb and finger, and gently pull straight out (see Figure 4-10) Release Tabs Dust Covers

Figure 4-10. SC Connector, Typical ♦

86

For MT-RJ connectors, orient the connector body with the slot in the bulkhead adapter. Push the connector until it clicks. To remove, push the release tab between your thumb and finger, and gently pull straight out (see Figure 4-11)

4. Maintenance

B0700AZ – Rev L

Release Tab

Dust Cover

Figure 4-11. Multimode MT-RJ Connector ♦

For LC connectors, orient the connector body with the slot in the bulkhead adapter. Push the connector until it clicks. To remove, push the release tab between your thumb and finger, and gently pull straight out (see Figure 4-12). Release Tab Dust Covers

Figure 4-12. Multimode Duplex LC Connector

Cleaning Fiber Optic Cable Contamination of Fiber Optic Connectors and Sockets  Fiber optic cable connector parts are particularly sensitive to contamination which reduces the light transfer from one side of the interface to the other. Finger oils, dust, fuzz, and so forth, can attenuate the photon transfer across the cable connector to socket interface. Experience has shown that minute quantities of contaminants can have major effects on the communication between controllers and workstations. The first line of defense is to prevent contamination from building up on the sensitive fiber optic surfaces. The second approach is to remove the contamination once it has occurred. Prevention of contamination is easier and cheaper than cleaning up after the fact.

87

B0700AZ – Rev L

4. Maintenance

Contamination Prevention  The easiest way to prevent contamination from affecting fiber optic sockets is to keep the protective covers (see Figure 4-10, Figure 4-11 and Figure 4-12) on all sockets which are not currently in use. You should retain these dust covers (rubber plugs) in a place where they can be retrieved or maintain a ready supply of covers so that they may be inserted into sockets when cables are removed. All fiber optic switches, control modules (for example, FCP270 and ZCP270), and interface modules (for example FCM100Et, FCM100E, Fiber NIC, ATS) come with protective plugs installed in each socket. Figure 4-13 shows an example of microscopic lint which has lodged in a fiber optic LC socket. The contaminant in this example totally prevented the port from functioning. This type of contamination similarly affects MT-RJ and SC connectors, especially when static charges on the connectors attract charged particles.

Figure 4-13. Lint in Fiber Optic LC Socket

Cleanliness is equally necessary with FO cable connectors’ plugs. Finger oils as well as dust and lint can negatively affect the cable plug’s transmission abilities. As with the sockets, the easiest defense is to keep the dust covers, which come shipped on the connectors, in place until just before insertion into a socket.

Contamination Removal  In spite of following the best contamination prevention procedures, you will eventually encounter a “dirty” connection, in which case you must clean the connection. There are basically three methods to remove contamination from FO connections; blowing, chemical washing, and abrasion. The blowing method consists of removing dust or lint from a connector with a blast of dusting gas such as Tetrafluoroethane, usually from an aerosol can. This is effective for removing material in depressions (such as in a socket) which is loosely coupled to the surface, for example, as by static electricity. Because dust can be transferred into connector sockets through the normal insertion of cable plugs, it is good practice to dust off the connector before plugging it in. Chemical washing basically consists of flushing or wiping the face of a fiber with isopropyl alcohol. The alcohol can be wiped on the fiber face by applying the fluid to a lint-free wipe or lint-free swab, or by using pre-saturated wipes. These alcohol saturated wipes are most useful in cleaning the connectors on the ends of cables where the exposed fibers are most easily accessible. The alcohol saturated swabs are needed for the female sockets in which the fiber is located in deep recesses.

88

4. Maintenance

B0700AZ – Rev L

The abrasion technique is useful for cleaning cable connectors such as MT-RJ style; it is not used to clean sockets. It is a dry method which does not require isopropyl alcohol. It is generally available in the form of a fabric tape housed in a dispenser which exposes a short length of tape over  which you wipe the end of the connector. The fabric tape acts as a fine abrasive which removes contamination from the glass fiber end. Examples of cleaning products are listed below. It is suggested that the user of fiber optic cables keep on hand similar items so you can quickly get back on-line when a contamination situation arises. It is also highly recommended that you observe the storage and use suggestions provided by the vendor of these cleaning products since some are classified as hazardous material. Obtain and use the Material Safety Data Sheets (MSDS) and U.S. Department of Transportation DOT-E 10232 literature associated with these products as handling guides.

Examples of Fiber Optic Connector Cleaning Products  The following products are available through various distributors: ♦

Techspray Fiber Optic Cleaning Kit, Part number 1602 (see: www.techspray.com) Contains: Isopropyl alcohol, alcohol saturated wipes, aerosol duster, swabs, and wipes.

♦

Techspray Aerosol Duster, Part number 1671-15S (see: www.techspray.com) Contains an Aerosol duster (can of compressed Tetrafluoroethane gas).

♦

♦ ♦

Cletop S Cassette cleaner, P/N 14110611 (see: www.cletop.com)  An abrasive-type cable connector cleaner. Cletop 1.25 mm Q-tip (swab) stick cleaner, P/N 14100401(see: www.cletop.com) Corning Cable Systems, P/N 2104359-01(see: www.corning.com/cablesystems)  An abrasive-type cleaning cassette.

89

B0700AZ – Rev L

90

4. Maintenance

5. Combining Two or More MESH Control Networks  This chapter provides information on combining two or more instances of The MESH control  network.

Overview  When combining two or more MESH networks, many variables must be taken into consideration. Combining MESH networks actually refers to combining I/A Series systems. When I/A Series systems are combined, a new committal must be generated which contains all the stations from the combined systems. All workstations must be recommitted and any station that receives a new letterbug, NSAP, MAC address or IP address will have to be rebooted. The following examples cover most of the basic situations and list the essential principles which must be followed. Care must be taken when combining networks in order to prevent system or network problems and even possible system shutdown. Rules: 1.  A complete inventory of all device IP, MAC and NSAP addresses on all networks must be performed to ensure no duplicate addresses are present. 2. Establish the topology of both networks obtaining the following required information. a.  Which switches are designated as the root bridge switches (Root/Back-up root) on all networks b.  What topology configuration are the networks (Star, Tree, Ring, etc.), and are they compatible with each other (with the consideration of the step c below)? c.  When the two networks become, as one, will any two switch communications exceed the maximum limit of seven hops? d. If VLANs are deployed on one or more networks do the VLAN assignments match between the networks? e. If Loop Detection Policy (LDP) is deployed on one or more networks, how will they function between each other when considering the rule requirements for LDP? 3.  When combining two or more “live” networks, the process can be dangerous. It is very important that all network and device components attached to the network be 100% functional and the functioning status of all switches must be known (root, back-up root, uplink port status, configuration and location). The following is the process by which two or more networks are combined.

Planning Stage One must assume if two or more networks are being combined that there is more than one I/A Series system in operation. To combine two or more I/A Series systems, the systems in whole 91

B0700AZ – Rev L

5. Combining Two or More MESH Control Networks

must be re-committed. When re-committing the systems, the Bridge switch functionality must be considered - see below.

Station Addressing 1. If duplicate IP, MAC and NSAP addresses on I/A Series devices or stations exist  within the networks, then those stations or devices will have to have new addresses assigned. 2. Stations and devices with new addressing will have to be marked PCHANG and rebooted after the whole system has been committed. 3. If duplicate IP addresses exist on a network switch, the switch must be re-configured  with a new IP address and the switch host workstation must be re-committed to acknowledge the new IP address of the switch.

Bridge Switch Functionality 1. Establish which pair of switches will be the root and t he backup root of the new combined network. a. The root and backup root should be the switch within the network that has the largest throughput capability. b. The root and back-up root should have the lowest set of IP addresses among the network switches. (This is not mandatory but it is highly recommended.) c.  Any former root/backup switches will need the priority of their Bridge IDs changed.

Topology Constants  When merging two or more networks, consider the physical layout of the topology. For example, if a four-layer Tree topology is to be combined with a Star topology, the Tree cannot be added to the Star. However, the Star can be added to the Tree but only at the root or layer one switches.  Adding the Tree to the Star results in a five-layer Tree, exceeding the limit of seven switches (hops) between any two devices on the network. Adding the Star to the Tree's root or at the layer two switches only results in a broader Tree which does not exceed the seven hop limit. There are an unlimited number of ways in which network topologies can be combined. However, the rules established in this document must be followed. Because of the unlimited ways of combining networks, take each case with its own considerations. Although this document cannot cover every possibility, the following are a few methods in which networks should be combined.

92

5. Combining Two or More MESH Control Networks

B0700AZ – Rev L

Combining Star Network Topologies In Figure 5-1 below, A1 and B1 are root switches with the A2 and B2 as backup root switches  within their respective Star topology networks. To combine the two Star networks with minimum impact to the physical layout of switch and cabling, the two Star networks are combined to form a three layered Tree topology, shown in Figure 5-2 below. A1

Star

A2

B1

Star

B2

Figure 5-1. Combining Two Star Network Topologies Into One Tree Network Topology (Before)

A1

B1

A2

B2

Figure 5-2. Combining Two Star Network Topologies Into One Tree Network Topology (After)

To perform this, switch B2 must be taken offline, removed from the B network and configured as an edge switch in the A network. (Refer to The MESH Control Network Operation, and Switch Installation and Configuration Guide  (B0700CA) for the definition of an edge switch.) It is assumed that A1 and B1 are the root bridge switches within their respective networks and A2 and B2 are the back-up root bridges within their respective networks. To do this, the following actions are required: 1. Verify that the A1 and A2 switches have a Bridge Priority value at least two (2) less than the B1 switch. If they do not, reconfigure the A1 and A2 switches with a lower priority value. 2. Power down the B2 switch and physically remove it from the B network, removing all cables attached to this switch.

93

B0700AZ – Rev L

5. Combining Two or More MESH Control Networks

3. The B2 switch must be re-configured as an edge switch in the A network. (Refer to The MESH Control Network Operation, and Switch Installation and Configuration Guide  (B0700CA) for configuration information.) 4.  With the B2 switch powered down, physically connect the B2 switch to the A1 and  A2 switches via uplink (Trunk) cables one for each A root switch. The A root switch uplink (Trunk) port must be configured to accept the new cables (if previously disabled or if 100Mb ports are to be used). 5. Remove the previously cross-linked cables between the B1 and B2 switches from the B1 switch. Reconnect the B network edge switches to the B2 switch. 6. Power up the B2 switch. When this is completed, the B1 switch will re-span, releasing its root bridge function to the A1 switch. All devices on the B network have now been migrated to the A network. 7. Power down the B1 switch and physically remove it from what is now t he A network, removing all cables attached to this switch. 8. The B1 switch must now be re-configured as an edge switch in the A network. (Refer to The MESH Control Network Operation, and Switch Installation and Configuration Guide  (B0700CA) for configuration information.) NOTE

If the two networks are to be separated for maintenance at some point, insert a lower bridge priority value (i.e. 32000) for this B1 switch. At the time of separation, the switch will re-establish its root functions. The same can be done for the B2 switch as well, but with a higher bridge priority value (i.e. 32001). 9. The B1 switch must be added to the A network via uplink (Trunk) cables - one for each A root switch. The A root switch uplink (Trunk) ports must be configured to accept the new cables (if previously disabled or if 100Mb ports are to be used). 10. Make the physical connections to the A root switches utilizing the old cross-linked root connections to the B2 switch. 11. Reconnect the B1 switch to the former B network edge switches. 12. Power up the B1 switch.  You have just converted two Star network topologies into one Tree network topology. If two Star network topologies are to be combined and it is desired that a Star topology be maintained (see Figure 5-7 on page 97), you must also apply the process of eliminating the seconded pair of root switches, discussed in “Combining Tree Network Topologies” on page 96.

94

5. Combining Two or More MESH Control Networks

B0700AZ – Rev L

Combining Star with Tree Network Topology The process for combining a Star network Topology with a Tree network Topology is the same as for the previous procedure, “Combining Star Network Topologies” on page 93. See Figure 5-3 and Figure 5-4 below. Tree A1

A2

B1

Star

B2

Figure 5-3. Combining a Star Network Topology with a Tree Network Topology Into One Tree Network Topology (Before)

A1

A2

B2 B1

Figure 5-4. Combining a Star Network Topology with a Tree Network Topology Into One Tree Network Topology (After)

95

B0700AZ – Rev L

5. Combining Two or More MESH Control Networks

Combining Tree Network Topologies The process for combining two Tree network topologies is the same as for the above two Star net work procedures. Combining two Tree network topologies is possible if one or more of the topologies has less than four tiers (layers). If two or more Tree network topologies has four tiers, then this combination process is not possible. A tier (layer) must be removed from one of the topologies, as only one topology can have four tiers. This will cause an increase in effort and a higher risk in having a network failure. Tree A1

Tree A2

B1

B2

Figure 5-5. Combining Two Tree Network Topologies Into One Tree Network Topology (Before)

A1

A2

B2 B1

Figure 5-6. Combining Two Tree Network Topologies Into One Tree Network Topology (After)

For two four-tier topologies, the root and back-up root switches of the second network must be removed (eliminated) and the second tier switches of this network must be connected to the root and back-up root of the first topology (see Figure 5-7).

96

5. Combining Two or More MESH Control Networks

A1

A2

B0700AZ – Rev L

B1

B2

Figure 5-7. Combining Two (4) Tier Tree Network Topologies Into One (4) Tier Tree Network Topology (After)

Combining Ring Network Topologies Ring topologies are not recommend to be combined since the seven hop limit restricts the size of the network topology to a maximum of seven switches in the network Ring topology. When combining Ring topologies, it is recommended that the Ring topology migrate to the Star or Tree topology that best suites the physical layout of the site requirements. Because of the potential excessive amount of re-cabling, it is recommended that a plant shutdown be considered. Even combining a Ring with a Star or Tree topology, it is recommended that at least the Ring topology network side be shut down.  A Ring topology can migrate to a Star topology before combining the networks without requiring a plant shutdown. To migrate from a Ring topology to a Star topology, first determine which switches are the root and backup switches. If a root and backup are not configured yet, they should be configured first. A Star topology can be created by manipulating the uplink “Trunk” cables as shown in Figure 5-8 below. If there are no extra 1Gb uplink ports, the 100Mb uplink ports of the B1/B2 switches can be used. Ideally, if two rings are to be combined, the final root and backup root switches should be replaced with C2 Series or N-Series switches so that all uplinks utilize 1Gb connections. By using Figure 5-8 below and Figure 5-7 above (which combines two stars), two rings could be converted into stars and then combined.

97

B0700AZ – Rev L

5. Combining Two or More MESH Control Networks

B1

B1

B2

Root and Backup

B2 Move cables one at a time

B1

B2

Added cables Figure 5-8. Combining Ring Network Topologies

98

Appendix A. COMEX Fault Handling on The MESH Control Network  This chapter provides information on Communication Executive (COMEX) fault handling in The MESH control network.

COMEX Layers COMEX Applications Layer This layer provides connectionless and exposed LLC communications services. Connectionless messages are contained in unit data  packets. There are 2 types of connectionless messages - acknowledged  and unacknowledged . For acknowl-  edged  communications, there is a 2-way acknowledgement (handshake), where the receiver of the unit data packet replies with an ack1 packet to the initial sender, which then responds with an ack2 packet. No replies (acks) are returned for unacknowledged   message types. However, these message packets are sent twice, first out the primary  port, and then 1 second later, out the secondary port. The Invensys multicasts are unacknowledged  connectionless messages. The Invensys “Broadcast” is actually one of the Invensys Multicasts. Cable selection is based on an array in COMEX. The number of transmits is used as an index into this table to select the cable. Seven (7) transmission attempts (Versatile Real-Time Executive (VRTX) devices) are allowed, with the number of transmits starting at 1. Therefore, the first entry in the array (index 0) is a filler and not used. The remaining entries indicate the cable selection as follows: Primary, Primary, Secondary, Secondary, Primary, Primary, and Secondary. On the third transmission attempt, a cable test is requested before attempting to send the message. When the number of transmits exceeds the maximum (7 for VRTX devices), the message is flushed. For the I/A Series MESH and 7.x Control Network stations (Windows XP ®  workstations and Nucleus Plus devices), this maximum is also 7.

Application Layer Protocol and Timers   A transmitted acknowledged connectionless Unit Data packet requires an Ack1 response from the receiver. Upon receiving the Ack1 response, the initiator responds with an Ack2 packet.  A transmitted unacknowledged connectionless Unit Data packet is directed to the Primary cable, and requires no response. Rather, after waiting 1 second, the packet is re-directed to the Secondary cable. (Since the Data Link Layer of the I/A Series MESH and 7.x Control Network systems determines the active communications port, the above cable directions are ignored.) A reference delay is used to time the receipt of Ack1's and Ack2's and the duplicate transmission of unacknowledged packets. This time delays are as follows: round trip time delay - 1.01 seconds ♦ 99

B0700AZ – Rev L

♦

Appendix A. COMEX Fault Handling on The MESH Control

This is a basic delay period for stations on the same Node, for which some other delays are based. For stations on another Node on the same LAN, 1 is added to this value. For stations on another LAN on the same Site, 2 is added to this value. Note that the Site in the comms hierarchy was never implemented. acknowledged UD packet transmitted (retransmit time-out) - 1 second time-out for receipt of Ack1 response Failure to receive response within the time-out period results in the UD packet being re-transmitted, up to the limit of retries. This value is subject to ongoing optimization in The MESH network devices, and may change without notice.

♦

 Ack1 response sent - Time-out period is taken from the original acknowledged UD packet (its retransmit time-out) and modified, for receiving the Ack2 response. The time-out period is multiplied by the maximum number of connectionless retries allowed and 1 added. Then, the round trip delay is added to the time - 1.01 seconds for same node, 2.01 seconds for same LAN, and 3.01 seconds for same site. Therefore, the reference delay is between 9.01 and 11.01 seconds, for VRTX stations. (Because, the site hierarchy part of the communications system was never implemented, the same LAN would be the furthest round trip delay - 10.01 seconds.) Failure to receive the  Ack2 response is the same as if the Ack2 was received - resources are released.

♦

unacknowledged UD packet transmitted - 1 second time-out. Timer expiration results in sending duplicate unacknowledged UD packet on other cable. This is always the case, since no response is expected that would stop the timer.

♦

unacknowledged UD packet received The time-out period is taken from the original unacknowledged UD packet and multiplied by 2. Then, the round trip delay is added to the time - 1.01 seconds for same node, 2.01 seconds for same LAN, and 3.01 seconds for same site. Therefore, the reference delay is between 3.01 and 5.01 seconds. (Because, the site hierarchy part of the communications system was never implemented, the same LAN would be the furthest round trip delay - 4.01 seconds.) When the duplicate packet is received, or with timer expiration, receive resources are released.

Transport Layer This layer provides connected communications services. The protocol of this layer is based on a reduced subset of what appears to be class 4, as defined in RFC 905, “ISO Transport Protocol Spec-  ification, ISO DP 8073 ”, dated April 1984. The message is contained in a data transfer packet (DT). Connections are set up by sending a Connection Request  packet (CR). The response is a Connection Confirm  packet (CC) to accept the connection, or a Disconnect Request  (DR) to refuse the connection. When the initiator receives an acceptance (CC packet), an Ack  (AK) packet is sent to acknowledge the connection. DT packets are acknowledged with an AK packet. In addition, if either side of the connection has not sent a packet in a certain time-out period, an AK packet is then sent as a “heartbeat” to show that the connection is still active. Disconnects are initiated with a DR packet. The disconnect is accepted by replying with a Disconnect Confirm  (DC) packet. DT packets are sequenced and use a “window” controlling the number of outstanding DT packets before an AK is received. The sequence number starts at 0 and is incremented for successive DT packets. The receiver of the DT packet responds with an AK, indicating the sequence number of the next expected packet. If a DT packet is received in error, then the AK will contain the sequence number of that packet. A receive credit value is given in the CR, CC, and AK packets, 100

Appendix A. COMEX Fault Handling on The MESH Control Network

B0700AZ – Rev L

indicating the size of the “window” for the receiver. In effect, the transmitter can have no more than the receive credit number of transmitted packets outstanding with the receiver.  When a connection is requested, (or any packet expecting an ack response) the initiator will wait the Retransmit Timer period for the response, before retransmitting the packet. When a connection is established, DT packets or heartbeats (AK's) will be sent every Window Timer  period. A DT or AK packet is expected to be received within the Inactivity Timer period, in order to maintain the connection. When a DT packet is received, the receiver will wait no longer than the AK Timer  period before transmitting a response (AK). Messages could be received in multiple data packets. This is called a multi-packet receive. The packet indicates if it is the last packet of a transmission (EOT bit of the sequence number octet). Multi-packets need to be linked into a single message before being delivered to IPC. In addition, during multi-packet receives, the receive credit get boosted above normal and an AK set to change the credits with the sender. With the last packet, the receive credit gets set back to normal, and an  AK is then sent to change the credit back to normal with the sender. (At the sender, it becomes his transmit credit.)  When a connection exists, all messages to be transmitted are placed in the Wait Ack Queue (WAKQ) to wait for their acknowledgment. Received packets (DTs or ACKs) that are out of sequence with what is expected are handled according to the following table: Table A-1. Out of Sequence DT and Ack Packets

Received Packet

Action

Retransmit Timer

Ack Timer  

 ACK seq < outstand-  Acknowledged packets ing DT pkt's last seq. removed from WAKQ, if present. Available new packets sent up to xmit credit limit.  ACK seq same as last Ignored.  ACK seq received.  ACK seq > outstand- Out of sequence ing pkts last seq + 1 counter incremented.  ACK ignored. DT seq < next Dumped. expected - 1. DT seq = next Dumped and ACK expected - 1 sent.

Restarted if any packets on WAKQ acknowledged and new ones sent. If WAKQ empty, timer stopped. No action.

-

No action.

-

-

DT seq > next expected

-

Start timer if not already running. (This packet was the last one acknowledged and timer should already be stopped.) No action.

Dumped.

-

-

 Transmissions expecting a response are attempted 8 times (7 retries), before the packet is discarded. For the I/A Series MESH and 7.x Control Network stations, this limit is 8 attempts for both Windows XP workstations and Nucleus Plus devices (7 retries) . Transmissions not expecting a response (AK and DC packets) are only sent once. The cable selection is governed by an array. 101

B0700AZ – Rev L

Appendix A. COMEX Fault Handling on The MESH Control

The number of retries is used as an index into the array. Cables in this array are selected as follows: primary, primary, secondary, secondary, primary, secondary, primary, secondary. Packets expecting a response will use this array. Packets not expecting a response (AK and DC), use another array. The result of using this array, is that the cable is swapped between the primary and secondary,  with every transmit. Because AK and DC packets use private resources that is never released (these packets are created in the TL), the transmit count will keep incrementing, so that these packets alternate cables. The retry count is for an individual message (not a response) on a given connected channel. The transmit count is only for responses being sent using the private resources. Individual messages are retried. Responses (AK, DC) are not retried, because they do not get a response.  When the retry count is 2, a cable test is requested. When the retry count exceeds the maximum, the message is flushed.

Transport Layer Protocol and Timers  The timers used by the Transport Layer are: ♦

Retransmit Timer - 0.5 seconds for VRTX devices (for the I/A Series MESH and 7.x Control Networks. this value is 0.5 seconds for Nucleus Plus devices and 0.5 seconds for Windows XP workstations) Maximum time allowed between sending a packet and receiving an AK for it. This timer automatically restarts itself. Timer expiration causes a transmission reattempt of all packets in the Wait Ack Queue, up to the retry limit. When outstanding packets are AK'ed, this timer is stopped (and packets removed from the queue).

♦

♦

♦

Inactivity Timer - 10 seconds Maximum time allowed to receive heartbeat AK's (or DT's). This is a one-shot timer. It is restarted whenever a packet is received. At timer expiration, the connection is aborted.  Window Timer - 2 seconds for VRTX devices (for the I/A Series MESH and 7.x Control Networks, this value is 2 seconds for Nucleus Plus devices and 2 seconds for  Windows XP workstations) Maximum time allowed between sending heartbeat AK's (or DT's) This timer automatically restarts itself. It is also restarted whenever an AK or DT is sent. At expiration, an AK packet is sent.  AK Timer - 75 msecs Maximum time allowed between the first DT received and transmitting its AK. This is a one-shot timer. At expiration, an AK packet is sent. Normally, when the number of received DT packets equals the limit, an AK packet is sent. In this case, the timer provides a cap on the time between received data and its  AK transmission. The limit is 1.

 When a connection is requested with a CR packet, a CC packet response is generated by the receiving end. The Retransmit Timer is used to time the response. If the timer expires before getting the response, the CR packet is resent, up to the maximum retries, before the packet and the connection are aborted. If the protocol is successful, a connected channel is created. Then the Inactivity Timer is used by both ends of the channel for receiving packets from the other end of the channel. These packets can be either DT packets or Acks (DT acknowledgements or heartbeats). If the Inactivity Timer expires, the connection is aborted. 102

Appendix A. COMEX Fault Handling on The MESH Control Network

B0700AZ – Rev L

 With a connection established, and a DT packet sent, an Ack response is expected. The Retransmit Timer is used to time the response. If the timer expires before getting the response, the DT packet is re-sent, up to the maximum retries, before the packet and the connection are aborted.  With a connection established, and a DR packet sent, a DC response is expected. The Retransmit Timer is used to time the response. If the timer expires before getting the response, the DR packet is re-sent, up to the maximum retries, before the packet and the connection are aborted.  An aborted connection causes a DR packet to be sent. The expected response and timers are the same as previously described. The difference is that instead of IPC requesting the disconnect, the request is initiated by the Transport Layer.

Network Layer The Network Layer provides routing to the Nodebus, to another Nodebus (through the LAN Interface), and back into the box (upper layers).

Operation  There are no states in the Network Layer operation, because no protocol is performed. The primary responsibility of this layer is routing the packets. Packets are routed to the local node, to the LAN Interface in VRTX devices for other nodes, and back to the upper layers in the same station. In The MESH network devices, off-node packets are routed to a specific Address Translation Station (ATS), rather than to a LAN Interface. Point-to-point packets are routed to the upper layers when the destination NSAP (site, LAN, and node ID's) is the same as the source NSAP. The destination and source are on the same station and the packet is routed to the upper layers as if it was a received packet. Multicast packets are always sent out the station. They are also routed to the upper layers of the source station, if the Reroute bit (in the packet) is set and the multicast group is active in the station.

Logical Link Control Layer The implementation of this layer is based on IEEE 802.2, class 1 (Type 1 Operation only), using only unnumbered UI commands (information). No other U commands are supported. Therefore, the poll/final bit is not used (it is not valid for UI commands).

Transmit Operation  There are no states used in an LLC transmit operation. The LLC header is simply added to the packet and it is passed on to the MAC layer.

Receive Operation  The LLC distributes received packets according to whether they are normal data packets, diagnostic packets, or exposed LLC packets. Several additional packet types are defined for The MESH network devices. The routing of connected and connectionless packets is based on the Network Layer  destination LSAP (Link Service Access Point). These packets specify network routing (Network Layer LSAP ), providing for routing by the destination LLC Layer to the destination Network Layer. For exposed LLC packets, 2 LSAPs are supported. The Application Layer 802.1 individual  LSAP is used for downloading station images, and the Application Layer 802.1 group  LSAP is used for 103

B0700AZ – Rev L

Appendix A. COMEX Fault Handling on The MESH Control

cable test messages (diagnostics). These packets are distributed directly to the Application Layer. The diagnostic packet used for a cable test is sent by NFD with a Station Manager  destination LSAP, via exposed llc (through IPC and the Application Layer). Received Station Manager packets are passed by the LLC Layer directly to SMD (station manager diagnostics), which echoes them back directly through the LLC Layer, as an Application Layer 802.1 group  destination LSAP. These are then passed by the receiver LLC Layer directly to the Application Layer (and thus, back to NFD). The MESH network devices provide for additional destination LSAPs:

NFD PROXY  for The MESH network to Nodebus NFD cable test proxy packet. ♦ MAC QUERY  for a Mac Query packet. (Other LSAPs are used by The MESH network comms drivers and ATS devices for validating net work integrity.) ♦

MAC Layer The MAC layer is designed for the individual hardware for which it interfaces.

104

Appendix B. The MESH Network Fault Handling  This chapter provides information on fault handling in The MESH control network. control  network. Fault Fault handling on The MESH network is handled on a per station basis. This error handling process is protected by a patent. This description applies to all dual port COMEX stations which have two roles for their ports, including Workstations, FCP, ZCP, and ATS Stations: ♦ ♦

an active port based on the system address for that station. a standby port that only is used for testing with a different MAC address until it is determined that the active port is not functioning correctly. correctly.

COMEX implements seven retries on the active port spaced over 3.5 seconds for all connected messages. Connectionless acknowledged messages perform seven retries over seven seconds. COMEX relies on lower level mechanisms to select the best possible port as the active port. The port selecting behavior of the stations is as follows.

LINK The first line of defense is the Link Integrity Signal. This signal is derived by the PHY Interface hardware hardware and is indicative of the presence of “IDLE” symbols received. If a port does not have “LINK”, it cannot send or receive. If the Link signal goes inactive, the port is not usable and the driver will switch the station address on the module to the other port if it does have a “LINK” signal. Typically Typically switches stop sending if they detect no activity on a port (to conserve power) so link becomes a bi-directional indication. Loss of LINK switchover will be indicated to the customer as PORT A (or B) failure in the smon_log and SMDH. Causes of loss of link: 1. Switch power down 2. Switch failure 3. Cable failure of either the transmit or receive fiber 4. Dirty cable 5. Module hardware failure

“PORT TEST” Packets If both LINK signals are active two “PORT TEST” packet are sent by the driver every 300 milliseconds, one “PORT TEST” packet is sent from the “A” port to the “B” port and one “PORT TEST” packet from the “B” to the “A”. In the body of the packet is the text “Prim Port Test” and “Alt port Test”. Reception of these packets in both directions indicate that both ports have equal connectivity to the rest of the network. As long as “LINK” is good and the “PORT TEST” TEST” packets succeed, no port switching will occur.

105

B 0 7 0 0 A Z – Re v L

Appendix B. The MESH Network Fault Handling

 A single port test packet failure failure is flagged and the Excessive Excessive Collision Collision Counter is incremented but but no switchover occurs until two “PORT TEST” in a row have experienced a failure. A “PORT TEST” failure is defined as a packet in either direction failing. If two successive “PORT “PORT TEST” tests fail due to the inability to receive both packets the module  will initiate the “PING” “PING” test. The failure of this this test indicates that there there is a fault in The MESH MESH  which prevents prevents normal communications. communications. The port test packets will will cease for 400 milliseconds milliseconds  while the “PING” test attempts to detect the best way to survive the the problem. After the llc_ping test runs the module goes back to sending port tests t ests until the network is healthy. healthy. Causes of “PORT TEST” failure: 1. Switch failure causing islanded switches 2. Message storm substantially interfering with normal communications 3. Internal switch failure of the switch connected to the “A” port or the “B” port 4. Module hardware failure 5. Broken or damaged cable

“LLC_PING” Test This test involves sending a “PING” packet to up to seven stations that have previously been in communication with this controller. controller. It is used to allow the module to select the port with the highest connectivity with its most recent COMEX partners based on a most recently received table. This IS NOT the standard IP “PING” packet instead it is a COMEX llc_ping type (DSAP = 0x0A, SSAP = 0x0A). Only COMEX stations are sent this message to guarantee that the targeted station is running the SW that t hat will reply to this t his “PING”. The packet indicates that this is a llc_ping request and is sent on both the “A” “A” port and the “B” port. When any COMEX station receives this request it responds with a LLC_PING_response packet with the same format (DSAP = 0x0A, SSAP = 0x0A) to the senders MAC address. The station who initiated the llc_ping message then tallies the responses and make a decision whether to switch ports or not. 1. IF all all res respo pons nses es are received on both cables this is a “Good” case and no switchover is performed and no messages are sent. 2. IF more more resp respon onse sess  (for example 4 responses on alternate and 2 on Primary) are received on the Alternate port a port switch is performed (no Hot-Remarry should occur on FT stations). This port switch places the Alternate as the Primary port and re-assigns the MAC addresses addresses of the ports. A broadcast is sent to update all switches  with the new MAC MAC address location. A message is sent to the alarm printer printer and the smon_log if enabled. PING Test Sent = 7 Replies = Pri 2 Alt 4 new cable = B (or A)

NOTE

 A workstation does do es not currently send s end any messages message s to the alarm printer/smon_lo prin ter/smon_log. g.  Workstatio  Workstations ns run in promiscuous mode. mode . Upon a Primary port swap, the NIC is not reprogrammed reprogrammed but the software does use the Primary port Mac address on the Alternate port when a swap occurs. 3. IF AN EQU EQUAL AL number number of of respons responses es is recei received  ved  and  and the number is less than the total sent a one-time switchover is performed. This switchover is to resolve situations with 106

Appendix B. The MESH Network Fault Handling

B0700AZ – Rev L

an equal number of local and remote stations causing islanding. It is done to cascade all stations over to a healthy path. No more switchovers on “tie” situations are allowed until successful “A” “A” to “B” port tests are detected indicating a return to healthy MESH. A message is sent to the alarm printer PING Test Sent = 7 Replies = Pri 3 Alt 3 new cable = B (or A)

4. IF more more resp respon onse sess  are received on the Primary port, no switching is performed and a “PORT FAILURE” message for the Alternate port is sent to the alarm printer.

ZCP-FCM Communications  All communications communications between the ZCP ZCP and the FCM use use raw IP packets. The ZCP ZCP sends an alternating request/reply transaction transaction on each Ethernet cable to talk to the FCM pair at all times. On discovery of any failure, the field connection has its reliability score reduced. The ZCP uses the more reliable cable exclusively while background mechanisms try to rehabilitate the failed cable. The ZCP maintains a quality of connection for each FCM separately. separately. If one FCM has a problem on channel A, that FCM will prefer to use channel B. All others would still use A or B normally. normally. There are no retries. All Ethernet, IP, and port addresses are unique to the particular network. NOTE

If the user has required re quired the ZCP to use the t he “A” “A” Ethernet cable cabl e and the “A” cable is in fact failed, the ZCP will blindly follow the requirement to use the “A” cable even though it does not work. This is done to aid debugging of problems.

NOTE

Two FCMs can have the same addresses if they are on separate networks; for example, behind different ZCPs and not on The MESH. There is no LLC3 protocol on the I/O side between the ZCP and the FCM. ZCP and FCM communication behaves as discussed below in the following cases: Case 1: 1: The ZCP fails to receive a response on the primary port to an FCM and/or switch. Behavior : The ZCP retains a health score on each of its connections. It maintains communications on the “A” or “B” bus, whichever is more healthy. However, the FCM is also trying to recover the bad cable, and is sending health packets on the bad cable. A successfully delivered packet raises the ZCP’s health score, so the bad cable’s health score may increase. Multiple Multiple successes raise the cable’s cable’s health score to completely good. When the score is equal that of the other cable, the ZCP will start to alternate between cables again for its communication to the FCM. Case 2: 2: The ZCP communicates with several FCMs and does not receive a response on a port. Behavior : The ZCP maintains a quality of connection for each FCM separately. separately. If one FCM has a problem on the “A” bus, that FCM will switch to the “B” bus. All other FCMs 107

B0700AZ – Rev L

Appendix B. The MESH Network Fault Handling

 would still use “A” or “B” normally. A message indicating this change should be generated by the ZCP on the FCM's behalf.

DIAGNOSTIC Information Diagnosing MESH network failures is complicated but here is some general information 1. PORT FAILURE MESSAGE a. Most likely caused by loss of the LINK signal. b. Most likely caused by a cable problem or dirty (very common) Fiber connections. c. Loss of link may be reported by both the switch and the station “PORT FAILURE” message and the SMDH port field. d. Port Failure is also indicated by a persistent path failure from the Primary port to the Alternate port. The Alternate port will display the port failure message. 2. FT LINK Problems a. On FT station both stations must have a common link on at least one cable to stay married. b. If the Primary loses both links it will reboot and the Shadow will become Primary. c. Modules will not marry without at least one common link. If no common link is found a module will go RED/GREEN. 3. ZCP LINK problems a. For ZCPs link failure on the Control network are handled in the same fashion as failure on the Application network. b. For ZCP I/O functionality as of 8.2 there must be a path from the “A” port of the ZCP to the “A” port of the FCM or the “B” port to “B” port in order for the ZCP to maintain comms. The ZCP cannot talk from the “A” port to the FCM “B” port MAC address. This means that cross-wired systems have multiple single points of failures as follows: ♦

ZCP port “A” to Switch 1 port 1

♦

ZCP port “B” to Switch 2 port 1

♦

FCM port “A” to Switch 2 port 2

♦

FCM port “B” to Switch 1 port 2

♦

Switch 1 port 3 to Switch 2 port 3

If this configuration was built the ZCP to FCM communications would work fine  with all hardware functional. Any of the following failures would cause loss of I/O communications: ♦

Switch 1 -power down or total failure

♦ ♦

Switch 2 -power down or total failure Switch 1 port 3 or switch 2 port 3

♦

Cable Switch 1 port 3 to Switch 2 port 3

4. SMDH counters a. MAC resets indicate either Hot-Remarries or Port problems. 108

Appendix B. The MESH Network Fault Handling

B0700AZ – Rev L

b. Excessive Collisions indicate either continuous (evenly incrementing approximately 1 per second) or intermittent MESH problems (incrementing at less than 1 per second) causing “A” to “B” failures. c.  At the present time, workstations do not indicate “A” to “B” port test failures unless a port switch occurs.

109

B0700AZ – Rev L

110

Appendix B. The MESH Network Fault Handling

Index  C  Cable fiber optic 60, 61 specifications 59 twisted-pair 66 Cabling  fiber optic 63 Cleaning  fiber optic cable 87 D  Display, Switch Configuration Information 80 Display, Switch Domain 81 Display, Switch Port Equipment Change 83 Display, Switch Port Equipment Information 84 Display, Switch Ports 82 Display, Switched Equipment Information 78 Display, Switched Network 76, 77 E  Ethernet switches connecting 67 indicators 85, 87 F  Fiber optic cable 18, 60, 61 cleaning 87 handling 85 Fiber optic cabling 63 G  Gigabit Ethernet uplink ports 68 H  Handling  Fiber optic cable 85 I  I/O network 12 design rules 36

111

B0700AZ – Rev L

illustration 14 topology configuration 58 Indicators Ethernet switches 85, 87 media converter 85 IOC (I/O Controller) 12

M  Multimode fiber. See  Fiber optic cable, multimode N  Network components connections 67 Network topology 36 Null hub 18, 66 P  Prefabricated cables 66 S  SCAS - Switch Configurator Application Software 17 Single mode fiber. See  Fiber optic cable, single mode Site planning 23 SMDH 74 Switch Configuration Information display 80 Switch Domain display 81 Switch Equipment Information display 78 Switch Port Equipment Change display 83 Switch Port Equipment Information display 84 Switch Ports display 82 Switched Network display 76, 77 System Management displays 74 T  Topology, network 36 Troubleshooting 72 Twisted-pair cable 66 U  Uplink port 68 V  VLANs 3, 23, 30

112

Index

 Index

B0700AZ – Rev L

113