Download Telecommunications Infrastructure Standard for Data Centers ANSI/TIA-942...
Telecommunications Infrastructure Standard for Data Centers ANSI/TIA-942
Chris DiMinico
[email protected] MC Communications
IEEE 802.3 HSSG
Data Center Standards
ANSI/TIA-942 Telecommunications Infrastructure Standard for Data Centers Co-chairs: Chris DiMinico & Jonathan Jew Published 2005 – available through TIA at www.tiaonline.org ANSI/NECA/BICSI-002 Data Center Design and Implementation Best Practices co-chairs: Jonathan Jew & John Kacperski best practices – complements TIA-942 – 2007 target IEEE 802.3 HSSG
Who Developed TIA-942?
Developed by the TIA TR-42.1.1 Network Distribution Nodes subcommittee as Project No. 3-0092 Participants included: Architecture & Engineering Firms Consultants End Users Manufacturers
IEEE 802.3 HSSG
Purpose of TIA-942
Encourage early participation of telecom designers in data center design process Fill a void by providing standards for planning of data centers, computer rooms, server rooms, and similar spaces. The standard encompasses much more than just telecommunications infrastructure. Close to half of the technical content deals with facility specifications.
IEEE 802.3 HSSG
Purpose of TIA-942
Define a standard telecommunications infrastructure for data centers Structured
cabling system for data centers using standardized architecture and media Accommodates a wide range of applications (LAN, WAN, SAN, channels, consoles) Accommodates current and future protocols (e.g., 10+ GbE ) Replaces unstructured point-to-point cabling that uses different cabling for different applications
IEEE 802.3 HSSG
Purpose of TIA-942
Specifications for data center telecommunications pathways and spaces Recommendations on media and distance for applications over structured cabling Establish a standard for data center tiers to replace several proprietary standards. The TIA data center tier standard is: • A tool to evaluate existing data centers • A tool to communicate design requirements IEEE 802.3 HSSG
Unstructured Cabling C
D
E
F
G
H
I
J
K
L
M
N
O
P
Q
R
S
T
U
Heater - A/C
1
TABLE 247-7589
AB
AC
AD
AE
AF
TABLE
FI L E
F IL E
CABINET
CABI NET
AI
AJ
AK
AL
AM
AN
AO
AP
AQ
F IL E CABINET
FIL E CABINET
POMONA A/C UNIT #6
AR
AS
HP PRINTER
C D ROM SCANNER
UNIT
N ETWORK CABINET E7
NETWOR K
CABI NET I1
NE TWORK CABI NET I2
NE TWORK
CABI NET I3
I5
NE TWORK C ABINET I4
NETWORK CAB INE T
NETWORK
CABINET I7
NETWOR K C ABINET I6
CAB INE T I8
DESK
NETWOR K
H8 N ETW OR K CABINET H7
N ETW OR K CABINET H6
NETWORK CABINET G7
IBM CAB
CABINET F6
N ETWOR K CABINET G6
N ETW ORK C ABINET F5
NETWORK CABINET G5
N ETW ORK C ABINET
NETWORK C ABINET E6
IBM 3745-A11
IBM 3746-900
NETWORK C ABINET
IBM
TAPE RACK
POMONA A/C UNIT #8
TAPE RACK
NETWORK CABIN ET H4
NETWORK CABIN ET G4
N ETW ORK C ABINET F3
NETWORK
NETWORK CABIN ET
N ETW ORK C ABINET F2
CABIN ET H3
NETWORK CABIN ET H2
IBM DASD
IBM S-390 R34
IBM DASD
F4
NETWORK CABIN ET H1
G3 NETWORK CABIN ET G2
LAN CABINET F1
NETWORK C ABINET E4 NETWORK C ABINET E3
NETWORK C ABINET E2
NETWORK C ABINET E1
3745-41D
IBM 3745-17A
IBM 3745-L13 SAA CABIN ET D2
N ETW OR K CABINET A6
NETW OR K CABINET
NETWORK CABINET
C6
B7
N ETW OR K
N ETWOR K
CABINET A5
N ETW ORK
E5
11
AH
Heater - A/C
POMONA A/C UNIT #7
N ETW OR K CABINET
NETWORK CABIN ET H5
8
AG
AT &T
7
10
AA
DEFINIT Y
6
9
Z
TABLE
He ater - A/C
5
Y
NETW OR K CABINET C5
CABINET B6
HALON PANEL
NETWORK CABINET A4
NETW OR K CABINET C4
N ETWOR K CABINET B5
NETW OR K CABINET C3
N ETWOR K CABINET B4
N ETW OR K CABINET
NETW OR K
N ETWOR K
A2
CABINET C2
CABINET B3
NETW OR K CABINET C1
N ETWOR K CABINET B2
N ETW OR K CABINET A3
NETWORK
SAA CABIN ET
CI SCO ROU TE R
4
X
CONS OLE
3
W
Heater - A/C
STORAGE CABINET
DESK
2
V
F/X 247-7543 HALON ABORT HALON RELEASE
C ABI NE T A1
B
PA C BELL
A
D1 N ETWOR K CABINET B1
NETWORK CABIN ET G1
12 247-7538
13 14 STORAGE C ABINE T
TAPE RACK
15
IBM AS -400
IBM AS-400
INSTALL A CABLE WHEN YOU NEED IT (SINGLE-USE, UNORGANIZED CABLING) TABLE
16
IEEE 802.3 HSSG
LOCKER
STORAGE CABINET
LIEBERT PDU -C
3174 CABINET STORAGE CABINET
3174 CABINET
3174 CABINET
3174 CABINET
3174 CABINET
3174 CABINET
Structured Cabling B
C
D
E
F
G
H
I
J
K
L
M
N
O
P
Q
R
S
T
U
V
Heater - A/C
1
TABLE 247-7589
NE TWORK C ABINET I1
NETWORK CABINET I2
CAB IN ET I4
NETWORK CABINET I3
NETWORK
NE TWORK
CABIN ET I6
NETWORK C ABI NET I5
NETWOR K CAB INE T I7
NE TWORK CABI NET I8
IDF
NETWORK C ABINET H6
IDF
N ETWOR K C ABINET H5
T APE RACK
TABLE
AE
AF
AG
FIL E
FILE CABINET
FIBER MDF COPPER & CORE MDF
POMONA A/C UNIT #8
NETWORK C ABINET E7
NETWORK CABINET G7 NETWORK CABIN ET G6 N ETWOR K C ABINET G5 NETWORK CABIN ET
NETWOR K C ABINET F6 NETWOR K C ABINET F5
NETWORK CABIN ET F4
G4
NETWORK CABIN ET F3
N ETWOR K C ABINET H3
NETWORK CABIN ET G3
NETWORK CABIN ET F2
N ETWOR K C ABINET H2
NETWORK CABIN ET G2
LAN CABINET F1
N ETWOR K C ABINET H1
NETWORK CABIN ET G1
C ABINET H4
TAPE RACK
N ETWORK CABINET E6
AH
AI
AJ
AK
AL
AM
AN
AO
CABINET
FIL E CABINET
FIL E CABINET
POMONA A/C UNIT #6
CD ROM SCANN ER
AP
AQ
AR
AS
HP PRINTER
IBM S-390 R34
IBM DASD
IDF
IDF
IBM 3745s IBM CAB
NETWOR K NETWORK
IBM 3745-A11
IBM 3746-900
C ABINET C6
NETWORK CABINET B7
NETWORK C ABINET C5
NETWORK CABINET B6
NETWORK C ABINET
NETWORK CABINET
C ABINET A6
CABINET E5 N ETWORK CABINET E4 N ETWORK CABINET E3 N ETWORK CABINET
N ETWORK CABINET E1
IBM
NETWOR K C ABINET A4
C4
B5
NETWOR K C ABINET
NETWORK
A3
C ABINET C3
NETWORK CABINET B4
NETWORK C ABINET C2
NETWORK CABINET B3
3745-41D
IBM 3745-17A
IBM 3745-L13 SAA CABINET D2
NETWORK
IDF C ABINET C1
SAA C ABINET D1
IDF
IDF
F/X 247-7543 H ALON ABORT
NETWOR K C ABINET A2
NETWORK CABINET B2 NETWORK CABINET B1
H ALON PANEL
NETWOR K C ABINET A5
N ETWORK
E2
Mainframe IBM DASD
11
AD
Heater - A/C
POMONA A/C UNIT #7
NETWORK CABINET H8 NETWORK C ABINET H7
N ETWOR K
10
AC
UNIT
7
9
AB
AT&T
6
8
AA
DEFINITY
DESK
5
Z
TABLE
Heater - A/C
4
Y
CISCO ROUTE R CON SOLE
3
X
Heater - A/C
STORAGE CABINET
DESK
2
W
H ALON RELEASE
PAC B ELL C A BIN ET A1
A
IDF
IDF IDF
12 247-7538
13 14 15
S TOR AGE C AB IN ET
TAPE RACK
IBM AS-400
IBM AS-400
STRUCTURED CABLING SYSTEM (ORGANIZED, REUSABLE, FLEXIBLE CABLING) T ABLE
16
IEEE 802.3 HSSG
STORAGE CABINET
LIEBERT PDU- C
LOC KER
3174 CABINET STORAGE CABINET
3174 CABINET
3174 CABINET
3174 CABINET
3174 CABINET
3174 CABINET
Design Elements
Cabling Design Facility Design Network Design
Informative annex’s: Provide best practices Annex A - Cabling Design Considerations Annex B- Telecommunications infrastructure administration Annex C-Access provider information Annex D- Coordination of equipment plans with other engineers Annex E- Data center space considerations Annex F- Site selection IEEE 802.3 HSSG
Design Elements
Cabling Design:
Copper and fiber cabling performance Connectors, cables, distribution hardware Cabling distances Space management
Facility Design: Data center sizing Power distribution methodologies Pathways and spaces HVAC, security, operations, and administration. Flexibility, scalability, reliability and space management
IEEE 802.3 HSSG
Design Elements
Network Design: Support
of legacy systems Enable rapid deployment of new and emerging technologies such as 10 GbE and 10+ GbE copper and fiber applications.
IEEE 802.3 HSSG
Relationship of Spaces BUILDING SITE BUILDING SHELL GENERAL OFFICE SPACE
TELECOM ROOMS & EQUIPMENT ROOMS for spaces outside data center
OFFICE BUILDING SUPPORT SPACE
DATA CENTER SUPPORT STAFF OFFICES
OPERATIONS CENTER
ENTRANCE ROOM(S)
TELECOM ROOM(S) for data center support spaces
COMPUTER ROOM
IEEE 802.3 HSSG
DATA CENTER ELECTRICAL & MECHANICAL ROOMS
STORAGE ROOMS & LOADING DOCKS
Data Center Topology Entrance Room
Entrance Facility
Carriers
Carriers
Active Equip interconnect
Offices, Operations Center, Support Rooms Secondary Entrance Room Active Equip interconnect
Active Equip interconnect
MC MC
Carriers Carriers
Back bone Cabling HC IC
HC IC
HC IC Horizontal Cabling
interconnect Active Equip Storage Area Devices
interconnect Active Equip
ICHC Zone Distribution interconnect
interconnect Active Equip Active Equip
Data Center IEEE 802.3 HSSG
TIA-942 Spaces
Entrance Room (ER) - location of interface with campus and carrier entrance facilities Main Distribution Area (MDA) – location of main cross-connect (MC) Horizontal Distribution Area (HDA) – location of horizontal cross-connect (HC) Zone Distribution Area (ZDA) – location of zone outlet (ZO) or consolidation point (CP) Equipment Distribution Area (EDA) – location of equipment cabinets and racks IEEE 802.3 HSSG
Data Center Cabling Main Distribution Area
MC MC
Backbone Cabling
Horizontal Cabling
HC IC
interconnect Active Equip
interconnect Active Equip
ICHC
Horizontal Distribution Area
interconnect
Zone Distribution Area
Active Equip
Equipment Distribution Area
Horizontal cabling is the cabling from the horizontal cross-connect (in the main distribution area or horizontal distribution area) to the outlet in the equipment distribution area or zone distribution area. IEEE 802.3 HSSG
Horizontal and Backbone Cabling
Recognized Cables: a) 100-ohm twisted-pair cable (ANSI/TIA/EIA-568B.2), category 6 recommended (ANSI/TIA/EIA568-B.2-1) b)
multimode optical fiber cable, either 62.5/125 micron or 50/125 micron (ANSI/TIA/EIA-568-B.3), 50/125 micron 850 nm laser optimized multimode fiber is recommended (ANSI/TIA-568-3-1)
c)
singlemode optical fiber cable (ANSI/TIA/EIA568-B.3)
d)
75-ohm (734 and 735 type) coaxial cable (Telcordia Technologies GR-139-CORE) IEEE 802.3 HSSG
Horizontal cabling distances
The maximum horizontal distance is 90 m independent of media type. The maximum channel distance including equipment cords is 100 m. The maximum cabling distance in a data center not containing a horizontal distribution area is:
300 m for an optical fiber channel including equipment cords. 100 m for copper cabling including equipment cords.
IEEE 802.3 HSSG
Backbone Cabling
Includes cabling from MDA to ER, HDA Optional cabling between HDAs allowed Maximum backbone cable lengths depend on applications to be supported Centralized optical fiber cabling supported with interconnect, splice, or pull-through at the HDA Star topology with no intermediate crossconnects Various topologies permit redundancy and flexibility to support various data center sizes IEEE 802.3 HSSG
Computer Room & Entrance Room Requirements
Min clear height of 2.6m/8.5 ft Min door size 1m/3ft wide 2.13/7ft high Min dist floor loading 7.2 kPA/150lbf/ft2, recommended min 12 kPA/250 lbf/ft2 20 degrees C to 25 degrees C 40% to 55% relative humidity (reduces ESD) Any sprinkler systems must be pre-action system Common bonding network (CBN) – equipotential ground reference
IEEE 802.3 HSSG
Main Distribution Area
Location of Main Cross-Connect (MC), the central point of distribution for data center structured cabling system Centrally located to avoid exceeding maximum distance restrictions (typically for E1s, E-3s, T-1s and T-3s) Install separate racks for Fiber, UTP, and coaxial cable distribution
IEEE 802.3 HSSG
Horizontal Distribution Area
Location of Horizontal Cross-Connect (HC), the distribution point for cabling to equipment distribution area Distribution LAN, SAN, KVM switches and console servers located in HDA MDA may also include an HC for nearby equipment distribution area Number of HDAs depends on the density of cabling and the size of the data center
IEEE 802.3 HSSG
Horizontal Distribution Area
The capacity of the cable tray system and the size of the cross-connect creates practical limits on the size of the HC Guideline is maximum of 2,000 4-pair UTP or coax cable terminations per HDA Arrange patch bays to minimize patch cable lengths and to simplify cable management Separate racks for fiber, UTP, and coax Locate switches and patch panels to minimize patch cord lengths
IEEE 802.3 HSSG
Zone Distribution Area
Rack, cabinet, or under floor enclosure that houses a zone outlet (ZO) or consolidation point (CP) ZO - structured cabling termination for floor-standing equipment that cannot accept patch panels (e.g. mainframes and large servers). CP - intermediate termination point (e.g. cabling to areas where floor plan is uncertain or dynamic) No cross-connects within the ZDA No active equipment shall be located in the ZDA Maximum of 144 connections in a ZDA Maximum of one ZDA within a horizontal cable run
IEEE 802.3 HSSG
Equipment Cabinets
IEEE 802.3 HSSG
Front rails of cabinets must be recessed to provide adequate room for patch cables and wire managers Recommend 1-to-1 ratio of patching to cable management Arrange switches and patch panels to minimize patching between cabinets & racks Perforated tiles at front of cabinets One edge of cabinets placed at edge of tile
Raised Floor
Better appearance than overhead cabling. Allows higher power densities, better control of cooling, and more flexibility in location of cooling equipment Most stand-alone computer systems are designed for cabling from below Coordinate under floor cabling with mechanical & electrical engineers Recommend wire basket cable trays in hot aisles for telecom cabling
IEEE 802.3 HSSG
Example of Wire Basket Cable Trays For Cabling Under Raised Floor 24" 1.375"
RAISED FLOOR TILE
DIVIDER 18" 9"
5" 4"
FIBER
COPPER CABLING
SUPPORT STRUT
36"
12"
18" 5" 4"
FIBER
COPPER CABLING
SUPPORT STRUT
Open Area 13.5"
24" On Center
IEEE 802.3 HSSG
1"
1"
Overhead Cable Trays
Less expensive than raised floor systems Cable trays can be attached to the top of racks and cabinets (if they are uniform in height) Cable trays suspended from the ceiling provides more flexibility for supporting cabinets/racks of various heights and for adding and removing cabinets/racks. Cable trays can be installed with several layers Coordinate location with lighting, ducts, overhead conduits, overhead power IEEE 802.3 HSSG distribution
Overhead Cable Tray Example 3 Layer cable tray system: Bottom layer – signal Middle layer – power Top layer – fiber Signal Reference Grid in brackets attached to lower layer of trays Fiber patch cables may be in fiber duct attached to threaded rods
IEEE 802.3 HSSG
Infrastructure Administration
Informative annex with TIA-606-A standards compliant labeling scheme for all components. Labeling scheme extended for use in data centers Cabinets and racks labeled by location using tile grid or row/position identifiers All cabinets, racks, patch panels, cables, and patch cords should be labeled IEEE 802.3 HSSG
Labeling Example AA
AB AC AD
AE
AF
AG AH
AI
AJ
AK AL AM AN AO AP AQ AR AS
AT
AU AV AW AX AY
"X" COORDINATE AZ BA
BB BC BD
01 02
Cat 6 x 24
03 04 05
RIGHT FRONT CORNER
06 07
CABINET AQ03
CABINET AJ05
08 09 10 11 12 13
"Y" COORDINATE
Patch Panel for 24 Cat 6 from 1st Panel in Cab AJ05 to 2nd Panel in Cab AQ03 AJ05-A to AQ03-B Ports 01-24 01
02
03
04
05
06
07
08
09
10
11
12
13
14
A
IEEE 802.3 HSSG
15
16
17
18
19
20
21
22
23
24
A
Site Selection
Informative annex with guidelines for selection of a site for a data center Architectural Electrical Mechanical Telecommunications Security Other
IEEE 802.3 HSSG
Facilities Specifications & Tiers
Informative annex with general architectural, structural, electrical, mechanical, and telecommunications recommendations requirements Annex includes detailed architectural, security, electrical, mechanical, and telecommunications recommendations for each Tier Recommended specifications by tier are a uniform way to rate aspects of a data center design and are a starting point for initiating design requirements with qualified architects and engineers
IEEE 802.3 HSSG
Data Center Tiers
Tier 1 – basic data center no redundancy Tier 2 – redundant components Single distribution path with redundant components Tier 3 – concurrently maintainable Multiple distribution paths with only one active Tier 4 – fault tolerant Multiple active distribution paths IEEE 802.3 HSSG
Redundant Topologies Carriers
Carriers
Carriers
Secondary Customer Maintenance Hole (Tier 2 and higher)
Primary Customer Maintenance Hole (Tier 1 and higher) r3
Tier
2
Primary Entrance Room (Tier 1 and higher) Tie
Secondary Entrance Room (Tier 3 and higher)
r4
Tie
Main Dist Area (Tier 1 and higher)
Telecom Room
Tier 3
Tier 4
Tier 1
Offices, Operations Center, Support Rooms
Tier 3
Tier 1
Data Center
Tie
Carriers
r3
Tier 4
Secondary Dist Area (Optional for Tier 4)
Computer Room Horiz Dist Area Optional Cabling
Horiz Dist Area
Horiz Dist Area
Equip Dist Area
Equip Dist Area
Zone Dist Area
Equip Dist Area
IEEE 802.3 HSSG
Data Center Tiers
Higher tiers correspond to higher availability, but also have higher construction costs Data Center can have different tier ratings for different portions of its infrastructure (architectural, security, mechanical, electrical, telecommunications) The overall rating for the data center is equal to the lowest tier rating Capacity of systems may need to be upgraded to maintain tier rating as data center load increases Human error and operating procedures have a major impact on availability
IEEE 802.3 HSSG
Conclusion
TIA-942 is the first standard to specifically addresses data center infrastructure. Primarily a telecom infrastructure standard, but about half of the content deals with facility requirements. Provides a flexible and manageable structured cabling system using standard media. Builds on existing standards, when applicable Guidelines on a wide range of subjects useful to someone designing or managing a data center. An official tiering standard for determining the quality of a center. A way to objectively compare one center with another.
IEEE 802.3 HSSG
