DC110 Data Center Design and Best Practices Version 1.4 Student Guide September 2010

Dat:a Cent:er Best:

Version Student: Cuide

•

•

DC110 Data Center Design Best Practices

o Copyrlgh l 201 0 by BICSI

Printed in the United States of America

Bicsi"

Version 1.4, First Prin ting

All righlS re\!'"rwd. No pan of this workbook may be used, reproduced, or uan'>min ed in any form or by any means, electronic or mechanical. including photocopymg, recording. or by any information storage and retrieval system, without prior agreement and "'fltten per mission from t he pu blisher, except for the inclusion of brief quotat ions i n a review. Thecoments of th is wo rkbook are subject to revision without nol ice due to continued progress in met hodology, design, and manufacturing. THE GUIDE IS PART OF TH E BleSI DESIGN COURSE CURRICULUM AND IS PRESENTED H 1\ WITHOUT WARRANTY OF ANY KIND, RESPECTING THE CONTENTS OF THIS GUIDE INCLUDING BUT NOT UMITED TO IMPUED WARRANTIES FOR THE GUIDE'S QUALITY, PERFORMANCE, MERCHANTABILITY, OR FITNESS FOR ANY PARnCULAR PURPOSE. BI CSI SHAll NOT BE LIABLE TO THE PURCHASER OR ANY OTHER ENTITY WITH RESPECT TO ANY LIABILITY, lOSS, OR DAMAGE CAUSED DIRECTlY OR INDIRECTLY BY TH IS GUIDE. All brand names, trademarks, and registered trademarkS are th e property of their respective holders. BleSI lX'odd He.ld ,"" ~ - ' t .··~ mind

-

f

~ l' '''

"""

"1.1-0",

EI ",,~

~~

IV,'"..

....'''.

1'I c:.c.o:

-

- - - - -

I I

-

--I I I

lon:~" olI,. NC1'.I.oro

- - -

-

-

;>'

--

- -

H"",,~vr, I !orI~'

I

-

L

JiIl.!'''''~ 1

~~~r~L=I.l.;lln=

-

-I I

IIr.d: "~I

- -- ---

- -

t ".crf~~ ~ · Aor",,~. ~

... ; ...

-

J

~~~ '"

.. » I r)~ , . 5.~~,.:t!

wcu"'" 1

p~ntion

• Data Center risk.analysis addresses three key design fealllres: Operational reqUirements-the opportunity to suspend operations lor maintenance Availability reqUirements-the targeted uptime of the system during operations

Impact of downtime-the Impact of unplanned dis,upHons

RIsl< Analysis & the High Nines of Reliability

2.21

Wh~e customers and clients may seek "6 nines" In terms of reliability. risk analysis I~ the process of balancing that desire against the high costs assodated with Insuring limited downtime

O2IItO

.1CSi'

Reliability 99%

as hours

Visruption TIme per Year

99.9% 99.99% 99 .999% 99.9999%

8.8 hoors S3 Minutes 5.3 Minutes 32 seconds

Data ~ & The D.,.lgn Proceso

student Guide

2.28

Phase II: Problem DefInition

rusk-analysis estabflshes a target I'l!4labllity level for a facility while problem definition concerns tne task of meeting this ta'llet In terms of: • Facility space

• rr assets • Project locatk>n • Proj ect budget The end result of the problem definition phase of design i. the creation of the fBdflty program

2.29

Phase III: Solution Development

The third stage of Oata Center design Is the development of one (occasionally two) design solution(.) to be submitted to the dient for approval The design solution phase usually involves: • Planning • Space Programming

• Cost Estimation • Const",Clion Documentation SoluUon Development: Planning

2.30

• Planning typically Involves coordination between technologlcol consultants and architects regarding: -

Space

-

Power

-

·Cooling

-

Security

-

Fioer leading

-

Grounding

-

Eledrical protection

-

Telecommunication pathways

"J."". 'Ql \ .. '1.'\

."

a!csIP~~NotcQFIY

StucMnt Guld.

0$ C."tar& & Tho DesIgn Pro:esl

Solution Development: Spac:e Programming

2.31

Space Progrclfnming requires estimating data center requirements at rull capacity f1Jr:

• IT hardware &. ",cks • Electrical Equipment • HVAC equipment

• Space for personnel amI airllow • TelecQmmunications equipment and cabling

• Furu re growth Solution Development: Cost Estimation

2.32-33

Cost estimation involves COflsideration of one-time, recurrent, and Intangible costs:

• One time costs: -

Real estate costs

-

Local tax

-

Cost to bring utilities to Site

-

Demolition and site prepitrittion costs

-

Cost of temporary 5e/Vlces to support IT migration

-

Cost of equipment relocation

-

ConSillrant costs

Incentlve~

• Recurrent 'Costs -

Utility usage costs (power, water, sewer, gas)

-

Telecommunication services

-

Wages

-

Lease costs

-

Taxes

• Intangible Costs -

Availability of alternate or mu ltiple telecommunications access providers

-

Availability of utility se/Vices

-

Availability ot transportation

H.

811:11 f'rupri.w.y p~ Mot Copy

ON Centers &Thll o.sign P'tocMI

Student 200

Level 2

>0

Level 3

0

Level 4

DeIIcrlpilon

Where tunctions are operational 24 "vu,~ a day. 7 days a week, for 5.(,·';; Rt~""\' Horizontal Distributions Areas)

• Server and equipment racks/cabinets (lf1e Equipment Distribution Area)

• Digital storage I SAN (Storage Area Network) frames • Special ceoling equipment (Computer Room Alr Conditioning units) • Cabling pathways • Fire prnrection equlpment

It will sometimes c:ontaln:

,

• Power distribution units (POU) .- "", • Remote power panels (RPP)

Physical Network Topologies The"" are two basic possibilities for a physica' IT networ1mmercial building spaces : • The ma in distMbutioo area (MDA) In a data center Is the equivalent of a commercial building equipment room (ER) • The horizont.11 distribution area (HDA) in a dilt.1 ceoter is the e"uivillent of a ~ommercla l building TR or TE • The zone distribution area (ZOA) In a dat.1 center Is the equ ivalent of a commercial building consolidation point (CP) • The equ ipment distribution area (EDA) in a datil center Is the equivalent of a commercia l buildin9 work area (WA)

Standard Building Topology

~G'AJ! TEHJ~ f.tWA1-' [kp --\

"-

j~ I

~"

fWA ~ EF

CI2O'I'OBICSI"

TR

,,

.

BO .

I TR_I _

c. ,

BockOOM

co, ..

~ n.

ER

...

lIeSi ~y~p~N~Copy

II ,

The ecmputer Room

Siudoni Ouido

Data Center Topology Actee .. IiIAyf_,.. O'rk1d communIcation cabling cross one another, they shou lei do 50 at right i1ngles to one another

cable Pathways

6.27

If both power and telecommunications cabling are distributed from below a raised fioor then :

• Power

cablin~

should be routed either adjacent to or within the cold

aisle

• Telecommunications cabling should be routed adjacent to or within tile hot aisle

Fiber 0Jjtic cabling &. Ac:c:ess Floors

6.28

If fiber optic cabling Is Installed beneath a raised ncor it should be protected from damage by plaCing it into it closed lid -containment system

Fir. ConlrOl Systams &. Acceu Floors

6.29

Fire deteCtlon/"",,ventlon systems must be insta llecl at the highest eievatioo under the raised Hoor

0-1.

IIlIte.t PI'Opr'ital1')' PW~ Ho1 CoJ2'1

Stud.nt Guido

Lighting

6.30

When occupied, the computer room shQuld h.we a minimum of 500 lux (SO It .:and les) maintained in the horizontal plane and a minimum of 200 lux (20 It cand les) maintained In the v~rtH:a1 plane, both measured at 1 m (3 tt) above the fini!hed Hoor • PortatNe, battery-powered lanterns are suggeste(i in all computer

rooms

&.31

Compute< Room Drainage computer rooms should always contain a drainage system • Computer room noor drains valves

s~oukf

always be Iltted with back·fIow

• The computer room ceiling should be constructed water away from the computer room

to drain Or 9 uIde

• The computer room should have a supply of waterproof sheets or covers available to cover all handware and equipment • An industrial wet/dry vacuum cleaner should be kept clo .. to the computer room in the event of a woter leak·

Ramps

6.32

The computer room may need to be equipped with eRher a ramp or elevator to Msure wheelchair "nd equipment access • The maximum $loJ:te for ramps is B de:grees for movement of cabinets with equipment

• ADA specifies ~ maximum ri!ie of 1: 12, Or about 4.8 degrees for wheelchair access • I'm wh .... lchalr access, the ramp must be at least 36 inches I 9&lIQor tile stringer

-

• Racks shou Id be Ieyeled throughout a row

-

• ShlmmlOg of anchor points shOUld not exceed '1.0 inch (12.5 mm)

-~:'

• Adjacent racks should be ganged together belon! anchors are ~ \J ........J -.. '1 'inylr19

ex it

Biometrics: If Fingerprinl/Eye/Face recognition hardware Is used, It snould be used in conjunction with card access to decrease databese seard> times

8.20

Doors & Electl'i!chge.,

lonizationlPho~ l ectric

Sal!ery & UPS Room.

loniutionlPncloeloc1nc

G"""rator Room.

Th."".lor Flame Oolocun

C~iij ... Room

l""itO"

- - -

- +

It

PDU

~

J

9.63

UPS Sizing

UPS salng is linked to power requirements, dass ratin9, and system desIgn • lnitial sIZIng Is based on fully meeting the critical power load with the fewest modules or pathways available during maintenance or fal lure modes of operolion Othe r sizing factors to consider

-

Load Growth

-

Safety Factor

UPS Under-slzlng

9.64

Whi le scalability is a primary goal of Data Center design, under loading can lead to system inst ability for la~e scale data centers -

CllI10 Bleil"

A UPS should be operated at no less than 20% of its capadty

student Guide

UPS Sizing & Safety Factors

'.65

It is inadvisable to load any electrical system to its full capacity; best practi~ always allows for a safety factjlT with the UPS •

A ".fety factor of at least .250/0 of capacity should be employed; 90% of capacity is reccmmended

-

for example, if a 200kW load is being served, the UPS should be sized to serve at least a 210kW load, and preferably a 220kW load

Poww & EI..mcily

\

UPS SY$- & Classes'> UPS 5ya..,.

9.86

CI.u '1

MiIIlllUnlnc. eypu.

Byptcs pOIll'et takeR

fr1Im IJtiIIt\' VOIt.te~

.....

POu.t...r .. U ceJIIIpIb!r' eqIIIprna .. ",ad _

lor ~rgB"

'10

..........-

Il..tu. .nt UPS

a .......

ByPSSS"powe:r .. fhJ';' utI/!V .

_

E5y~

T.... frorn 0

-

......... ...,;

, .OOVfor_

,

He

y••

Yes

flO

y..

y"

/'dom~or

~e

_ryUf>S,. A.otaltDg MoG set

R.DtzI'1J', AutoI11.IC

141O/108V ~p to IOIld5

t20/ 208V up to Of 1440 tvA , 4S: for all under noor or overhead receptacles are require"

• Loding receptacles are requlrectrodes shall be bonded III any other 9 rounding electrode system at the site The Building Grounding Syatam •

9.90

The building grounding sv#31)' ur ""e"'5d air is.'",~. CRIIC units do not offer the designer the flexibility with fan selectio D tu ... 8 td, tl,e -dtict performance.

-

• Central air handlers also allo... the use of air- side economizers for improved effiCiency when the phYSical conflguratlon of the building and location are ravorable

ave .......d Air & Ratum AJr Paths

10.22

II ceiling height of ~t least 10 It pm) above the raised acceS5 Hoar will typica lly allow for an effective 110t air area abOve racks and cabinet< and optimize the return air path •

Ceilil'l!ls higher than 10 It (3m) are recommended lor data centers w"illil1igh clensity server areas 1$.0 BQI Ptofrilta" ~ Not Cc:I~

Underfloor Air DIstribution

10.23

The mo:;t commo!l..ilir cooling design for data centers involves cool air delivery under a raised floor through perforate< tile with self conta Ined Computer Room Air ConditiOning (CRAQ units

-

• This sy.tem allows for maximum flexibility of cool air delivery th rough modification of perforated tile placement

Raised Floor Cooling Solutions

10.24

Underl100r air distribution can provide many advantages, including; • Allows flexibility in 1000000tion of load to CRAC unit

• Imposes fewer limits on locating CRAe units In the space • Piping services may be co"c:ealed below the raised floor • More compatible with gravity conderlSate drainage frem cooling coi Is and h urn Idlf\ers • No overhead supply ductwork to obstruct the return air path. or to interfere with lighting, sprinkler heads, or overhead power/cable distribution sy5tems • Permits the use of nearly any cooling technology, regardl..ss 01 air supply/return configuration • The use of Individual self-contained mom coolers makes under-floor cooling design more easily compatible with redundant (i.e. fault tolerant) mechanical system design Und&rfloor Cooling; Retum Air Path

10.2$

The retum air p~th is typically 'through the room' i" tflis type of system with many CRACs h avin g retum air duct extensiOns built over tflem

,.

,

~ ,

iJ;,

I';

~

cH- ,

c-'-

~ f-l- fL

I-

~

. ,,

-

I-

eaGtllal" ~C:l 1

':'

~ :.

r

:..;

C-

I: '

I .. '

'

-

,

r-

1"1.

•

~ - ~ ..J...

c-

. T_

PropMta'J PrOlNl~Do tfo4; COPt

. .'

I'

rr

I-

Studa., Gu,d

The

~piMrizornhll Dln,l"",nolfl. ~ H c> r Ib:I J)'IoJo Dn.! IlootlDflo

...

lntr.m=fbam

~,

~

~"

....

C~m.IIIlClIItlon

~In

Dbmhutlon

/\"J4

I

."""~ 1,,,,,,,,,, RboM

12.25

The Main DIstribution Area

The main dlslribullon area typically houses: • The Main Cross-connect • Core roLlters • Core LAN

swltc~

• CDre SAN swilehes

• PBXjJ PBX "'1 ulpment

• M13 multiplexers

rnl:t'UllIIC3r

12'·10 lreSI ~'Yp,~ Not Copy

T".commUftJeat~n.

Backbone Cabling Topology & the MDA

& elbing

12.26

The backbone cabling shall use the hienJrchir:alflt1llr topology. where earn horizontal cross-connect (He) in a hori:terns shall leave suffldent space to allow for the addition of palm panels needed for any migration • Sufficient cable SlaCK shall exist In the HDA to .Ik>w movement of the cable~ when migrating to a cross-connection - Optical fiber slack sha ll be stored in protective endosures - Optical fiber ch~nnel~ shall be limited all patch cord.

to 300 m (984 ftl, including 12.32

Backbone Cabling & Redundancy

Redundant backbone cabling protects against an outage due to da mage to one of the backbone cables, and can be provided In several

ways : - One way to provide backbone cabling redundan,y is simply to have two cables running fnlm the MDA, prefera~y along different routes, to a single horizontal cross-connect • A second way would be to have a backup data cable nunning between two horizontal cross-connects

M,,.

.DO

"". €l2110 IiCit'

'2·"

I IC81p .. ___ ..... , P/'6f*1)-Cc ~ot: Copy

T.~mmunfc~n. & CalMIng

S...-.cGuldo

Fully Redundant Bac:ltbon& Cabling •

Th

i

-"""

t.J=-~-CCJn lM'!:t

Equipment~!I$tIIUatlon

Pathways to the Racks

12.34

Data and po .. er cables sha lll:>e Installed In cable pathwaysconduits/raceways:, trays, and ladders, are recommended • The pathways shall have 1>0 sharp edges, bum;, can damage the cable Insulation

Dr

projections that

• Pathways may be installed eitt1er overhead"o, beneath a raised floor

ibii1' Ilear

11-14 81CS. "'q~ Pt9pe..,....oo Hen COpit

Te4ec:ommUnjcoti.,... II. Cablk>g

O\tet"Mad Cabling Pathways

12.35

Advantages of overhead cabling include:

• Cable lengtl1s are typically kept to a minimum • Adc itiOna I cables can usually be placed with minimal difficulty

• Dee. not interfere with under-floor cool air delivery to racks Disadvantages include; • May pick up interference f rom Ilg htlng fixtures • Are visible and poten bally more prone to sa botage • May IntecommunlcaUons

dass F2 data centers provide a redundant ao:ess providerfeed and corn ponent ~vel redundao"l for netwonc routers and sw itches

"Ftntrois for a single su b-systern, such as an air handling unit

• Distributed ProceHors-system-level controls for a diverse system, such as the CO1 systems usually c.an rommunlcate wlltl power and med1an ic:a1 systems using any oflhe open protocols; it is the owner/ operat or'. dedsion as 10 which prntocol best suits their needs .

BAS Cabling-()ut of the Control Room

13.8

The BAS cabling system will be deployed throughout the data center and should :

• Integ rate common services • Accommodate d iverse. BAS appl ications • Provide the capacity for rapid deployment of new equipment and

services • PrOvide redundancy to satisfy safety and reliability d ess requirements

BAS Cabling & Inbograted Design

13.9

While BAS cabj ing has often been Installed prior to voice/data cabling, combining t he instalia~o n of the two will allow for ltle .foUowing benefits:

• Reduction of initial construction and ongo ing maintenance co5I:s by allowing one project team to Install and manage .all of the cabl ing • Reduction of cabling installation hours • Reduction of damage to finished surfaces by having all cabling installed during in itia I cons\nlction • Provision of future Hexl billty and easier response to ~uests for changes

studant Guida

Bulldlng AIlIomlllon & Conlrol System.

BAS Cabling Syatem Guidelines BAS

C~bling

13.10·11

System best prar:tlces Include:

• BAS cabling $'(Stems fur data centers should be capable of operation without Interruption in case of power outage. fire; and broken cables

• BAS cab~ shall meet local FJre autnority codes to ensure the· system can survive in case of fire • Building automation cabling shall not share the same ·sheath with

communkattons 'cabl,ng • D.,,;igners shall """sider ensuring BAS system survivability by

eltner -

CAl Dual cabling that takes .diverse pathways

-

(6) Redundant BAS controller hardware storen

Yes

y""

·Yes

yO