Facebook Network Appliance Deployment | 1
© 2015-2016 Facebook. All rights reserved. Cisco®, Cisco Nexus®, and the Cisco Logo are registered trademarks of Cisco Systems, Inc. FreeBSD® and the FreeBSD Logo are registered trademarks of The FreeBSD Foundation. IBM® is a trademarks or registered trademarks of International Business Machines Corporation in the United States and/or other countries. Juniper Networks® and Junos® are [registered] trademarks of Juniper Networks, Inc. in the United States and other countries. Linux® is a registered trademark of Linus Torvalds in the United States. Mac® and Mac OS® are registered trademarks of Apple Inc. Hewlett-Packard® and HP® are registered trademarks of Hewlett-Packard Company. Windows® is either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.
Facebook Network Appliance Deployment | 2
Facebook Network Appliance Deployment | 3
Facebook Network Appliance Deployment | 4
Thank you for choosing to install the Facebook Network Appliance (FNA)! Facebook Network Appliance is Facebook’s content caching program. FNA provides Internet Service Providers (ISPs) with an efficient means of delivering static Facebook content from within their existing network. After deploying an FNA kit, an ISP will offload a significant amount of Facebook content from its backbone network and vastly improve the Facebook user experience. A single FNA kit consists of a Top-of-Rack (ToR) switch, four servers, 4-post rack mounting hardware, transceivers and cables. The hardware is suitable for deployment in data centers, colocation facilities, and outside plant environments (the industry standard 19-inch form factor allows integration into most network environments). This document describes how to install, configure, and operate FNA equipment.
Facebook Network Appliance Deployment | 5
This section lists the equipment required for building a single FNA cluster. Ensure that you have received all of the equipment required for either your SKU1 or SKU2 order (Table 1 and 2, respectively).
The following table lists equipment that is included with a SKU1 order. If equipment is missing, refer to the Support section for contact information to seek assistance.
Table 1: FNA SKU1 Provided Equipment
(1)
Cisco Nexus 3132Q-X switch2
1RU
(4)
HP ProLiant DL380 Generation 9 (Gen9) server3
2RU
(4)
4-Post rail kit
N/A
(8)
C13-C14 AC Power Cord
2 Meters
(4)
Network Cable
3 Meters
(1)
HP ProLiant DL380 Generation 9 (Gen9) server user manual
N/A
(1)
Cisco Nexus 3132Q-X switch user manual
N/A
(2)
SFP+ optical transceiver, LR or SR, (2) per SKU4
N/A
1See
Appendix I: Hardware Dimensions and Weight for more dimension details.
2Hereafter
referred to as Nexus switch.
3Hereafter
referred to as ProLiant DL380 server.
4Caution:
SFP+ optical transceivers are sensitive to static electricity.
Facebook Network Appliance Deployment | 6
The following table shows equipment that is included a SKU2 order. If equipment is missing, refer to the Support section for contact information to seek assistance. Table 2: FNA SKU2 Provided Equipment
(4)
HP ProLiant DL380 Generation 8 (Gen9) server2
(4)
4-Post rail kit
(8)
C13-C14 power cable
2 Meters
(4)
Network cable
3 Meters
(4)
HP ProLiant DL380p Generation 8 (Gen8) server user manual
N/A
(1)
Text configuration file
N/A
(2)
SFP+ optical transceiver, LR or SR, (2) per SKU3
N/A
1See
2RU1 N/A
Appendix I: Hardware Dimensions and Weight for more dimension details.
3Hereafter 3Caution:
referred to as ProLiant DL380 server.
SFP+ optical transceivers are sensitive to static electricity.
SKU2 does not include a Cisco Nexus switch. However, the order includes SFP+ optical transceivers for additional uplink capacity. Use the existing switch at your FNA deployment. The quantity of servers you receive depends on your current and forecasted Facebook traffic volume. Servers are shipped in groups of four. The equipment will arrive at your facility in one shipment. IBM (FNA’s logistics partner) will contact you with shipping information.
Facebook Network Appliance Deployment | 7
This section shows additional equipment required for the FNA installation that is not included with a SKU order. Table 3: FNA SKU1 and SKU2 Equipment Required but Not Included
(2)
Fiber Patch Cables from Cisco Nexus 3132Q-X switch to router1,2
1
The FNA accepts both SR and LR optical transceivers. Facebook provides the most optimal optical transceiver type corresponding to the type you currently use to connect your routers. 2 Fiber
connectivity on the FNA side is the Lucent Connector (LC) type. Fiber connectivity on the router side can be LC-LC, LC-SC, (Lucent Connector – Subscriber Connector), etc.
Facebook Network Appliance Deployment | 8
This section describes how to install equipment to build an FNA cluster. After installing the equipment, you will need to configure network settings to ensure a peering session is established. This information is provided in Section 4 Network Configuration.
Use the pre-installation checklist to confirm that your facility has the capacity and resources required to install an FNA cluster: Table 4: FNA Pre-Install Checklist
✓ n x 10 Gigabit (10Gb) available ports on ISP switch or router1 n available SFP+ (or compatible) optics of same type - Long Range (LR) or Short Range (SR)1 n available fiber patch cables (multi-mode or single-mode)1 n available input power connections2 Power connections are divided evenly between two different power sources Sufficient power (1kW) for each Stock Keeping Unit (SKU) Sufficient rack space is available3 1 Where
n = 2 for every FNA SKU deployed.
2 Where
n = 2 times the number of devices (e.g., SKU1 is (5) devices, so (10) input power connections are required). 3
For initial deployment (SKU1), 9RU (nine rack unit) is required. For additional capacity (SKU2), 8RU is required.
Facebook Network Appliance Deployment | 9
Install the servers into a compatible server rack. Utilize the manufacturer instructions for the server model included in your order: DL380P Gen8: Server User Guide (Edition: 6) DL380 Gen9: Server User Guide (Edition: 5) Connect C13-C14 AC power cables from each server to your power distribution units (PDUs). If you install the servers in a 2-post rack, you must provide the proper 4-post to 2-post conversion hardware.
Install the switch in the rack. Utilize the instructions in the Cisco Nexus 3000 Series Hardware Installation Guide (April 22,2015).
Facebook Network Appliance Deployment | 10
Connect each server to the switch using the supplied network cables (see Figures 3 and 4). Each server’s NIC and network cable connector have a green sticker to show the correct port connection and orientation. Ensure that the green sticker on the network cable connector is facing up and connect it to the port with the green sticker. Begin with the server at the highest location in the rack (labeled fna001) and work your way to the bottom-most server. Connect fna001 to port 32 on the switch, fna002 to port 31, and so on. See Figure in Appendix F: Additional Installation Images for a reference on server connectivity order. The Cisco Nexus switch QSFP+ ports 9 ~ 12 are not used and should not be utilized for any FNA or non-FNA equipment. Each Cisco Nexus switch supports up to twenty servers.
Connect the switch to your facility’s router.
Facebook Network Appliance Deployment | 11
Connect SFP+ ports 1 and 2. These ports are pre-configured for the Link Aggregation Control Protocol (LACP). See Appendix C: Link Aggregation/Link Aggregation Control Protocol (LACP) Configuration for more information. The Cisco Nexus’ (4) SFP+ ports and QSFP+ port 1 cannot be utilized at the same time. These ports are internally multiplexed. See Figure 26 for more information.
Power on the FNA hardware. Press the power buttons on the servers and power on any interfacing systems (i.e., PDUs, existing router, etc.). Connect AC power cables to the switch’s power supplies.
Facebook Network Appliance Deployment | 12
The Cisco Nexus switch does not have power buttons. They receive power when connected to a PDU with adequate power.
Verify proper server operation by checking the behavior of PSU, NIC, and Solid State Drive (SSD) LEDs on each server. For LED definitions, refer to the LED Definitions section in Appendix B: Troubleshooting.
Facebook Network Appliance Deployment | 13
Check the light levels. For deployments utilizing a Cisco Nexus switch, refer to Measuring the Optical Level through IOS, Document ID: 16150. Light levels need to be between -2dB and -7dB for proper operation. Light levels outside of this range may be indicative of a connectivity issue. If you find the levels out of range, see Appendix B: Troubleshooting. This completes the FNA hardware installation procedure. Figure 8 depicts a completed installation for one FNA cluster.
If you are adding additional equipment to your initial FNA installation, see section 6 System Upgrades.
Facebook Network Appliance Deployment | 14
This section describes the process required to configure FNA equipment to peer with Facebook’s network. The FNA kit is preconfigured with the IP addresses that were specified in the order form. If IP addresses need to be changed at any time, refer to the latest version of the FNA USB Boot Disk Creation and IP Reconfiguration Guide (available upon request).
Connect a VGA monitor and USB keyboard to one of the FNA servers. After powering on the servers, you will see the FNA Main Screen in Figure 10:
Configure the point-to-point connection. This connection will be configured on the Link-Aggregation Control Interface (LACP). See Appendix C: Link Aggregation/Link Aggregation Control Protocol (LACP) Configuration for additional LACP configuration details. Untag the Virtual LAN (VLAN) link-aggregation interface. Set a static route to the allocated subnets via the link-aggregation interface. Facebook requires that the FNA have internet connectivity before BGP peering can be established. The BGP peering session does not need to be routing before the FNA has internet connectivity.
Configure BGP settings. Use the IPv4 and IPv6 addresses originally supplied in the FNA order form. Enable Graceful Restart (GR). See Appendix D: BGP Configuration Details for additional configuration information. While enabling GR is not required, it is recommended to enable it. If you have questions about GR for FNA deployments, contact
[email protected].
Facebook Network Appliance Deployment | 15
Enable External BGP (EBGP) Multi-hop. The Autonomous System Number (ASN) that you will be peering with is 63293. Configure your router’s BGP settings. See Appendix D: BGP Configuration Details for configuration information. Once the router’s BGP settings have been properly configured, a latency based routing test will be initiated to optimize routing paths. It is required that all FNA servable netblocks be announced for the purpose of this test. This process takes about twenty-four hours to complete. BGP establishment is not required during this time. See section 4.4 Latency Based Routing for more information. The BGP peering session is intended to be a router-to-server configuration and not a switch-to-switch configuration. Multi-hop sessions are supported.
After the BGP settings have been saved, the FNA will begin downloading application data to prepare it for operation. The time to finish the download depends on the cluster’s connection speed. Within about two business days, BGP peering will be established: The connection will be initiated by one of the FNA servers. The connection will be coming from the 16th IP address in each allocated subnet (e.g. 192.168.1.16, fe80::10). FNA will not advertise any routes/prefixes. The remote software delivery session will begin only after all of the FNA hardware has been powered on, the point-to-point connection has been established, and routing to the internet is enabled.
This completes the network configuration of the FNA. Contact
[email protected] to notify that the installation has been completed.
FNA applies inbound filtering to reject the following address spaces on BGP peering sessions to ensure successful BGP establishment:
Prefixes smaller than /24 (255 IPs)
Default route
Bogon prefixes
RFC 1918
CGNAT
Facebook’s IP prefixes
Facebook Network Appliance Deployment | 16
It is critical that every FNA servable netblock is provided to Facebook in order to optimize routing paths for your FNA deployment.
FNA utilizes a latency based routing test to optimize traffic routing. It is critical that all servable netblocks be announced to optimize traffic routes. Prefixes announced to FNA implies those prefixes are eligible to serve Facebook traffic. The latency based routing test evaluates the round-trip latency of your network. This is accomplished by sending a small sample of static content from the netblock population you announced. Initially, only a small sample set is sent to the FNA cluster. The sample volume will ramp up during this test. This makes it possible for Facebook to observe and construct the most optimal routing paths for your FNA deployment. The accuracy of the latency based routing service is dependent on the percentage of FNA servable netblocks announced. Providing a large percentage of FNA servable netblocks allows the service to construct more optimal (low-latency) routing paths. A small percentage of netblocks announced (i.e., a test netblock) will not allow the test to function as intended.
Facebook Network Appliance Deployment | 17
Please complete this checklist after you have installed the FNA equipment to verify proper installation.
✓ All ProLiant DL380 servers in the cluster are powered on. Each power supply of each ProLiant DL380 server indicates an ON, Steady Green state. The Life (L) and Status (S) LEDs for each SSD in each ProLiant DL380 server indicate an On, Steady Green state. All accompanying LEDs for the connected fiber uplink SFP+ ports are green. All network cables are seated properly to each ProLiant DL380 server. The port 2 (P2) Link LED (LNK) on each NIC in the cluster indicates an On, Steady Green state. Light level readings for all uplinks are between -2dB and -7dB
Facebook Network Appliance Deployment | 18
This section describes how to add additional capacity your original FNA cluster and provides a suggested growth path reference.
This procedure requires updating the Link Aggregation (LAG) configuration. Updating the LAG may cause service interruption. If this is the case, it is recommended to drain network traffic first. See section 7 Maintenance and Monitoring for instructions on how to drain network traffic.
Refer to section 2.2 SKU2 Provided Equipment for a list of provided hardware. Make sure all of the hardware is available. Rack the additional servers, connect the peripheral cables (power and network), and power on the servers. Utilize section 3 Hardware Installation for reference. Connect (2) additional 10G fiber uplink cables from SFP+ Port 3 (S3) and SFP+ Port 4 (S4) on the switch to the same router (or switch) next to the initial two 10G fiber uplinks are connected (see Figure 11).
Update the Link Aggregation (LAG) configuration on the router (or switch) to accommodate the additional interfaces. All cluster augments are implemented with an increment of four ProLiant DL380 servers.
Facebook Network Appliance Deployment | 19 This procedure requires updating the Link Aggregation (LAG) configuration. Updating the LAG may cause service interruption. If this is the case, it is recommended to drain network traffic first. See section 7 Maintenance and Monitoring for instructions on how to drain network traffic. This upgrade process requires installing QSFP to SFP+ adapter module. Go to section 6.2 QSFP to SFP+ Adapter Module Upgrade for more information about this part.
Refer to section 2.2 SKU2 Provided Equipment for a list of provided hardware. Make sure all of the hardware is available. Rack the servers, connect the peripheral cables (power and network), and power on the additional servers. Utilize the Hardware Installation section for reference. Connect the provided QSFP to SFP+ adapter module assembly to QSFP Port 2 (Q2) on the Cisco Nexus switch. See Figure 12 for adapter module location.1 If you are upgrading from a (16) server to a (20) server cluster, connect the 2nd provided QSFP to SFP adapter module to QSFP Port 3 (Q3) on the switch.
Connect (2) additional 10G fiber uplink cables for each group of (4) servers from the QSFP to SFP+ adapter module to the same router (or switch) where the initial uplinks are connect (S1 ~ S4). Update the LAG configuration on the router (or switch) to accommodate the additional interfaces. The provided QSFP to SFP+ assembly only supports LR single-mode connections. Refer to Figure : System Architecture Diagram for a diagram of a (20) server FNA cluster.
When upgrading an FNA cluster beyond eight servers, a QSFP to SFP+ adapter needs to be installed for each additional group of four servers. If upgrading to a 12, 16, or 20 server FNA cluster, specifiy that this part will be needed.
Facebook Network Appliance Deployment | 20
Manufacturer: Cisco Part Number: CVR-QSFP-SFP10G
This section provides a reference for best practices regarding augmentations. This growth path optimizes for deployment failover scenarios by reducing the quantity of Single Points Of Failure (SPOF).
When building an FNA cluster beyond eight servers, it is best practice to install the additional servers in a new rack and maintain equal size clusters (as depicted in growth path 3 and 4).
Facebook Network Appliance Deployment | 21
When performing either singular or recurring maintenance jobs, email
[email protected] to report the planned maintenance. Provide the following information in the email:
To:
[email protected] Subject: [FNA ISP] Maintenance Notification Body: Provide the following: o Company name and ASN o Cluster name (e.g., FBSB1-1) o When the maintenance will be performed (start and end times) o If the maintenenace will be recurring (specify the interval) o Reason for maintenance o Contact information
Before powering down the FNA cluster for maintenance, all network traffic must be properly drained. This procedure describes how to properly drain traffic from the cluster: This procedure requires withdrawing prefixes. It is important that the BGP peering session is maintained while withdrawing prefixes.
Withdraw prefixes. While maintaining the BGP peering session, begin withdrawing BGP prefixes. Traffic will begin to drain. Traffic will fully drain in less than one hour.
You should shut down the FNA only if it is absolutely required and traffic has been properly drained from the system (see 7.2 Draining Network Traffic). This should be a rare occurrence. To shut down the FNA, follow the procedure outlined below or contact
[email protected] for assistance: Connect a monitor and keyboard to the server you want to power off. Use the arrow keys navigate to the Shut Down text. Press the [Enter] key. The FNA will power off.
Facebook regularly monitors internet reachability from various simulated endpoints. When reachability issues are detected on these Virtual IPs (VIPs), Facebook may drain traffic from an FNA cluster. This measure ensures that Facebook maintains a high quality of service for the Facebook platform.
Facebook Network Appliance Deployment | 22
The Terms of Service agreements that you signed when you installed the FNA in your facility included specific language pertaining to the disposal of storage media. That language is provided here for your convenience. Upon signing the Terms of Service, the ISP agrees to:
Only deal with and process Edge user data in compliance with, and subject to, the instructions received from Edge and will not use or process the Edge User Data for any other purpose whatsoever. Take custody of and secure, and support the destruction of, all server storage media (example: hard disk, solid state disks, flash memory cards, etc.) that are uninstalled from the servers, and provide secure handling, storage, packing, and shipping of the storage media per IBM provided instructions and at IBM's expense to an IBM designated asset recovery center for destruction.
Facebook Network Appliance Deployment | 23
The Network Partner Portal is the central hub for ordering FNA kits, reviewing deployment information and metrics, and troubleshooting issues you may encounter with your deployment. This section describes how to use various features within the Network Partner Portal. URL: https://partners.facebook.com/network/
Logging into the NPP is as easy as logging into Facebook. If you have an existing Facebook account associated with your work email address, the portal will use that account to log in. If you have a Facebook account associated with your personal email address, you must associate your work email address with your Facebook account. If you don't have a Facebook account, you need to create one. Go to http://www.facebook.com and follow the instructions for creating an account. To associate your work email address with your Facebook account, perform the following steps: Click the Menu icon (
) located at the top right of a Facebook page. Click Settings.
Click Email. Click Add another email or mobile number. Enter your work email address in the New Email field. Click Add. Re-enter your Facebook password when prompted. A confirmation code will be sent to verify the email address. Click Confirm and enter the code. Click OK. Click Save Changes.
Facebook Network Appliance Deployment | 24
The portal allows you to authorize others as administrators and set their permissions.
To add a new administrator, perform the following steps: Click the Settings icon ( ) located at the top right of the portal page. Enter the name and email of the user you wish to make an administrator. Click Add. An invite will be sent to the user's email address. Their permission status will remain "pending" until they accept the invite.
Admin access grants access and control to all settings and data within the portal. You can also set the user's permissions to "Read-Only Access". This gives the user complete access to all portal data, but does not allow access to any settings or controls. To set a user's permission to read-only access, follow the steps in the Modifying Permissions section and select Read-Only Access.
To modify an existing user's permissions, perform the following steps: Click the Settings icon ( ). Click the Edit link on the user's card. Click the chevron icon ( Access.
) below the user's email address. Select either Admin Access or Read-Only
Facebook Network Appliance Deployment | 25
Click Done.
To remove a user from the portal, perform the following steps: Locate the user you would like to remove. Click the Remove link on the user's card. Enter your Facebook password if prompted.
Facebook Network Appliance Deployment | 26
FNA deployments need to connect to your network. To enable this connection, you need to provide your ISP's Autonomous System Number (ASN). Facebook requires this data to estimate the number of servers your FNA deployment will require. Facebook also uses this information during the provisioning process to properly route traffic to the FNA. To add your ASN, follow these steps: Click the Settings icon ( ). Click Autonomous Systems located in the left menu. Enter your ASN and AS Name. Enter your ISP's ASN and AS Name. Click Add.
To remove an ASN, follow these steps: Click the Settings icon ( ). Click Autonomous Systems located in the left menu. Click the Remove link next to the ASN entry you wish to remove. You can enter multiple ASNs. All ASNs should be kept up-to-date. If the ASN entry has a yellow clock icon (
), it is currently being reviewed by Facebook.
Requesting an FNA kit is a multi-step process that takes several weeks to complete. Facebook evaluates
Facebook Network Appliance Deployment | 27
submissions for qualified ISPs that can utilize an FNA. If your application is approved to receive an FNA, Facebook will send an FNA kit to the designated ISP address. To request an FNA kit, follow these steps: Click the Support link at the top of the page. The Support page will appear. Click the Request a New Kit hyperlink. Enter the deployment country where the FNA will be installed. Click Next. Select the Type of Kit. Review the information and click the Begin Order Form button. An online order form will appear in a new window or tab. Use table 6 as a guideline for order form field entry:
This can be the engineering contact.
Shipping Contact
Provide contact information for the primary contact that will receive FNA shipments.
This can be the engineering contact.
Installation Contact
Provide contact information for the primary contact that will install or lead installation of the FNA equipment.
Engineering Contact
Provide contact information for the primary engineering contact that can answer technical questions about the network.
Shipping Address
Provide the location where FNA equipment will be sent.
Site Installation Address
Provide the location where FNA equipment will be installed.
Special Shipping Instructions
Provide any notable unique shipping instructions.
This is an optional field.
Rack Name
Enter a name for the FNA rack.
This is an optional field.
Facebook Network Appliance Deployment | 28
Facility / Deployment Technical Information
Networking and IP Information
IPv4 Settings
Provide power source requirements for the FNA deployment (AC or DC). Provide the type of optical transceiver your deployment would require – either long range (LR) or short range (SR) Provide the power outlet type your deployment would require. Provide the primary peer ASN. Provide any additional ASNs or communities to be served.
Provide IPv4 allocation for caches, gateway address and network, cache address and network, and Border Gateway Protocol (BGP) peer address.
Provide IPv6 allocation for caches, gateway address and network, cache address and network, and BGP peer address. IPv6 Settings
Gateway and Cache Peer-to-Peer (P2P) IP addresses must be within the same subnet. BGP address / network should not overlap with IP allocation for caches. Gateway and Cache Peer-to-Peer (P2P) IP addresses must be within the same subnet. BGP address / network should not overlap with IP allocation for caches.
Fill out and submit any required order forms located in the Required Forms section near the top of the order form. Forms that require submission are denoted with the "X" icon ( ). Forms that are currently pending review are denoted with the "exclamation" icon ( ). Click Submit if you have completed the order form or Save Draft if you plan to complete the order form at a later time.
Facebook Network Appliance Deployment | 29
If Facebook selects you to participate in the FNA program, International Business Machines (IBM) will contact the shipping contact you indicated with further details and will make arrangements to deliver FNA equipment. IBM is Facebook's logistics partner for FNA equipment.
Facebook Network Appliance Deployment | 30
Once you have installed your FNA equipment and it is brought online, access the Manage FNA to:
Review Installed Kit Information
View Traffic Information
Review Contact, Exchange, and Facility Information
View Peering Sessions
The installed kit information page allows you to view the number of servers per cluster, IPv4 and IPv6 network addresses, advertised prefixes, and disk image. To view this page, follow these steps: Click the Caching tab located at the top of the portal. Click Manage FNA. This page displays FNA information sorted by cluster name. You can also request a new kit by clicking Request a New Kit in the top-right corner.
Facebook Network Appliance Deployment | 31
FNA traffic metrics are located in the Insights page. This page allows you to observe cache-fill and retransmit information sorted by FNA cluster name. To view traffic information, follow these steps: Click the Caching tab located at the top of the portal. Click Insights. The FNA Traffic information will appear. Each plot represents either "To Subscribers" or cache fill for each FNA cluster installed. To view traffic retransmission information, click the button at the top left of the graph. Click FNA Retransmits. This page will show traffic retransmission as a percentage of the total FNA traffic.
The contact information you provided on the FNA order form can be found in the Peering page. To review this information, follow these steps: Click the Peering tab. Click Peering Summary (this should load by default). This page allows you to review your contacts, exchanges, and facilities information.
Facebook Network Appliance Deployment | 32
The contact information you provided on the FNA order form can be found in the Peering page. To review this information, follow these steps: Click the Peering tab. Click Peering Sessions. This page allows you to view peering session details, including ASN, location, peer type, link speed, router address, peer port, and status.
Facebook Network Appliance Deployment | 33
If you have issues receiving equipment or installation and configuration issues, use Table 7 to contact the most appropriate support team for assistance.
Table 7: Support and Contact Information
Email:
[email protected] Facebook Network Appliance
Email:
[email protected]
IBM
Network Support is Required BGP Peering Issues Assistance with Draining Network Traffic
Hardware Support is Required
Notification of Hardware Installation Completion
Hardware is missing from order. Provide the following: o Missing Item(s) o ISP Name o ISP Address o Equipment Received Date o Your Contact Information
In some cases, it may be necessary to return the equipment. It is recommended to keep the original packaging material for two months.
Facebook Network Appliance Deployment | 34
What type of data is being stored on FNA? An FNA cluster will cache the following types of data:
Static Facebook content Static Instagram content
FNA does not cache WhatsApp or Oculus content.
How much traffic can the FNA cluster serve? The amount of traffic that the FNA cluster may serve varies based on:
The size of the cluster and the number of available machines The amount of prefixes being advertised / number of users The RTT latency to advertised prefixes Content type variance: profile pictures, large images, and video
Each server in the cluster is capable of delivering ~4Gbps of traffic. However, depending on the deployment region, service type (e.g., mobile vs. fixed line), and variance in content type (e.g., images and video), the FNA cluster may deliver less overall traffic (~3Gbps is expected). Generally, a high variance in content type will yield lower traffic throughput and low content type variance will yield higher traffic throughput.
The installation procedure indicates installing the supplied switch directly to our existing router. Our facility aggregates to our existing switches before connecting to a router. How does this modify the installation process? FNA provides aggregate connections from the cache servers directly to the router. However, if you connect the supplied FNA switch to an intermediary aggregation switch, this configuration is fine and the logical topology will remain the same.
Is FNA Carrier-Grade NAT (CGNAT) compatible? Yes. FNA is compatible with CGNAT for the RFC 1918 and RFC 6598 address spaces.
Facebook Network Appliance Deployment | 35
I’ve advertised a certain prefix but it doesn’t seem to be served by FNA. Why is this happening? Advertising a prefix marks means it as eligible to be served from FNA, but does not guarantee it will be served by FNA. There are a few reasons why FNA may not be serving a prefix which has been advertised to it:
If it has been less than 24 hours since the FNA cluster has been online or since you have started advertising the prefix (see Latency Based Routing). Facebook may not have collected enough latency samples in order to decide whether the FNA cluster will serve the prefix. If Facebook is observing less latency to the prefix from a nearby Point of Presence (PoP), then the PoP will be preferred for serving the prefix. If the system resources (e.g. CPU utilization) have been exhausted.
Which routes should I advertise towards Facebook and how does failover occur between FNA’s? In most cases, the best option is to announce all customer prefixes. In cases where there may be congestion from region to region, the best practice is to announce local routes only. Why should I advertise all of my prefixes? Facebook’s content routing platform measures Round Trip Time (RTT) latency between each Point of Presence (PoP) and FNA deployment to all prefixes being advertised by a peer ISP. Based on this measurement and a number of other factors (such as available system resources and link capacity), a client is routed to the most optimal location for making content requests.
What is the FNA cluster naming convention? FNA cluster naming is based on the International Air Transport Association (IATA) code, the site number, and the cluster number. ‘F’ is prepended before the IATA code (which represents the nearest airport recognized by the IATA); the site number and cluster number are enumerated values. For example, FBKK5-3 represents the 3rd cluster in the 5th location in Bangkok (where ‘BKK’ represents the IATA code for Suvarnabhumi Airport).
How does an ISP qualify to participate in the program? The following is a summary of the qualification criteria to receive an FNA kit:
The ISP is considered an ‘eyeball’ ISP (not a transit ISP). The installation site has at least 5Gbps of peak Facebook traffic in a region where a Facebook PoP is not located. The ISP maintains the ability to deliver Facebook content from a single deplyment location in an efficient manner. The ISP has the ability to allocate IPv6 address space for the FNA cluster. The installation site has the ability to provide the following (summarized) space, power, and network requirements: o Space: 9RU
Facebook Network Appliance Deployment | 36
o o
Power: Minimum of 1kW Network: 2x10G uplink connectivity, BGP peering session capability IPv4: Up to /29 and /26 address space IPv6: Up to /125 and /64 address space
The installation site is located in a developing market / high-growth region.
Facebook Network Appliance Deployment | 37
This section provides troubleshooting resources for events that may be encountered during installation or operation.
Each ProLiant DL380 server and Nexus switch has LEDs that can be used to diagnose the FNA for issues. These LEDs also useful for verifying whether the FNA was installed correctly and for receiving quick technical support.
Activity (A)
Life (L)
Off
____________________
The server is powered off or the power supply has failed.
On, Steady Green
____________________
The power supply is on.
On, Blinking Green
____________________
There is data activity on the card.
On, Steady Green
____________________
The drive has sufficient life for programming and erasing the memory.
On, Steady Yellow
____________________
The drive has approximately 10% or less life remaining for programming and erasing the memory.
On, Steady Red
____________________
The drive no longer has program or erase cycles. Data can be read, but writing data is at risk.
On, Steady Green
____________________
The drive is operating normally.
On, Blinking Green
____________________
A network administrator is toggling the LED to locate the server. Drive warning. One of the following has occurred:
Status (S) On, Steady Yellow
____________________
On, Blinking Red
____________________
The maximum temperature threshold has been reached. Another issue has occurred.
A firmware fault code has been issued.
Facebook Network Appliance Deployment | 38 Fatal drive warning. One of the following has occurred:
Status (S)
On, Steady Red
____________________
Drive failure. Drive has exceeded its maximum temperature threshold. The RAID volume has failed. The RAID volume is not configured. The drive’s internal backup power supply has failed. Another issue has occurred.
Activity (ACT)
On, Blinking Green
____________________
There is network activity.
Link (LNK)
On, Steady Green
____________________
A connection exists between the server and the network.
Power Supply
On, Steady Green
____________________
The power supply is on.
Fan
On, Steady Green
System Status
____________________
The fan is powered on.
On, Steady Green
____________________
The system is operating normally.
On, Steady Orange
____________________
The system has posted an environmental code.
On, Steady Red
____________________
The system has posted an environmental code.
issue
issue
Facebook Network Appliance Deployment | 39
In cases where light levels are outside the normal range (between -2dB and -7dB), utilize Table 9 to resolve the issue. Table 9: Signal Level Troubleshooting
Above -2
The signal is too strong. This could be caused by many reasons.
Fix with your normal operating procedures for strong signals.
None (Normal Operating Range)
None (Normal Operating Range)
-2 to -7
-8 to -20
-21 to -40
A degraded fiber optic cable is installed A degraded SFP+ optical transceiver is installed A fiber optic cable has a dirty connection
A fiber optic cable is bent Network connectivity has not been established All cables are connected to their proper ports, but the port(s) is disabled All ports are enabled, but a cable is connected to the incorrect port
Disconnect, clean, and reconnect the fiber optic cable(s) and/or SFP+ optical transceiver(s).
If this does not resolve the issue, either replace TX/RX hardware or solve with your normal fiber replacement procedures.
Inspect the fiber optic cable(s) along its length. Gently roll the fiber if you observe bends or kinks. Check that the port connection is enabled Ensure the cables are connected to the proper port interface. See Figure 12 for port interfaces.
Facebook Network Appliance Deployment | 40
The following are examples of LACP configurations for the most common Cisco IOS and Juniper Junos routers. Apply the lines of code (below) that are appropriate to your specific router model.
The following configuration settings should be applied to a Juniper Junos router for FNA LACP compatibility: ae1 { aggregated-ether-options { lacp { active; periodic fast; } } unit 0 { description FACEBOOK-FNA; family inet { address ; } family inet6 { address ; } } } xe-0/0/0 { description FACEBOOk-FNA; gigether-options { 802.3ad ae1; } } xe-1/0/0 { description FACEBOOk-FNA; gigether-options { 802.3ad ae1; } }
To confirm whether the LACP settings were properly applied, run the following commands: show lacp interfaces xe-4/3/3 show interfaces ae3 | grep Physical If the following output appears, the LACP settings were properly configured for the Juniper Junos router: show lacp interfaces xe-4/3/3 Aggregated interface: ae4 LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity xe-4/3/3 Actor No No Yes Yes Yes Yes Fast Active xe-4/3/3 Partner No No Yes Yes Yes Yes Fast Active LACP protocol: Receive State Transmit State Mux State xe-4/3/3 Current Fast periodic Collecting distributing show interfaces ae4 | grep Physical Physical interface: ae4, Enabled, Physical link is Up
Facebook Network Appliance Deployment | 41
The following configuration settings should be applied to a Cisco IOS-XR router for FNA LACP compatibility: ! interface Bundle-Ether1 description FACEBOOK-FNA ipv4 address ipv6 address ! interface TenGigE0/0/0/0 description FACEBOOK-FNA bundle id 1 mode active lacp period short ! interface TenGigE0/1/0/0 description FACEBOOK-FNA bundle id 1 mode active lacp period short !
To confirm whether the Cisco IOS-XR LACP was properly configured, run the following command: show port-channel sum interface port-channel 911 If the following output appears, the LACP settings were properly configured on the Cisco IOS-XR router: show port-channel sum interface port-channel 911 Flags: D - Down P - Up in port-channel (members) I - Individual H - Hot-standby (LACP only) s - Suspended r - Module-removed S - Switched R - Routed U - Up (port-channel) M - Not in use. Min-links not met -------------------------------------------------------------------------------Group PortType Protocol Member Ports Channel -------------------------------------------------------------------------------911 Po911(RU) Eth LACP Eth18/9(P) Eth18/10(P) Eth18/11(P) Eth18/12(P)
Facebook Network Appliance Deployment | 42
The following are examples of BGP peering code for the most common Cisco OOS and Juniper Junos routers. These examples allow you to configure BGP peering sessions to send Facebook traffic to the FNA cluster. Contact Facebook (
[email protected]) if you encounter problems when setting up BGP peering. After the FNA cluster has network access, it will begin downloading application data that enables the FNA to operate. All application data must be installed before the BPG session is established. The application data may take some time to download.
The following router BGP configuration should be used for FNA deployments that utilize a Juniper Junos router: protocols { bgp { group FACEBOOK-FNA-IPv4 { neighbor { multihop; family inet { unicast; } export FACEBOOK-FNA-EXPORT; peer-as 63293; } } group FACEBOOK-FNA-IPv6 { neighbor { multihop; family inet6 { unicast; } export FACEBOOK-FNA-EXPORT; peer-as 63293; } } } } policy-options { policy-statement FACEBOOK-FNA-EXPORT { term export { from { prefix-list FACEBOOK-FNA; } then accept; } term reject { then reject; }
While enabling GR is not required, it is recommended that it is enabled. If you have questions about GR for FNA deployments, contact
[email protected].
To enable BGP Graceful Restart on a Juniper Junos router, perform the following procedure: Utilize Juniper’s Configuring Graceful Restart Options for BGP reference document. Navigate to the appropriate hierarchy level and change the disable statement to enable.
Facebook Network Appliance Deployment | 43
Evaluate whether GR was successfully negotiated by running the following command: show bgp neighbor | grep Options Locate the applicable Junos device and evaluate whether GracefulResart is within the Options output: show bgp neighbor | grep Options Options: Options:
If the Options output string contains GracefulRestart, GR was successfully negotiated.
By default, GR is disabled on Junos devices. Enabling Graceful Restart on the Junos device does not imply that it is in use. Both sides of the session must support and negotiate Graceful Restart.
The following router BGP configuration should be used for FNA deployments that utilize a Cisco IOX-XR routers: ! router bgp x ! address-family ipv4 unicast ! address-family ipv6 unicast ! neighbor remote-as 63293 address-family ipv4 unicast route-policy FACEBOOK-FNA-EXPORT out ! neighbor remote-as 63293 address-family ipv6 unicast route-policy FACEBOOK-FNA-EXPORT out !
While enabling GR is not required, it is recommended that it is enabled. If you have questions about GR for FNA deployments, contact
[email protected].
To enable BGP Graceful Restart on a Cisco NX-OS router, perform the following procedure: Utilize Cisco’s Configuring a Graceful Restart procedure.
show ip bgp neighbors | grep "Graceful Restart"
Facebook Network Appliance Deployment | 44
If the output string ‘Graceful Restart capability: advertised received’ appears, GR was successfully negotiated.
Facebook Network Appliance Deployment | 45
The following diagram demonstrates the route of a Facebook request to your ISP both before and after the FNA deployment:
Facebook Network Appliance Deployment | 46
QSFP Port 1 is internally multiplexed with the (4) SFP+ uplink ports for both the Cisco Nexus 3132Q and 3132Q-X. The (4) SFP+ ports and QSFP Port 1 cannot be utilized at the same time.
The following diagram shows the topology of a (20) server FNA installation:
Facebook Network Appliance Deployment | 47
This diagram depicts an FNA cluster with the maximum number of supported servers per switch. However, when building an FNA cluster beyond eight servers, it is best practice to install the additional servers in a new rack and maintain equal size clusters (as demonstrated in Section 6.2 Growth Path).
Facebook Network Appliance Deployment | 48
Table 10: FNA System Benchmarks
Expected Bandwidth Offload
75%
Maximum Sustained Throughput
4Gbps per cluster
Facebook Network Appliance Deployment | 49
Table 11: AC Power and Temperature Requirements
Input Current (by PSU)
8.9A @100V 4.3A @ 200V
Steady State Power
750W @ 100 V to 120V 750W @ 200V to 240V
Frequency
50-60Hz
Dual Path Power
Required
Dual PDUs
Required
Operating Temperature Range
10°C to 35°C
Typical Operating Power
210W with Twinax @ 100% load; (2) PSUs 240W with SR4 optics @ 100% load; (2) PSUs
Maximum Power
340W
Input Voltage by PSU
100V to 240V
Frequency
50-60Hz
Efficiency
89% to 91% @ 220V
Dual Path Power
Required
Dual PDUs
Required
Operating Temperature Range
0°C to 40°C
Typical Heat Dissipation
717 BTU/hr (with Twinax at 100% load) 819 BTU/hr (with SR4 optics at 100% load)
Maximum Heat Dissipation
1160 BTU/hr
Cordset Rating
10A @ 250V
SKU1
~ 800W (per SKU)
Facebook Network Appliance Deployment | 50
Table 12: DC Power and Temperature Requirements
Input Voltage Range (V DC)
36-72V DC
Nominal Input Voltage (V DC)
36/48/72V DC
Maximum Rated Output Wattage Rating (W)=
750W @ 36V DC / 48V DC / 72V DC
Nominal Input Current (A)
23A @ 36V DC 17A @ 48V DC 11A @ 72V DC
Maximum Rated Input Wattage Rating (Watts)
815W @ 36V DC 805W @ 48V DC 795W @ 72V DC
Maximum British Thermal Unit Rating (BTU-Hr)
2780 BTU-Hr @ 36 V DC 2740 BTU-Hr @ 48 V DC 2720 BTU-Hr @ 72 V DC
Operating Temperature Range
10°C to 35°C
Typical Operating Power
210W with Twinax @ 100% load; (2) PSUs 240W with SR4 optical transceiver @ 100% load; (2) PSUs
Maximum Power
340W
Input Voltage by PSU
–40V DC to –72V DC
Maximum Current
33A
Power Supply Efficiency
88% @ -48VIN (Typical Power Draw) 85% @ -48VIN (Maximum Power Draw)
Dual Path Power
Required
Dual PDUs
Required
Operating Temperature Range
0°C to 40°C
Typical Heat Dissipation
717 BTU/hr (with Twinax @ 100% load) 819 BTU/hr (with SR4 optical transceiver @ 100% load)
Maximum Heat Dissipation
1160 BTU/hr
Facebook Network Appliance Deployment | 51
Refer to the specifications for the equipment included in your FNA kit (this can vary according to when the order was placed). Table 13: Hardware Dimensions and Weight
Dimensions (Height x Width x Depth)
8.74 x 44.55 x 69.85 cm (3.44 x 17.54 x 27.50 in)
Weight (Approximate)
Dimensions (Height x Width x Depth)
8.73 x 44.55 x 73.02 cm (3.44 x 17.54 x 28.75 in)
Weight (Approximate)
Dimensions (Height x Width x Depth)
4.4 x 43.9 x 50.50 cm (1.72 x 17.3 x 19.7 in)
Weight
9.3 kg (21.5 lb)
Dimensions (Height x Width x Depth)
4.4 x 43.9 x 50.50 cm (1.72 x 17.3 x 19.7 in)
Weight (Approximate)
9.3 kg (21.5 lb)
Length
2 Meters
Plug
SS10A
Connector
HS10S
American Wire Gauge (AWG)
30
Length
3 Meters
Dimensions (Height x Width x Depth)
8.5 x 13.4 x 56.5 mm
Weight
75 grams (2.64 ounce)
