GLOBAL TELECOM & TECHNOLOGY
GTT NOC Customer Support Guide
8484 Westpark Drive, Suite 720, McLean, VA, 22102 // 1866 767 2767 .
Customer Support Guide - Contents • 1.0 Welcome!
• 2.0 Global Telecom and Technology Network Operation Center (GTT NOC)
2.1 GTT NOC Commitments
2.2 GTT NOC Contact Details
• 3.0 Ticket Types and Severity Definitions
3.1 Trouble Ticket Types
3.2 Trouble Ticket Severity Categories
3.3 Change Requests Severity Categories
3.4 Mobility Severity Categories
3.5 RFO Requests
3.6 SLA Credits for Outages
3.7 Planned and Urgent Maintenance
3.8 Circuit Provisioning (Turn Ups)
• 4.0 Escalation Timeframes
GTT NOC Customer Support Guide // 1
• 5.0 Trouble Reporting and Management Process
5.1 Managed Network
5.2 Unmonitored Network
5.2.1 Outage Duration Calculation
5.2.2 Corrective Actions and Follow-up Tickets
5.2.3 Chronic Ticket Procedures
5.2.4 Critical Outage Procedure
5.2.5 Critical Outage Process Flow Chart
• 6.0 Fault Reporting
6.1 Fault Reporting Contacts
6.2 Fault Reporting Requirements
6.2.1 Trouble Tickets
6.2.2 Change Request Tickets
6.2.3 Mobility Service
6.2.4 Remote Hands Requests
• 7.0 Calls
7.1 Call Updates
7.2 Call Status Explained
7.3 Call Closure
• 8.0 Frequently Asked Questions
GTT NOC Customer Support Guide // 2
1.0 // Welcome!
Dear Customer,
As a GTT client, you have 24 / 7 / 365 access to our Network Operations Center (NOC) to manage your support needs. In order for you to understand our customer support process and use the NOC as effectively as possible, we have developed this support guide.
We sincerely hope you don’t have to use this document often, but we have created it in order to provide you with first class customer care. Thank you for your continued business.
Regards, The GTT Team
GTT NOC Customer Support Guide // 3
2.0 // Global Telecom and Technology Network Operation Center (GTT NOC) The GTT NOC provides 24 / 7 / 365 one-stop-shop fault management including: ‣ ‣ ‣ ‣ ‣ ‣
Monitoring and management of customer networks Responding and resolving customer reported faults Managing trouble tickets Overseeing maintenance on services (scheduled and emergency maintenance) Provisioning and testing services to ensure customer has connectivity end to end Fault escalation management
2.1 // GTT NOC Commitments: 1. GTT NOC will liaise with internal and external teams to resolve customer faults 2. GTT NOC will escalate tickets internally within GTT Operations Management and externally with Off-Net vendors until customer fault is resolved. (Please see Fault Escalation and Ticket Category section for more details) 3. Under certain circumstances your ticket may be assigned to other GTT teams: ‣ Client requested change has commercial implications ‣ Client requesting commercial/contractual (SLA) information ‣ Client requesting billing credits ‣ Client requesting design change If a ticket is assigned to a different team within GTT, the the GTT NOC will provide client with contact details of the relevant GTT Sales Account Manager or Project Manager. GTT NOC will also escalate this to NOC/ Operations Management if client requests escalations.
2.2 // GTT NOC Contact Details: NOC Direct UK: Europe Toll Free: NOC Direct USA: NOC US Toll Free: Fax: Email:
+44(0)20 7489 4200 00800 32743571 +1 703 442 5539 +1 877 385 5252 +1.866.250.3887 +44(0)20 7489 7300
[email protected]
GTT NOC Customer Support Guide // 4
3.0 // Ticket Types and Severity Definitions 3.1 // Trouble Ticket Types GTT NOC tickets are categorized into the following types: 1. Trouble Tickets 2. Change Requests 3. Mobility Services 4. Remote Hands 5. RFO Request 6. SLA Credit 7. Info Tickets 8. Maintenance Tickets 9. Turn-Up Tickets
Network/circuit faults Customer requesting changes to service Mobility service issues Request for remote hands at GTT managed co-lo Request for written RFO reports Request for SLA credits for outage Customer information request tickets Planned maintenance on customer services New service provisioning
3.2 // Trouble Ticket Severity Categories CATEGORY
DEFINITION
EXAMPLE
Priority 1 (P1)
Service down/unusable
BGP down
Priority 2 (P2)
Service usable but degraded performance
Errors, packet loss, circuit bouncing, latency
Priority 3 (P3)
Quality issues with services
Intermittent latency, connection to a destination
Priority 4 (P4)
Historic hits & ‘verbal reason for outage’ requests
Service was down or faulting but recovered before contacting GTT NOC
GTT NOC Customer Support Guide // 5
3.3 // Change Requests Severity Categories All Change Requests tickets are categorized as Priority 3. The following type of service changes will be classified as Change Requests Tickets: ‣ ‣ ‣ ‣ ‣
BGP prefix updates RDNS requests Usage Request New IP allocation IPv6 requests
3.4 // Mobility Severity Categories CATEGORY
DEFINITION
Priority 1 (P1)
All users blocked, roamserver down/unreachable
Priority 2 (P2)
Multiple and single user/PoP issues
Priority 3 (P3)
Single user issues
Priority 4 (P4)
Profile upgrade, profile changes, all other issues
3.5 // RFO Requests GTT NOC will provide a verbal RFO or written (via e-mail) for all Priority 1 and Priority 2 trouble tickets during closure of a ticket. If you require a full written RFO report the please send your request to
[email protected]. ‣ ‣ ‣
The GTT NOC will only provide written RFO for outages over 15 minutes RFO report will not be provided for outages reported after the incident (i.e. Historic Outage) RFO reports will not be provided for Priority 2 tickets*
*Under exceptional circumstances the GTT NOC management may provide RFO report for P2 tickets.
3.6 // SLA Credits for Outages All such requests should be made to
[email protected]. Please ensure that you include the following information in your email: ‣
GTT outage ticket number
GTT NOC Customer Support Guide // 6
‣ ‣
GTT RFO ticket number if one was opened with GTT NOC GTT circuit ID/PON number
* Under exceptional circumstances the GTT NOC management may provide RFO report for Priority 2 tickets.
3.7 // Planned and Urgent Maintenance Maintenance tickets are raised as Priority 3 and are categorized as follows:
Planned Notification is sent to customer at least 5 days in advance.
Emergency Emergency maintenance is carried out with less than 5 days notice. The GTT NOC makes every effort to reschedule any emergency or planned maintenance that are scheduled during a customer’s core business hours. However, if you receive an emergency or planned maintenance notification and would like it rescheduled, please contact
[email protected] or call the GTT NOC. Maintenance notifications will include: ‣ ‣ ‣
Timescales for the maintenance window Impact of the maintenance Reason for maintenance
Customers can call the GTT NOC anytime after the maintenance window to verify if maintenance was completed (please quote GTT ticket number).
GTT NOC Customer Support Guide // 7
3.8 // Circuit Provisioning (Turn-ups) All Provisioning Tickets are classified as Priority 4. Customers can call the GTT NOC for assistance with the provisioning/testing of a new circuit. Clients are advised to call the GTT NOC at least 48 hours prior to the turn-up of the circuit. The GTT NOC will then open a ‘provisioning ticket’ and arrange the required logistics to assist customer during the provisioning phase.
4.0 // Escalation Timeframes All Critical Faults (Priority 1) are managed as follows: 1. 1st Line NOC Tech opens the call and ensures all information is obtained from client and GTT NMS 2. 1st Line Tech will then either escalate ticket to 2nd Line Tech if the issue is “on-net” or open a ticket with the corresponding GTT “off-net” provider 3. 2nd Line Techs are available in the NOC to assist 1st line techs in isolating and testing both “on-net” and “off-net” faults 4. 2nd Line Techs will escalate ticket to 3rd Line GTT Engineering if a change is required within the GTT Core Network if the fault is isolated to the GTT Core Network The GTT NOC will proactively escalate faults using the intervals below, should the customer wish to speak with the escalation contacts listed below, they should call the GTT NOC. The customer will either be bridged in with the relevant point of escalation or contacted directly.
ESCALATION TABLE
P1
P2
P3 / P4
GTT NOC 1st Line Tech
0H
0H
0H
GTT NOC 2nd Line Tech
2H
4H
48 H
On-Duty Shift Manager
4H
8H
7 days
On-Duty NOC Manager
7H
24 H
10 days
VP Global Operations
12 H
72H
President &CEO
24 H
N/A
N/A
* The GTT NOC does not automatically escalate Provisioning Tickets. If you require an escalation of any provisioning tickets then please ask GTT NOC Analyst or the assigned GTT Project Manager.
GTT NOC Customer Support Guide // 8
5.0 // Trouble Reporting and Management Process GTT’s Trouble Reporting and Fault Management will vary depending on the types of services that GTT has provided.
5.1 // Managed Network The GTT NOC will pro-actively monitor all Managed Customer Networks and log fault tickets as and when alarms are seen on these networks. ‣ ‣ ‣ ‣
‣
The GTT NOC will log a ticket and investigate all alarms on the network The GTT NOC will notify the customer of any issues within 5 minutes of a network alarm being seen The GTT NOC will investigate the issue with the supplier responsible until the issue is resolved If the GTT NOC believes the problem to be at the customer’s site or due to the malfunction of the customer’s equipment, the GTT NOC will ask the customer to engage with GTT NOC staff in troubleshooting If the customer is unavailable then the GTT NOC will suspend the trouble ticket until the customer is available
5.2 // Unmonitored Network For unmonitored services customers should report all faults to the GTT NOC as soon as they become aware of any connectivity issues and have completed initial checks on their equipment. On certain Network Outages, the GTT NOC will proactively contact all customers (even in the case of unmonitored customers) to notify them of any outages or issues that GTT’s suppliers see on their circuits.
5.2.1 // Outage Duration Calculation Outage duration is calculated from the time the customer reports the fault to the GTT NOC until the issue is resolved as confirmed by the customer. In the event that the customer is unable to confirm the exact uptime of a circuit/service, the GTT NOC will use the time from GTT NMS (Network Management System) or data from the underlying provider (switchless/unmonitored services)
GTT NOC Customer Support Guide // 9
5.2.2 // Corrective Actions and Follow-Up Tickets If during a trouble ticket, the GTT NOC identifies any issues with: ‣ ‣ ‣
Circuit design A deficiency with supplier’s fault troubleshooting process Any other preventative measures
The GTT NOC will 1.
Open an info ticket to implement those measures if they can be implemented by the GTT NOC. OR
2. Will capture the information in GTT’s CMD Database under Corrective Actions and will process a claim against the relevant team (i.e. GTT NOC, customer, supplier). GTT NOC management will then address those issues as part of GTT’s Supplier/Customer or NOC review.
5.2.3 // Chronic Ticket Procedure GTT’s Ticketing System automatically identifies Chronic Tickets based on the following criteria: ‣ ‣
3 P1 SMO/FNF Outages over two months 5 P2 SMO/FNF tickets over two months.
Chronic Tickets are automatically escalated to the GTT NOC Manager, who will liaise internally of with the relevant “off-net” provider. The GTT NOC will draw up an action plan to identify the root cause and implement preventative measures.
5.2.4 // Critical Outages Procedure Once the GTT NOC identifies an issue as a major outage (single event affecting multiple circuits), the GTT NOC activates the GTT Critical Outage Process (see 5.2.5) whereby the issue is escalated to the GTT NOC Manager immediately.
GTT NOC Customer Support Guide // 10
5.2.5 // Critical Outage Process Flow Chart Customer: Outage Notification Received
Supplier: Outage Notification Received
Major Outage
Critical Outage Process
Immediate Escalation NOC Manager Level 3
NOC Manager Assessment of impact for restoration of Backup Services
NOC Manager Escalates to VP Level 4 email distribution (1hr) VP of Operations updates customer and GTT Management
VP of Operations escalates to CEO Level 5 (3hrs)
GTT NOC Customer Support Guide // 11
6.0 // Fault Reporting 6.1 // Fault Reporting: Contacts All faults should be reported to the GTT NOC by the customer on record, as follows: Via telephone: GTT’s 24/7 NOC: +1 703 442 5539 (USA) or +44 207 489 4200 Via e-mail: Fault and Troubles or Change Requests:
Test and Turn-up: Maintenance: RFO: Fault Escalations:
[email protected] [email protected] [email protected] [email protected] [email protected]
Via GTT CMD Client Portal:
http://portal.gt-t.net
Clients can view and report tickets via the GTT portal. To request portal access, please contact your GTT sales account manager or email
[email protected].
6.2 // Fault Reporting: Requirements As a minimum, the following information should be provided to the GTT NOC when opening a ticket: ‣ ‣ ‣
GTT Circuit ID # (For provisioning issues, a PON # is also useful) Start time of the problem A brief description of the problem
Additional information may be required, depending on problem types as described in the following section.
GTT NOC Customer Support Guide // 12
6.2.1 // Trouble Tickets Additional information is needed when reporting certain types of trouble tickets. Please reference the guide lines below for the ticket types that require more information. Circuit Down Customer is responsible for: ‣ Confirming that equipment at both ends of the circuit has been checked and verified to be functioning correctly ‣ Providing POC/LCON details ‣ Providing Any supporting logs (Router interface logs) or Alarms (and their type) witnessed BGP Down Customer is responsible for: ‣ Providing the Serial IP as well as the circuit ID and location ‣ Providing all Router/Switch logs Latency Customer is responsible for: ‣ Providing trace routes / ping tests ‣ Providing any documentation or supporting graphs/monitoring logs to show historical trends and changes reported Packet Loss Customer is responsible for: ‣ Providing trace routes / ping tests ‣ Providing any documentation or supporting graphs/monitoring logs to show historical trends and changes reported Historic (Hits, BGP Down / Flap, Errors, Latency, Packet Loss) Customer is responsible for: ‣ Providing the start date and time of the reported incidence ‣ Providing the end date and time of the incidence ‣ Providing any supporting logs / documents ‣ Providing an account of what troubleshooting steps were taken by the client to resolve the issue/s being reported, if any
GTT NOC Customer Support Guide // 13
6.2.2 // Change Request Tickets Please report all change requests via e-mail to
[email protected], quoting the circuit ID and details of the changes requested. GTT aims to complete all network change requests within five (5) working days of the initial request. If a change request requires an order change with commercial / technical implications, the GTT NOC will refer the issues to the relevant GTT Account Manager.
6.2.3 // Mobility Service In addition to the GTT Service ID, please provide the following: ‣ Description of the problem ‣ Location where the problem was experienced ‣ Error codes received ‣ Error logs ‣ Date and time the problem was experienced ‣ If the issue was experienced by a single user, please include that user’s login name
6.2.4 // Remote Hands Requests In addition to the GTT Circuit ID, please provide the following: ‣ Location information for the equipment (i.e. rack number, server labels) ‣ A Clear description of the work that needs to be carried out ‣ Desired / Mandatory timeline associated with the engagement
7.0 // Calls 7.1 // Call Updates The GTT NOC will update customers regularly on all trouble tickets. The following time frame can be used as a guide for trouble ticket Updates: ‣ ‣ ‣ ‣
P1 Tickets Hourly unless otherwise agreed with customer P2 Tickets Updates at least every 4 hours P3 Tickets Updates at least every 24 hrs P4 Tickets Updates as and when required
* For P1 Mobility tickets the update intervals may vary from the above. GTT NOC Customer Support Guide // 14
Updates for change request tickets will be sent once the requested change is completed.
7.2 // GTT Call Status Explained Work In Progress (WIP) The GTT NOC is working on this ticket with its supplier. Customer Hold The GTT NOC is waiting for the customer to perform some actions to progress the issue. ‣
The GTT NOC will actively request updates from the customer. However, if the GTT NOC does not receive any feedback or the required information from the customer in question 48 hours after the call went into Customer Hold status, then the GTT NOC will automatically close the ticket.
Resolved – Customer Testing The GTT NOC has resolved the issue and customer is testing the circuit. ‣
The GTT NOC will close the ticket 24 hours after the call is placed on such status, unless the customer contacts the GTT NOC advising of any faults.
Resolved – Monitoring The GTT NOC has resolved the issue and is monitoring the service. ‣
The GTT NOC will close the ticket 24 hours after the call is placed on such status, unless the customer contacts the GTT NOC advising of a re-occurrence of the fault.
7.3 // Call Closure The GTT NOC will close the ticket in agreement with the customer after the issue has been resolved. If the client is not reachable after a reasonable effort from the GTT NOC and the GTT NOC believes the issue has been resolved and there are no follow up actions necessary, then the GTT NOC will close the ticket after two (2) working days and assign it one of the codes below. GTT Call Closure Codes Explained: SMO (Supplier Mis-Operation)
Outage cause attributed to GTT
CMO (Customer Mis-Operation)
Outage attributed to customer
FNF (Fault not found)
Fault cause could not be isolated
GTT NOC Customer Support Guide // 15
Sub-Sea Cut (submarine cable-cut) For unrestorable services FMJ (Force Majeure)
Outage outside of GTT’s control
Provisioning
Turn-up of new/migrated services
Planned Work
Scheduled/emergency network maintenance
Access
Access and miscellaneous causes
8.0 // Frequently Asked Questions POST‐INSTALL NETWORK SUPPORT 1. How do I create a trouble ticket? Call the GTT NOC 703 442 5539, or e‐mail us at:
[email protected]. Refer to the questions below regarding specific information that must be provided. 2. What information will I need to provide to GTT for technical support calls? a. GTT Circuit ID b. Contact Name c. Callback Number d. Email Address 3. What information will I need to provide for a reverse DNS request (pointer record (PTR))? GTT will need the IP and the fully qualified domain name (FQDN) to associate with it. 4. What information will I need to provide for Border Gateway Protocol (BGP) route additions? a. Neighbour IP b. ASN (Autonomous System Number) c. IP Prefix to add 5. What information will I need to provide for High Latency and/or Packet Loss? a. Serial Interface IP b. Email in trace routes in both directions c. Interface errors (if any are shown) d. Start time of the problem e. Identify if the problem is sporadic or continuous throughout the day f. Specify whether the problem is happening at the current time GTT NOC Customer Support Guide // 16
6. What information will I need to provide for Interface Errors? a. Type of device b. Duplex settings on the customer side c. Start time of the problem d. Know if the customer tried to change ports or optics 7. What information will I need to provide for BGP Flapping? a. Neighbour IP b. ASN (Autonomous System Number) c. Start time of the problem d. Identify if there is packet loss or latency e. Identify if there are any interface errors
CUSTOMER CHANGE ORDERS 1. How do I submit a change order? Call the GTT NOC toll‐free number, 703 442 5539, or email us at either:
[email protected]. 2. What information will I need to provide to GTT to add a Border Gateway Protocol (BGP) Prefix? a. GTT Circuit ID b. IP range to be added c. ASN of the range 3. What information will I need to provide to add Reverse DNS? a. GTT Circuit ID b. New PTR (pointer record) to be added 4. What information will I need to provide to get cage access? (colocation customers only) a. Location b. Cage or rack # c. Names of visitors d. Time of visit
GTT NOC Customer Support Guide // 17
REQUESTING A ‘REASON FOR OUTAGE’ 1. How do I submit a request for a Reason for Outage (RFO)? Call the GTT NOC toll‐free number, 866‐927‐2669, or email us at:
[email protected] 2. What information do I need to provide to get an RFO? a. GTT Circuit ID b. Time of outage c. GTT Ticket number
SLA CREDIT How do I request SLA Credit? e-mail
[email protected] with following details: Circuit ID/PON number GTT Outage Ticket Number GTT RFO Ticket number if one was opened.
ANY OTHER QUESTIONS / CHANGES? If you have any other queries (i.e. SLA on a service) or requests, please request those from your respective GTT Account Manager.
GTT NOC Customer Support Guide // 18
