NMS Redundancy and Failover iDX Release 3.3
Technical Notes
November 20, 2014
Copyright © 2014, Inc. All rights reserved. Reproduction in whole or in part without permission is prohibited. Information contained herein is subject to change without notice. The specifications and information regarding the products in this document are subject to change without notice. All statements, information and recommendations in this document are believed to be accurate, but are presented without warranty of any kind, express, or implied. Users must take full responsibility for their application of any products. Trademarks, brand names and products mentioned in this document are the property of their respective owners. All such references are used strictly in an editorial fashion with no intent to convey any affiliation with the name or the product's rightful owner. A Family of iDirect Companies
iDirect, a subsidiary of VT Systems, is a global leader in IP-based satellite communications providing technology that enables our partners to optimize their networks, differentiate and expand their businesses. The iDirect Intelligent Platform™ allows our partners to run their entire business operations more efficiently via a single, unified IP-based satellite architecture, whether it's providing core IP applications to the enterprise or specialized services to any number of diverse vertical markets. iDirect is the #1 name in global satellite communications in key industries including maritime, military/government, and oil and gas. iDirect Company web site: www.idirect.net ~ Main Phone: 703.648.8000 TAC Contact Information: Phone: 703.648.8151 ~ Email:
[email protected] ~ Web site: tac.idirect.net
iDirect Government Technologies (iGT), created in 2007, is a wholly owned subsidiary of iDirect and was formed to better serve the U.S. government and defense communities. iDirect Government Technologies™ Company web site: http: www.idirectgt.com ~ Main Phone: 703.648.8118 TAC Contact Information: Phone: 703.648.8111 ~ Email:
[email protected] ~ Web site: tac.idirectgt.com
iDirect Asia Pte Ltd was established in Singapore during 2008 to enhance iDirect’s value-add and responsiveness to customers in the Asia Pacific region. iDirect Asia Pte Ltd Company web site: www.stengg.com ~ Main Phone: 65.6521.7888 TAC Contact Information: Phone: 703.648.8151 ~ Email:
[email protected] ~ Web site: tac.idirect.net
Document Name: TN_NMS_Redundancy_3.3_Rev B_112014.pdf Document Part Number: T0000605
ii
NMS Redundancy and Failover iDX Release 3.3
Revision History
The following table shows all revisions for this document. To determine if this is the latest revision, check the TAC Web site. Refer to Getting Help on page x for TAC access information. Revision
Date
Updates
A
7/31/2014
Initial release of document.
B
11/20/2014
Updated the document template to use the Co-branded template. Replaced all Warning messages with Caution messages to adhere to the document convention. Corrected an error in a command syntax for backing up the NMS database.
Network Upgrade Procedure Guide for iDX Release 3.0.0.x and Later to iDX Release 3.3.x.x
iii
Revision History
iv
Network Upgrade Procedure Guide for iDX Release 3.0.0.x and Later to iDX Release 3.3.x.x
Contents
Figures
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . About
viii
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Purpose. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Document Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Getting Help . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x iDirect Contact Information: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x iGT Contact Information: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi Document Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
1 Single NMS Redundancy and Failover
. . . . . . . . . . . . . . . . . . . . . . . . . . .1
1.1 Configuring Single NMS Server Redundancy. . . . . . . . 1.1.1 Configuring the CRONTAB on the Servers . . . . . . . . 1.1.1 Configuring the CRONTAB on the Primary NMS . . 7. Configuring the crontab on the Backup NMS . . . . . . 1.1.2 Verify That the System Backup is Working . . . . . . .
... ... ... ... ...
... ... ... ... ...
... ... ... ... ...
... ... ... ... ...
.... .... .... .... ....
. . .1 . . .1 . . .2 . . .3 . . .3
1.2 Configuring Single NMS Failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2.1 Configuring the Network Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 Distributed NMS Redundancy and Failover . . . . . . . . . . . . . . . . . . . . . . . 9 2.1 Distributed NMS Redundancy and Replication . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.2 Configuring Distributed NMS Redundancy . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.2.1 Configuring the CRONTAB on the Servers . . . . . . . . . . . . . . . . . . . . . . . . . . 10
NMS Redundancy and Failover iDX Release 3.3
v
Contents
2.2.1 Configuring the crontab on the Primary NMS . . . . . . . . . . . . . . . . . . . . . 10 7. Configuring the crontab on NMS 2 and NMS 3 . . . . . . . . . . . . . . . . . . . . . . . 11 5. Configuring the crontab on the Backup NMS . . . . . . . . . . . . . . . . . . . . . . . . 12 2.2.2 Verify That the System Backup is Working . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3 Configuring Distributed NMS Failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.3.1 Preparing the Backup NMS Server to Take Over . . . . . . . . . . . . . . . . . . . . . . 13 2.3.2 Configuring Backup to Take Over for Primary . . . . . . . . . . . . . . . . . . . . . . . 14 2.3.3 Backup NMS Server Takes Over for an Auxiliary Server . . . . . . . . . . . . . . . . . 14
vi
NMS Redundancy and Failover iDX Release 3.3
Figures Figure 1-1. ifcfg-eth0 Results Display. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Figure 1-2. Network File Display . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Figure 1-3. Etc/Hosts File Display . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
NMS Redundancy and Failover iDX Release 3.3
vii
Tables Table 1-1. ifcfg-eth0 File Entries ........................................................................... 5 Table 1-2. Network File Commands ........................................................................ 7
NMS Redundancy and Failover iDX Release 3.3
viii
About
Purpose This technical note presents the following procedures: •
Configuring the crontab files for redundancy in a single NMS network.
•
Configuring the crontab files for redundancy in a distributed NMS network.
•
Bringing the Backup NMS server online in single NMS network.
•
Bringing the Backup NMS server online in a distributed NMS network.
Audience This technical note is written for network operators using the iDirect iDX Release 3.3 software suite (iBuilder/iMonitor/iSite), network architects, and any other personnel who operate or monitor an iDirect network. A basic knowledge of TCP/IP concepts, satellite communications, Linux and MS Windows operating systems and tools (including WinSCP and PuTTY) is assumed. Prior experience with the operation and monitoring of an iDirect network is desirable but not required.
Contents This document contains the following major sections: •
Single NMS Redundancy and Failover
•
Distributed NMS Redundancy and Failover
Document Conventions This section illustrates and describes the conventions used throughout this document. Convention Description Command
Used when the user is required to enter a command at a command line prompt or in a console.
NMS Redundancy and Failover iDX Release 3.3
Example Enter the command: cd /etc/snmp/
ix
About
Convention Description
Example
Terminal Output
Used when showing resulting output from a command that was entered at a command line or on a console.
crc report all
Screen Reference
Used when referring to text that appears on the screen on a Graphical User Interface (GUI).
1. To add a remote to an inroute group, right-click the Inroute Group and select Add Remote.
Used when specifying names of commands, menus, folders, tabs, dialogs, list boxes, and options. Hyperlink
Used to show all hyperlinked text within a document or external links such as web page URL.
8350.3235 : DATA CRC [ 1] 8350.3502 : DATA CRC [5818] 8350.4382 : DATA CRC [ 20]
The Remote dialog box has a number of userselectable tabs across the top. The Information tab is visible when the dialog box opens. For instructions on adding a line card to the network tree, see Adding a Line Card on page 108.
WARNING: A Warning highlights an essential operating or maintenance procedure, practice, condition, or statement which, if not strictly observed, could result in injury, death, or long term health hazards. CAUTION: A Caution highlights an essential operating or maintenance procedure, practice, condition, or statement which, if not strictly observed, could result in damage to, or destruction of, equipment or a condition that adversely affects system operation. NOTE: A Note is a statement or other notification that adds, emphasizes, or clarifies essential information of special importance or interest.
Getting Help The iDirect Technical Assistance Center (TAC) and the iDirect Government Technologies Technical Assistance Center (iGT TAC) are available to provide assistance 24 hours a day, 365 days a year. Software user guides, installation procedures, FAQs, and other documents that support iDirect products are available on the respective TAC Web site.
iDirect Contact Information: •
Web site: http://tac.idirect.net
•
Telephone: (703) 648-8151
•
E-mail:
[email protected]
For sales or product purchasing information contact iDirect Corporate Sales at:
x
•
Telephone: (703) 648-8000
•
E-mail:
[email protected]
NMS Redundancy and Failover iDX Release 3.3
About
iGT Contact Information: •
Web site: http://tac.idirectgt.com
•
Telephone: (703) 648-8111
•
E-mail:
[email protected]
iDirect and iGT produce documentation that is technically accurate, easy to use, and helpful to our customers. Please assist us in improving this document by providing feedback. Send comments to: •
iDirect:
[email protected]
•
iGT:
[email protected]
Document Set The following iDirect documents are available on the TAC Web site and contain information relevant to installing and using iDirect satellite network software and equipment. •
iDX 3.3 Release Notes
•
iDX 2.3 and Later to iDX 3.3 Network Upgrade Procedure
•
iDX iBuilder User Guide
•
iDX iMonitor User Guide
•
iDX Technical Reference Guide
•
iDX Satellite Router Commissioning and Installation Guide
•
iDX Features and Chassis Licensing Guide
•
iDX Upgrade/New Installation Survey
•
iDX Link Budget Analysis Guide
•
iDX Hardware Matrix
•
iDX Software Release Matrix
NMS Redundancy and Failover iDX Release 3.3
xi
About
xii
NMS Redundancy and Failover iDX Release 3.3
1 Single NMS Redundancy and Failover This chapter describes how to configure the Primary NMS to back up the databases and how to bring the Backup NMS server online in the event that the Primary NMS fails. It includes the following major topics: •
Configuring Single NMS Server Redundancy
•
Configuring Single NMS Failover
Beginning with iDX Release 3.1, iDirect supports near real-time replication of NMS databases from the Primary NMS server to one or more Backup NMS servers in addition to the traditional idsBackup and idsRestore. When the NMS Database Replication feature is enabled, changes to the NMS databases on the Primary NMS are automatically replicated to the Backup NMS server and the idsBackup script is automatically configured to run on the Backup server. For a complete description of the NMS Database Replication feature, including configuration, see the Technical Reference Guide for iDX Release 3.3. If the NMS Database Replication feature is currently being used, the sections in this document on configuring single and distributed NMS redundancy are not applicable. However, there are slight differences to the procedures to bring the Backup NMS server on line in case of a Primary NMS failure. These differences are noted in Configuring Single NMS Failover on page 4 and in Configuring Distributed NMS Failover on page 13.
1.1
Configuring Single NMS Server Redundancy This section describes how to configure redundancy in a single NMS network. NOTE: This procedure is not applicable if NMS Database Replication feature is enabled on the NMS Servers.
1.1.1 Configuring the CRONTAB on the Servers The crontab file must be configured in order to correctly run the idsBackup and idsRestore scripts.
NMS Redundancy and Failover iDX Release 3.3
1
Configuring Single NMS Server Redundancy
Configuring the CRONTAB on the Primary NMS Perform the following on the Primary NMS: 1. Log on to the Primary NMS and switch to root. 2. Edit the crontab file by entering the command: crontab -e
The crontab file opens in the Vi editor. 3. On the Primary NMS, remove the comment symbol (#) from the beginning of all lines in the crontab file, except for this line: #30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date -keep 2
4. Add the following line to the crontab file, replacing the IP address shown with the Backup NMS server’s address, and, if desired, substituting the filename tokens shown with your own: 30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --inst
[email protected]:/var/idirect/backup/\%ip-\%date --keep 2
NOTE: The idsBackup command creates a local backup, and when using the “--inst” option, it creates a remote backup and installs it on the specified NMS. NOTE: A forward slash character (\) is required before the token “%” character for the crontab to execute properly. For example, use “\%ip” and not “%ip” 5. Optionally, change the time and frequency with which backups are performed by modifying the string 30 0 * * * (highlighted above). This is a five-field string set by default to run a backup every night at 12:30am. The time and frequency of the backups can be modified by changing these fields according to the following definitions. The fields below are numbered left to right. • Field 1 (00 by default): Minute of the specified Hour (0 - 59) •
Field 2 (1 by default): Hour of the Day (0 - 23)
•
Field 3 (* by default): Day of the Month (1 - 31)
•
Field 4 (* by default): Month (1 - 12)
•
Field 5 (* by default): Day of the week (0 - 6, with Sunday = 0)
6. Optionally, the number of local backup copies kept can be changed by modifying --keep 2, for example, to --keep 1 to save disk space. 7. Save changes to the file and exit by pressing Esc and entering: :wq!
An example of an edited crontab file for the Primary NMS is shown below. 0 0 * * * /opt/idirect/nms/utils/db_maint/cons.pl >> /opt/idirect/nms/utils/db_maint/cons.output 2>&1 0 4 * * 6 /opt/idirect/nms/utils/db_maint/check_db.pl >> /opt/idirect/nms/utils/db_maint/check_db.output 2>&1 #30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date -keep 2
2
NMS Redundancy and Failover iDX Release 3.3
Configuring Single NMS Server Redundancy
30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --inst
[email protected]:/var/idirect/backup/\%ip-\%date --keep 2
Configuring the crontab on the Backup NMS Perform the following steps on the Backup NMS: 1. Log on to the Backup NMS and switch root. 2. Edit the crontab file by entering the command: crontab -e
The crontab file opens in the Vi editor. 3. Remove the comment symbol (#) from the following line: 0 4 * * 6 /opt/idirect/nms/utils/db_maint/check_db.pl >> /opt/idirect/nms/utils/db_maint/check_db.output 2>&1
This command performs a recommended, daily maintenance check on the database. 4. Ensure comment symbols (#) appear in front of the following lines: #0 0 * * * /opt/idirect/nms/utils/db_maint/cons.pl >> /opt/idirect/nms/utils/db_maint/cons.output 2>&1 #30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date -keep 2
5. Save changes to the file and exit by pressing Esc and entering: :wq!
An example of an edited crontab file for the Backup NMS is shown below: #0 0 * * * /opt/idirect/nms/utils/db_maint/cons.pl >> /opt/idirect/nms/utils/db_maint/cons.output 2>&1 0 4 * * 6 /opt/idirect/nms/utils/db_maint/check_db.pl >> /opt/idirect/nms/utils/db_maint/check_db.output 2>&1 #30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date -keep 2
1.1.2 Verify That the System Backup is Working This procedure provides all necessary steps to verify the system backup script is working condition and that the appropriate backups are made on the Backup NMS server. The CRONTAB time setting can be modified to run the commands 1-3 minutes after the current time. Be sure to check the idsBackup.log to see if the backup and install is running properly. Perform the following: 1. On the Primary NMS, log on as idirect and switch to root. 2. Enter the following command to push SSH keys to the Backup NMS: pushSSHKeys
The following sample output displays: Pushing SSH key for user root to
[email protected]:
NMS Redundancy and Failover iDX Release 3.3
3
Configuring Single NMS Failover
The authenticity of host ’172.20.4.253 (172.20.4.253)’ can’t be established. RSA key fingerprint is 3e:28:f2:a5:50:22:a5:2f:51:33:69:39:ca:a4:8d:63. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added ’172.20.4.253’ (RSA) to the list of known hosts.
[email protected]’s password: remote_users_password SSH key for user root successfully pushed to
[email protected]
3. On the Primary NMS, get the current time. date Tue Aug 31 15:21:45 UTC 2010
4. In the crontab file, set the idsBackup command to run two minutes from the current time. crontab -e 0 0 * * * /opt/idirect/nms/utils/db_maint/cons.pl >> /opt/idirect/nms/utils/db_maint/cons.output 2>&1 0 4 * * 6 /opt/idirect/nms/utils/db_maint/check_db.pl >> /opt/idirect/nms/utils/db_maint/check_db.output 2>&1 #30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date -keep 2 23 15 * * * /usr/bin/idsBackup --inst
[email protected]:/var/idirect/backup/\%ip-\%date --keep 2
5. Check the idsBackup.log to monitor if the commands ran as expected. tail –f /var/idirect/log/idsBackup.log
6. Restore the original settings in the crontab file.
1.2
Configuring Single NMS Failover NOTE: This procedure is not applicable if NMS Database Replication feature is enabled on the NMS Servers. This procedure describes how to bring a Backup NMS on line if the Primary NMS fails. Perform the following: 1. If the primary NMS is still accessible, reconfigure the primary NMS eth0 interface with another unused IP address and physically unplug the eth0 interface. If the interface has been reconfigured, the NMS server must be rebooted. Restart the network services by entering the following command at the prompt: service network restart
2. If the primary NMS remains connected to the network, the iDirect NMS services MUST be shutdown by entering the following commands and the IP address must be different. Additionally, these services MUST remain down while the other new (backup) NMS server is online. service idirect_nms stop
4
NMS Redundancy and Failover iDX Release 3.3
Configuring Single NMS Failover
service idirect_nms stop nms_config cd /etc/init.d chkconfig --level 2345 idirect_nms off
3. Log on to the Backup NMS server as idirect and switch to root. 4. Verify that the routing configuration on the Backup NMS server is correct. Any static routes that were configured on the Primary NMS server should also be configured on the Backup NMS server. Make sure they are configured as persistent routes so they remain in effect after a reboot. 5. If NMS Database Replication is enabled, enter the following command on the Backup NMS server to stop the server from being a MySQL slave: cr8DbSlave -d
6. Proceed to Configuring the Network Scripts.
1.2.1 Configuring the Network Scripts Perform the following steps to configure the network interface configuration: 1. Go to the /etc/sysconfig/network-scripts directory. cd /etc/sysconfig/network-scripts
2. Using the Vi editor, open the ifcfg-eth0 file for editing by entering the following command: vi ifcfg-eth0
3. Change the command lines in the ifcfg-eth0 file as shown in Table 1-1 (do not change any other values, and retain all other lines within this file). NOTE: Command lines are case-sensitive.
Table 1-1. ifcfg-eth0 File Entries Command
Description
DEVICE
eth0
BOOTPROTO
none
ONBOOT
yes
IPADDR
upstream IP address of the Primary NMS server
NETMASK
subnet mask of the NMS server
GATEWAY
IP address of the upstream router interface
MTU
1500
An example of a correctly configured network-script file for eth0 is shown in Figure 1-1.
NMS Redundancy and Failover iDX Release 3.3
5
Configuring Single NMS Failover
Figure 1-1. ifcfg-eth0 Results Display CAUTION: Do not delete any lines from the existing file. Doing so may have an adversed effect on network performance. 4. Record the hardware address (HWADDR) as displayed in Figure 1-1. 5. Save the file and exit the Vi editor. :wq!
6. Contact the TAC to see if a new license file is needed. Typically, the primary and backup NMS server MAC addresses are listed in the license file, so a new license file is not required. 7. Return to the /etc/sysconfig directory. cd /etc/sysconfig
8. Using the vi editor, open the network file for editing by entering the following command: vi network
The network file displays as shown in the example in Figure 1-2.
Figure 1-2. Network File Display
6
NMS Redundancy and Failover iDX Release 3.3
Configuring Single NMS Failover
9. Change the command lines in the network-scripts file as shown in Table 1-2 (do not change any other values, and retain all other lines within this file). Table 1-2. Network File Commands Command
Description
NETWORKING
yes
HOSTNAME
Company_Name_NMS_Primary
GATEWAYDEV
eth0
10. Save the file and exit the editor by entering the following command: :wq!
11. Go to the /etc directory by entering the following command: cd /etc
12. Using the vi editor, open the hosts file for editing by entering the following command: vi hosts
13. Add the following lines to the hosts file, as applicable to the IP addressing scheme being implemented. The two lines for 127.0.0.1 must be present. Enter the following command lines: 127.0.0.1 COMPANY_NMS_PRIMARY localhost.localdomain localhost 192.168.1.3 COMPANY_NMS_PRIMARY
The hosts file displays as shown in the example in Figure 1-3.
Figure 1-3. Etc/Hosts File Display 14. Save the file and exit the editor by entering the following command: :wq!
15. Configure the NMS services to start automatically at boot time by entering the following command: chkconfig idirect_nms on
16. Reboot the Backup NMS.
NMS Redundancy and Failover iDX Release 3.3
7
Configuring Single NMS Failover
The backup NMS becomes primary upon boot up. It may take a few minutes for the upstream router to update the ARP tables. After this update is complete, IP connectivity to the server is restored.
8
NMS Redundancy and Failover iDX Release 3.3
2 Distributed NMS Redundancy and Failover Special consideration is required for Hubs that are using a Distributed NMS. If the current system is a Distributed NMS, then additional configuration changes must be made on the Primary NMS server, Auxiliary servers, and Backup server (event server (evtsvr), latency server (latsvr), and the archive server (nrdsvr)). This chapter includes the following major topics:
2.1
•
Distributed NMS Redundancy and Replication
•
Configuring Distributed NMS Redundancy
•
Configuring Distributed NMS Failover
Distributed NMS Redundancy and Replication Beginning with iDX Release 3.1, iDirect supports near real-time replication of NMS databases from the Primary NMS server to one or more Backup NMS servers in addition to the traditional idsBackup and idsRestore. When enable the NMS Database Replication feature, changes to the NMS databases on the Primary NMS are automatically replicated to the Backup NMS server and the idsBackup script is automatically configured to run on the Backup server. On a distributed NMS with NMS Database Replication enabled, each NMS server replicates its database to a different backup machine. Assuming a standard DNMS setup with all databases being replicated, the system consists of one Primary NMS server (NMS 1), two Auxiliary NMS servers (NMS 2 and NMS 3), and three Backup NMS servers. During normal operation, each of the three on line Servers (NMS 1, NMS 2 and NMS 3) replicates its database to the corresponding backup server. (For details, see “NMS Database Replication on a Distributed NMS” in the “NMS Database Replication” chapter of the Technical Reference Guide for iDX Release 3.3) If a Primary or an Auxiliary NMS server fails, it is possible to switch to the corresponding Backup NMS server by disabling the failed server and reconfiguring the Backup server to take its place. This procedure is identical to the procedure for a Single NMS failure. Therefore, follow the procedure in Configuring Single NMS Failover on page 4. NOTE: Do not forget to execute Step 5 of Section 1.2.1, Configuring the Network Scripts on page 5 to disable Database Replication feature on the Backup NMS server before bringing it up as the active server. NOTE: If NMS Database Replication feature is enable on the Distributed NMS servers, then the remaining procedures in this chapter are not applicable.
NMS Redundancy and Failover iDX Release 3.3
9
Configuring Distributed NMS Redundancy
For a complete description of the NMS Database Replication feature see the Technical Reference Guide for iDX Release 3.3.
2.2
Configuring Distributed NMS Redundancy This section describes how to configure redundancy in a distributed NMS network. CAUTION: Do not perform this procedure if Database Replication feature is enable on the NMS servers. See Distributed NMS Redundancy and Replication on page 9.
2.2.1 Configuring the CRONTAB on the Servers The crontab file must be reconfigured in order to correctly run the idsBackup and idsRestore scripts.
Configuring the crontab on the Primary NMS On the Primary NMS, perform the following: 1. Log on to the Primary NMS and switch to the root account. 2. Edit the crontab file by entering the command: crontab -e
The crontab file opens in the vi editor. 3. On the Primary NMS, remove the comment symbol (#) from the beginning of all lines in the crontab file, except for this line: #30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date -keep 2
4. Add the following line to the crontab file, replacing the IP address shown with the Backup NMS server’s address, and, if desired, substituting the filename tokens shown with your own: 30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --inst
[email protected]:/var/idirect/backup/\%ip-\%date --keep 2
NOTE: The idsBackup command creates a local backup, and when using the “inst” option, it creates a remote backup and installs it on the specified NMS. NOTE: A forward slash character (\) is required before the token “%” character for the crontab to execute properly. For example, use “\%ip” and not “%ip” 5. Optionally, change the time and frequency with which backups are performed by modifying the string 30 0 * * * (highlighted above). This is a five-field string set by default to run a backup every night at 12:30am. The time and frequency of the backups can be modified by changing these fields according to the following definitions. The fields below are numbered left to right. • Field 1 (00 by default): Minute of the specified Hour (0 - 59)
10
NMS Redundancy and Failover iDX Release 3.3
Configuring Distributed NMS Redundancy
•
Field 2 (1 by default): Hour of the Day (0 - 23)
•
Field 3 (* by default): Day of the Month (1 - 31)
•
Field 4 (* by default): Month (1 - 12)
•
Field 5 (* by default): Day of the week (0 - 6, with Sunday = 0)
6. Optionally, the number of local backup copies kept can be changed by modifying --keep 2, for example, to --keep 1 to save disk space. 7. Save changes to the file and exit by pressing Esc and entering: :wq!
An example of an edited crontab file for the Primary NMS is shown below. 0 0 * * * /opt/idirect/nms/utils/db_maint/cons.pl >> /opt/idirect/nms/utils/db_maint/cons.output 2>&1 0 4 * * 6 /opt/idirect/nms/utils/db_maint/check_db.pl >> /opt/idirect/nms/utils/db_maint/check_db.output 2>&1 #30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date -keep 2 30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --inst
[email protected]:/var/idirect/backup/\%ip-\%date --keep 2
Configuring the crontab on NMS 2 and NMS 3 A distributed NMS has a special backup configuration that must be configured in the crontab file. Perform the following steps on NMS 2 and NMS 3 of the Distributed NMS: 1. Log on to the Auxiliary NMS (NMS 2 or NMS 3) as idirect and switch to the root account. 2. Edit the crontab file by entering the command: crontab -e
The crontab file opens in the Vi editor. 3. On the Auxiliary NMS, remove the comment symbol (#) from the beginning of all lines in the crontab file, except for this line: #30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date -keep 2
4. Add the following line to the crontab file, replacing the IP address shown with the NMS server’s address, and, if desired, substituting the filename tokens shown with your own: 30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --bkup
[email protected]:/var/idirect/backup/\%ip-\%date --keep 2
5. Save changes to the file and exit by pressing Esc entering: :wq!
An example of an edited crontab file is shown below. 0 0 * * * /opt/idirect/nms/utils/db_maint/cons.pl >> /opt/idirect/nms/utils/db_maint/cons.output 2>&1 0 4 * * 6 /opt/idirect/nms/utils/db_maint/check_db.pl >> /opt/idirect/nms/utils/db_maint/check_db.output 2>&1
NMS Redundancy and Failover iDX Release 3.3
11
Configuring Distributed NMS Redundancy
#30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date -keep 2 30 2 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date --bkup
[email protected]:/var/idirect/backup/\%ip-\%date --keep 2
Configuring the crontab on the Backup NMS Perform the following: 1. Log on to the Backup NMS and switch to the root account. 2. Edit the crontab file by entering the command: crontab -e
The crontab file opens in the Vi editor. 3. Remove the comment symbol (#) from the following line: 0 4 * * 6 /opt/idirect/nms/utils/db_maint/check_db.pl >> /opt/idirect/nms/utils/db_maint/check_db.output 2>&1
This command performs a recommended, daily maintenance check on the database. An example of an edited crontab file for the Backup NMS is shown below. #0 0 * * * /opt/idirect/nms/utils/db_maint/cons.pl >> /opt/idirect/nms/utils/db_maint/cons.output 2>&1 0 4 * * 6 /opt/idirect/nms/utils/db_maint/check_db.pl >> /opt/idirect/nms/utils/db_maint/check_db.output 2>&1 #30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date -keep 2
2.2.2 Verify That the System Backup is Working This procedure provides all necessary steps to verify the system backup script is working and that the appropriate backups are made on the Backup NMS server. The time in crontab can be modified to run the commands 1-3 minutes after the current time, and then check the idsBackup.log to see if the backup and install is running properly. 1. Log on to the Primary NMS and switch to the root account. 2. Enter the following command to push SSH keys to the Backup NMS: pushSSHKeys
The following sample output displays: Pushing SSH key for user root to
[email protected]: The authenticity of host ’172.20.4.253 (172.20.4.253)’ can’t be established. RSA key fingerprint is 3e:28:f2:a5:50:22:a5:2f:51:33:69:39:ca:a4:8d:63. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added ’172.20.4.253’ (RSA) to the list of known hosts.
[email protected]’s password: remote_users_password SSH key for user root successfully pushed to
[email protected]
12
NMS Redundancy and Failover iDX Release 3.3
Configuring Distributed NMS Failover
3. Get the current time. date Tue Aug 31 15:21:45 UTC 2010
4. In the crontab file, set the idsBackup command to run two minutes from the current time. crontab -e 0 0 * * * /opt/idirect/nms/utils/db_maint/cons.pl >> /opt/idirect/nms/utils/db_maint/cons.output 2>&1 0 4 * * 6 /opt/idirect/nms/utils/db_maint/check_db.pl >> /opt/idirect/nms/utils/db_maint/check_db.output 2>&1 #30 0 * * * /usr/bin/idsBackup --bkup /var/idirect/backup/\%ip-\%date -keep 2 23 15 * * * /usr/bin/idsBackup --inst
[email protected]:/var/idirect/backup/\%ip-\%date --keep 2
5. Check the idsBackup.log to monitor if the commands executed successfully. tail –f /var/idirect/log/idsBackup.log
6. Restore the original settings in the crontab file. 7. In a Distributed NMS configuration, repeat Step 1 through Step 6 on each Auxiliary NMS. NOTE: On the Auxiliary NMS servers, the line to edit ends in --bkup rather than --inst as shown below. 23 15 * * * /usr/bin/idsBackup --bkup
2.3
Configuring Distributed NMS Failover This section describes how to have the Backup NMS server takes over for either the Primary NMS Server or for an Auxiliary NMS Server. CAUTION: Do not perform this procedure if Database Replication feature is enable on the NMS servers. See Distributed NMS Redundancy and Replication on page 9.
2.3.1 Preparing the Backup NMS Server to Take Over Perform the following to prepare for the Backup NMS to take over: 1. If the primary NMS is still accessible, reconfigure the primary NMS eth0 interface with another unused IP address and physically unplug the eth0 interface. If the interface has been reconfigured, the NMS server must be rebooted. Additionally, the network services can be restarted by entering the following command at the prompt: service network restart
2. If the primary NMS remains connected to the network, the iDirect NMS services MUST be shut down by entering the following command and the IP address must be different. Additionally, these services MUST remain down while the other new (backup) NMS server is online. service idirect_nms stop service idirect_nms stop nms_config
NMS Redundancy and Failover iDX Release 3.3
13
Configuring Distributed NMS Failover
cd /etc/init.d chkconfig --level 2345 idirect_nms off
3. Verify that the routing configuration on the Backup NMS server is correct. If RIPv2 was running on the Primary NMS server, it has to be running on the backup. Any static routes that were configured on the Primary NMS Server need to be configured on the Backup NMS server. Make sure they are configured as persistent routes so they remain in effect after a reboot.
2.3.2 Configuring Backup to Take Over for Primary To bring the Backup NMS server on-line to replace the Primary server, perform the following procedures.
Procedure 1. Edit the eth0 Configuration Configure the Backup NMS Server eth0 interface with the primary NMS original eth0 interface address. The reason for this is to match the original primary NMS IP address that is configured in the configuration (options) files of the iDirect network elements. 1. Log on to the Backup NMS and switch to the root account. 2. Change the IP address and hostname, and configure the NMS services to start automatically at boot time by performing Configuring the Network Scripts on page 5. 3. Rebuild the Distributed NMS configuration. Refer to Appendix A of the iBuilder User Guide (Section A.4 “Setting UP a Distributed NMS Environment” through Section A.6, “Granting Read Permissions to NMS 2 and NMS 3”). 4. The network should now be running on the Backup NMS Server. The status on the Line Cards and the Remotes should be normal in iMonitor. 5. If needed, apply the Protocol Processor level and network level configuration to all networks via iBuilder.
2.3.3 Backup NMS Server Takes Over for an Auxiliary Server To bring the backup NMS server on-line in as an Auxiliary server, perform the following: 1. Log on to the Backup NMS and switch to the root account. 2. Find the restore file by entering the following commands to change to the backup directory and list the files: cd /var/idirect/backup ll
This example shows a list of files in the backup directory:
14
-rw-r--r-- 4 idirect 2012.02.27-00.30.01
idirect
49519268 Feb 27 00:31 192.168.76.81-
-rw-r--r-- 4 idirect 2012.02.28-00.30.01
idirect
49653853 Feb 28 00:31 192.168.76.81-
-rw-r--r-- 4 idirect 2012.02.27-00.30.01
idirect
45100228 Feb 27 00:31 192.168.76.82-
NMS Redundancy and Failover iDX Release 3.3
Configuring Distributed NMS Failover
-rw-r--r-- 4 idirect 2012.02.28-00.30.01
idirect
45100803 Feb 28 00:31 192.168.76.82-
-rw-r--r-- 4 idirect 2012.02.27-00.30.01
idirect
29449298 Feb 27 00:31 192.168.76.83-
-rw-r--r-- 4 idirect 2012.02.28-00.30.01
idirect
29443853 Feb 28 00:31 192.168.76.83-
3. Identify the backup file to restore, and enter the following command: /usr/bin/idsRestore --bkup /var/idirect/backup/
Where, is the name of the backup file. When this commands executes, the idsRestore script runs and the Backup NMS server becomes the Auxiliary server. In this example, it becomes NMS2. During the takeover process, the IP address is changed to that of the Auxiliary server. It may take a few minutes for the Upstream router ARP tables to update. When the update is complete, IP connectivity to the server is restored. 4. Configure the NMS services to start automatically at boot time by entering the following commands: cd /etc/init.d chkconfig idirect_nms on
5. Reboot the Backup NMS. 6. If needed, apply the Protocol Processor level and network level configuration to all networks via iBuilder. 7. Ensure that the backup is working by performing Verify That the System Backup is Working on page 12.
NMS Redundancy and Failover iDX Release 3.3
15
Configuring Distributed NMS Failover
16
NMS Redundancy and Failover iDX Release 3.3
13861 Sunrise Valley Drive, Suite 300 Herndon, VA 20171 +1 703.648.8000 +1 866.345.0983 www.idirect.net Advancing a Connected World
