NMS Redundancy and Failover iDX Release 2.0 and Earlier
Technical Note
October 1, 2008
Copyright © 2008 VT iDirect, Inc. All rights reserved. Reproduction in whole or in part without permission is prohibited. Information contained herein is subject to change without notice. The specifications and information regarding the products in this document are subject to change without notice. All statements, information, and recommendations in this document are believed to be accurate, but are presented without warranty of any kind, express, or implied. Users must take full responsibility for their application of any products. Trademarks, brand names and products mentioned in this document are the property of their respective owners. All such references are used strictly in an editorial fashion with no intent to convey any affiliation with the name or the product's rightful owner.
Document Name: TN_NMS_Redundancy_and_Failover_Rev F_01012008.pdf Document Part Number: T00000052
ii
NMS Redundancy and Failover
Revision History
The following table shows all revisions for this document. Refer to this information to verify that you have the latest version. Rev
Date Released
Reason for Change(s)
Who Updated?
A
August 7, 2006
Updated for new release.
E Hoffman
B
August 11, 2006
Updated for new release.
E Hoffman
C
September 25, 2006
Updated for new release.
E Hoffman
D
November 16, 2007
Updated for new release.
E Hoffman
E
April 21, 2008
Updated based on TAC feedback changes.
E Hoffman
F
October 1, 2008
Updated based on TAC feedback changes.
S Murphy
NMS Redundancy and Failover
iii
iv
NMS Redundancy and Failover
Contents About This Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .vii Purpose. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii Intended Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii Contents Of This Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii Document Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii iDX Related Documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii iDS Related Documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Hardware-Related Documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Getting Help . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x
1 NMS Redundancy Levels . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 Process Redundancy
.................................. 1
1.2 RAID-1 Disk Drive Redundancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
2 Configuring the dbBackup Script Parameters . . . . . . . . . . 3 2.1 Setting Up SSH between Primary and Backup NMS Servers
........ 3
2.2 Configuring The NMS to Send Database Backups to The Backup NMS . . 4 2.3 Defining dbBackup Script Default Behavior . . . . . . . . . . . . . . . . . . . 4 2.3.1 Changing the Number of Backup Copies to Save . . . . . . . . . . . . . . . . . . . 5 2.3.2 Changing Database Backups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3 Special Provisioning for a Distributed NMS . . . . . . . . . . . . 7 3.1 Configuring the Primary NMS Server . . . . . . . . . . . . . . . . . . . . . . . . 7 3.2 Configuring the Auxiliary NMS Servers . . . . . . . . . . . . . . . . . . . . . . . 8 3.3 Configuring the Backup NMS Server . . . . . . . . . . . . . . . . . . . . . . . . 8
NMS Redundancy and Failover
v
4 Configuring dbRestore Script Parameters . . . . . . . . . . . . .11 4.1 Configuring the Backup NMS to Restore the Databases . . . . . . . . . . 11
5 Changing Over to the Backup NMS Server . . . . . . . . . . . . .13 5.1 Procedures for Moving Component Files to the Backup NMS Server . . 13 Logging Into The Primary NMS Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Compressing the Component Options File Directories . . . . . . . . . . . . . . . . . . . . . 14 Copying the Component Options File Directories from the Primary NMS Server to Backup NMS Server 14 Extracting Component File Directories Onto Backup NMS Server . . . . . . . . . . . . . . 15
5.2 Procedures for Moving pp_controller Files To the Backup NMS Server 16 Logging Into The Primary NMS Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Compressing the pp_controller Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Copying the pp_controller Files from the Primary NMS Server to Backup NMS Server
17
Extracting the pp_controller Files Onto Backup NMS Server . . . . . . . . . . . . . . . . . 17
5.3 Bringing The Backup NMS Online . . . . . . . . . . . . . . . . . . . . . . . . . 19 Changing The Backup NMS Server To The Primary NMS Server . . . . . . . . . . . . . . . 20
vi
NMS Redundancy and Failover
About This Guide
Purpose The purpose of this document is to explain how to set up and automated backup/restore mechanism for your NMS configuration and/or archve databases.
Intended Audience This guide is written for network operators using the iDirect iDS iBuilder/iMonitor/iSite software suite, network architects, and any other personnel who operate or monitor an iDirect network. A basic knowledge of TCP/IP concepts, satellite communications, Linux and MS Windows operating systems and tools (including WinSCP and PuTTY) is assumed. Prior experience with the operation and monitoring of an iDirect network is desirable but not required.
Contents Of This Guide This document contains the following:
Document Conventions This section illustrates and describes the conventions used throughout the manual. Take a look now, before you begin using this manual, so that you’ll know how to interpret the information presented. Convention Description Blue Courier font
Used when the user is required to enter a command at a command line prompt or in a console)
NMS Redundancy and Failover
Example [SWITCH_PORT_n] vid = vlan_id
vii
iDX Related Documents
Courier font
Used when showing resulting output from a command that was entered at a command line or on a console.
Output similar to the following sample appears: [SECURITY] password = $idi2$/bFMhf$5H8mYAaP1sTZ0m1Ny/dYyLaS40/ admin_password = $idi2$146rgm$.KtDb4OH5CEBxzH6Ds2xM.ehHCH os_password = $1$UTKh0V$cc/UfNThFmBI7sT.zYptQ0
Bold Trebuchet font
Used when the user is required to type information or values into a field within a windows-type interface software.
1. If you are adding a remote to an inroute group, right-click the Inroute Group and select Add Remote. The Remote dialog box has a number of userselectable tabs across the top. The Information Tab is visible when the Dialog opens.
Used when specifying names of commands, menus, folders, tabs, dialogs, list boxes, and options. Blue Trebuchet font
Used to show all hyperlinked text within a document.
For instructions on adding an iSCPC line card to the network tree and selecting a Hub RFT for the line card, see “Adding an iSCPC Line Card” on page 108.
Bold italic Trebuchet font
Used to emphasize information for the user, such as in notes.
Note:
Red italic Trebuchet font (or see table below)
Used when the user needs to STRICTLY follow the instructions or have additional knowledge about a procedure or action.
Several remote model types can be configured as iSCPC remotes.
WARNING! The following procedure may cause a network outage.
iDX Related Documents Note:
The following list of documents applies to iDX Release 1.0 and up.
The following iDirect documents are available at http://tac.idirect.net and may also contain information relevant to this release. Please consult these documents as needed or indicated within this guide.
viii
•
iDX Release Notes
•
iDX Software Installation Guide
•
iDX iBuilder User Guide
•
iDX iMonitor User Guide
•
iDX Technical Reference Guide
•
iDX Software Installation Survey
NMS Redundancy and Failover
iDS Related Documents
•
iDirect Hub Readiness Checklist
•
Teleport Design Considerations
•
Hub Installation Guide
•
iDX Satellite Router Installation & Commissioning Guide
•
iDirect Hub Installation As-Built Template
•
iDirect Acceptance Test Plan
•
iDX 1.0.1 Link Budget Analysis Guide
iDS Related Documents Note:
The following list of documents applies to iDS and below!!
The following iDirect documents are available at http://tac.idirect.net and may also contain information relevant to this release. Please consult these documents as needed or indicated within this guide. •
iDS Release Notes
•
iDS Network Upgrade Procedure Guide
•
iDS NMS iBuilder User Guide
•
iDS NMS iMonitor User Guide
•
iDirect Technical Reference Guide
•
iDS Upgrade Survey
•
iDirect Hub Readiness Checklist
•
iDirect Teleport Design Considerations
•
iDirect Hub Installation Guide
•
iDX Remote Installation & Commissioning Guide
•
iDirect Hub Installation As-Built Template
•
iDirect Acceptance Test Plan
•
Link Budget Analysis Guide
Hardware-Related Documents Note:
This list is ONLY for the hardware-related Manuals
The following iDirect documents are available at http://tac.idirect.net and may also contain information relevant to this release. Please consult these documents as needed or indicated within this guide. •
iBuilder User Guide
•
iMonitor User Guide
•
Technical Reference Guide
•
iDirect Software Installation Guide (iDX) OR Network Upgrade Procedure Guide (iDS)
•
iDirect Hub Readiness Checklist
NMS Redundancy and Failover
ix
Getting Help
•
iDirect Teleport Design Considerations
•
iDirect Hub Installation Guide
•
Remote Installation & Commissioning Guide
•
iDirect Hub Installation As-Built Template
•
iDirect Acceptance Test Plan
Getting Help The iDirect Technical Assistance Center (TAC) is available to help you 24x7x365. iDS Software user’s guides, installation procedures, an FAQ page, and other documentation that supports our products are available on the TAC webpage. Please access our TAC webpage at: http://tac.idirect.net. If you are unable to find the answers or information that you need, you can contact the TAC at (703) 648-8151.
x
NMS Redundancy and Failover
1 NMS Redundancy Levels
The NMS Redundancy and Failover mechanism, when set up on your network, automatically backs up your primary NMS databases every night and restores them onto the Backup NMS Server. In the event of a catastrophic NMS Primary Server failure, you can switch to your Backup NMS Server with the assurance that most, if not all, of your data remains in tact. The iDirect Technologies Network Management System (NMS) has a number of redundancy levels that ensure robust operation, each covering a particular point of potential failure.
1.1
Process Redundancy The standard NMS installation includes a daemon1 process called nms_monitor. This process monitors all of the other NMS server processes on a constant basis. If any of these processes terminate abnormally, nms_monitor automatically restarts the process. To view the status of the NMS monitor and server processes, perform the following: 1. Logon to the NMS as root. 2. At the # prompt, enter the following command: service idirect_nms status
The output indicates the processes as either stopped or running. The event history of the NMS monitor can be viewed in the nms_monitor.log file. Each event is listed with a date and time stamp. This file is located in the /home/nms/utils/ directory. Optionally, nms_monitor can send email to a designated recipient indicating the restart condition. The email capability requires Linux sendmail to be configured on the NMS server. iDirect recommends setting up sendmail only if your NMS Server is behind a firewall.
1.2
RAID-1 Disk Drive Redundancy The NMS IBM server is configured with redundant RAID-1 disk drives. In a RAID-1 configuration, all data is written simultaneously to both drives. In the event that one drive fails, the second drive takes over and there is no loss of data. The NMS does not flag this situation, but a red LED displays on the front panel of the server to indicate the disk problem. 1. A daemon (or service) is a background process that is designed to run autonomously, with little or not user intervention. For example, The Apache web server http daemon (httpd) is one such example of a daemon. It waits in the background listening on specific ports, and serves up pages or processes scripts, based on the type of request.
NMS Redundancy and Failover
vii
RAID-1 Disk Drive Redundancy
Note: As part of a periodic maintenance schedule, inspect the servers (NMS and Protocol Processor Blades) for physical damage. Verify LED status during scheduled maintenance periods.
In the event that you experience a partial disk failure, contact the iDirect TAC for support. You can also contact IBM technical support directly.
viii
NMS Redundancy and Failover
2 Configuring the dbBackup Script Parameters The dbBackup script runs on the Primary NMS Server. By default, 30 minutes after midnight, this script automatically backs up the NMS configuration and archive databases and copies them to the Backup NMS Server. This script must be specifically configured at each Hub installation.
2.1
Setting Up SSH1 between Primary and Backup NMS Servers The dbBackup script uses Secure Copy (SCP) to copy the backed-up database files from the Primary NMS Server to the Backup NMS Server using SSH. By default, SCP prompts for a password. The tasks in this section ensure that SCP can copy the files without prompting for a password. This is allowed because security key-pairs are preconfigured. To configure SCP for copying the files without prompting for a password, perform the following: 1. In the /root directory on the Backup NMS Server, enter: ssh-keygen -t rsa
2. Press the Enter key at all prompts to choose the defaults. 3. In the /root directory on the Primary NMS Server, enter: ssh-keygen -t rsa
4. Press the Enter key at all prompts to choose the defaults. 5. Copy the generated file named id_rsa.pub on the primary NMS to the Backup NMS by entering: scp ~/.ssh/id_rsa.pub root@ipaddr:/root
Where: ipaddr = the IP address of the Backup NMS Server. 1. Developed by SSH Communications Security Ltd., “Secure Shell” is a program that allows you to log into another computer over a network to execute commands on a remote machine, and to move files from one machine to another. It provides strong authentication and secure communications over insecure channels. It is a replacement for rlogin, rsh, rcp, and rdist. SSH protects a network from attacks such as IP spoofing, IP source routing, and DNS spoofing.
NMS Redundancy and Failover
vii
Configuring The NMS to Send Database Backups to The Backup NMS
6. Enter the following command to connect to the Backup NMS Server. ssh root@ipaddr
Where: ipaddr = the IP address of the Backup NMS Server.
7. Enter the user ID and password when prompted. 8. Enter the following command to append the contents of the id_rsa.pub file to the end of the authorized_keys2 file: cat id_rsa.pub >> .ssh/authorized_keys2
9. Verify that this prodecure was a success by repeating Step 6. If you are not prompted for a password, the procedure was successful.
2.2
Configuring The NMS to Send Database Backups to The Backup NMS To configure the NMS to send the backup files to the Backup NMS, perform the following:
1. On the Backup NMS Server, create a directory called backup in the /root directory by entering: cd /root mkdir backup
2.
On the Primary NMS Server, backup and restore scripts and .ini files are contained in /home/nms/utils/db_maint. Using the vi editor, add the following line to the dbBackup.ini script for each target db backup machine you want to use: root@ipaddr:/root/backup
Where: addr = the IP address of each backup server receiving the backup files. The backup process creates two archive files, one for each DB: nms.##########.tgz and nrd_archive.##########.tgz, where ### is a random number.
2.3
Defining dbBackup Script Default Behavior By default, the dbBackup script saves two copies of your databases, and backs up both of your configuration and archive databases. You can change both of these defaultsby performing the following procedures. Note:
viii
Saving more than two copies uses more disk space on the Primary NMS.
NMS Redundancy and Failover
Defining dbBackup Script Default Behavior
2.3.1 Changing the Number of Backup Copies to Save To change the number of backup copies to save, perform the following: 1. Log onto the Pimary NMS Server as root. 2. Change to the directory where the dbBackup.ini file resides by entering: cd /home/nms/utils/db_maint.
3. Using vi editor, open the dbBackup.ini file. 4. Change the value in the line KeepVersions=2 to the number of backups you require.
2.3.2 Changing Database Backups To change which databases the dbBackup script backs up, perform the following: 1. Logon to the Primary NMS Server as root. 2. Enter the following command: cd /home/nms/utils/db_maint.
3. Using the vi editor, open the dbBackup.ini file. 4. If you only want to back up your configuration database, make the command line ad=nrd_archive a comment by adding the # character to the beginning of the line, shown as follows: #ad=nrd_archive
5. If you only want to backup your archive database, make the command line cd=nms a comment by adding the # character to the beginning of the line, shown as follows: #cd=nms
NMS Redundancy and Failover
ix
Defining dbBackup Script Default Behavior
x
NMS Redundancy and Failover
3 Special Provisioning for a Distributed NMS Special consideration is required for Hubs that are using a Distributed NMS. If you are using a Distributed NMS, you must make configuration changes on the Primary NMS Server, Auxiliary Servers, and Backup Server (event server (evtsvr), latency server (latsvr), and the archive server (nrdsvr)).
3.1
Configuring the Primary NMS Server If you are not running a Distributed NMS, skip this section and go to “:wq!” on page ix. To configure the Primary NMS Server, perform the following: 1. Logon to the Primary NMS Server as root. 2. Using the vi editor, open the dbBackup.ini file. 3. Make the following command a comment by adding # at the beginning of the line, shown as follows: #ad=nrd_archive
4. Add the following lines to the dbBackup.ini file: cd=nms KeepVersions=5 root@backup_nms_ipaddr:/root/backup
Where: backup_nms_ipaddr = the IP address of the Backup NMS Server.
5. Open the crontab file for editing by entering: crontab –e.
6. Enter the following lines to the crontab file: 30 0 * * * /home/nms/utils/db_maint/dbBackup >> /home/nms/utils/dbbackup.output 2>&1 0 0 * * * //home/nms/utils/db_maint/cons.pl >> /home/nms/utils/db_maint/cons.output 2>&1
NMS Redundancy and Failover
vii
Configuring the Auxiliary NMS Servers
7. Press Esc to exit Edit Mode. 8. Save and close the crontab file by entering the following command: :wq!
3.2
Configuring the Auxiliary NMS Servers To configure the Auxiliary NMS Servers, perform the following: 1. Logon to the Auxiliary NMS Server as root. 2. Using the vi editor, open the dbBackup.ini file. 3. Make the following command line a comment by adding the #character at the beginning of the line, as shown: #cd=nms
4. Add the following lines to the file: ad=nrd_archive KeepVersions=5
5. Optionally, copy the database to the Backup NMS Server by adding the following command line to the dbBackup.ini file: root@backup_nms_ipaddr:/root/backup-server_name
Where: backup_nms_ipaddr = the IP address of the Backup NMS Server server_name = the name of the Auxiliary Server. Note:
If the line already exists in the file and you do not want to backup the database, comment out this line by inserting a # at the beginning of the line, as shown: #root@backup_nms_ipaddr:/root/backup-server_name
6. Open the crontab file for editing by entering the following command: crontab –e
7. Enter the following lines: 0 */6 * * * /home/nms/utils/db_maint/cons.pl >> /home/nms/utils/db_maint/cons.output 2>&1
8. Press Esc to exit Edit Mode. 9. Save and close the crontab file by entering the following command: :wq!
viii
NMS Redundancy and Failover
Configuring the Backup NMS Server
3.3
Configuring the Backup NMS Server To configure the Backup NMS Server, perform the following: 1. Logon to the Backup NMS Server as root. 2. Using the vi editor, open the dbRestore.ini file. 3. Make the following command line a comment by adding # at the beginning of the line, as shown: #ad=nrd_archive
4. Add the following lines to the file: cd=nms KeepVersions=5 Location=/root/backup
5. Create the backup directories by entering the following commands: mkdir root/backup mkdir root/backup -server_name Where: server_name = the name of the Backup NMS Server.
6. Open the crontab file for editing by entering the following command: crontab –e
7. Enter the following lines: 0 4 * * * /home/nms/utils/db_maint/dbRestore >> /home/nms/utils/db_maint/dbrestore.output 2>&1
8. Press Esc to exit Edit Mode. 9. Save and close the crontab file by entering the following command: :wq!
NMS Redundancy and Failover
ix
Configuring the Backup NMS Server
x
NMS Redundancy and Failover
4 Configuring dbRestore Script Parameters This section provides the procedure to configure the dbRestore script parameters.
4.1
Configuring the Backup NMS to Restore the Databases To configure the Backup NMS to restore the databases, perform the following: 1. On the Backup NMS Server, change to the /home/nms/utils/db_maint/ directory. 2. Using the vi editor, open the dbrestore.ini script file. 3. Make the following command line a comment a command by deleting the # character and adding the backup location by changing the Location line to read: Location=/root/BACKUP
Note:
The cron job1 performs the restore on the backup mahine nightly.
To modify the cron job, perform the following: 1. Open the crontab file for editing by entering the following command: crontab -e
2. Open the vi editor and press the I key to turn on Insert Mode. 3. Add the following line: 30 1 * * * /home/nms/utils/db_maint/dbRestore>>/home/nms/utils/db_maint/dbrestore.output 2>&1
4. Press Esc to exit Insert Mode. 5. Save and close the crontab file by entering the following command: :wq! 1. cron is a Linux system process (daemon) that will execute a program at a preset time. To use cron you must prepare a text file that describes the program that you want executed and the times that cron should execute them. Then you use the crontab program to load the text file that describes the cron jobs into cron
NMS Redundancy and Failover
vii
Configuring the Backup NMS to Restore the Databases
viii
NMS Redundancy and Failover
5 Changing Over to the Backup NMS Server If you need to perform maintenance on the Primary NMS, you can do a change over to the Backup NMS Server during a scheduled maintenance window or during a catastrophic failure situation. In catastrophic failure situation, the Primary NMS Server may become inaccessible. There are three procedures required to move a file from the Primary NMS Server to the Backup NMS Server: Compress, Copy, and Extract. When you are changing from the Primary NMS Server to the Backup NMS Server, there are major processes that need to be followed, as outlined in this guide: • Section 2.1, Procedures for Moving Component Files to the Backup NMS Server • Section 2.2, Procedures for Moving pp_controller Files To the Backup NMS Server • Section 2.3, Bringing The Backup NMS Online
5.1
Procedures for Moving Component Files to the Backup NMS Server The required procedures consist of: • Logging Into The Primary NMS Server • Compressing the Component Options File Directories • Copying the Component Options File Directories from the Primary NMS Server to Backup NMS Server • Extracting Component File Directories Onto Backup NMS Server
Logging Into The Primary NMS Server Before you can begin to move from your Primary NMS Server to the Backup NMS Server, logon to the Primary NMS as root. Now you are ready to begin moving files. Perform the procedures in the order shown for the best results.
Compressing the Component Options File Directories To compress the component options file directories that reside on the Primary NMS Server, perform the following:
NMS Redundancy and Failover
vii
Procedures for Moving Component Files to the Backup NMS Server
1. Record any static routes that are configured on the Primary NMS Server. 2. On the primary NMS, run the backup script to ensure that you have an up-to-date database on the Backup NMS Server. If the database is not up-to-date, copy any necessary files over as required. 3. In addition to the database files, additional component files must be copied, shown in their appropriate directories, as follows: /home/nms/cfg/options_files/modems /home/nms/cfg/options_files/rmtdefs /home/nms/cfg/options_files/chassis /home/nms/cfg/options_files/pp_globals /home/nms/cfg/options_files/netdefs
4. To compress each of these directories, enter the following command: 4a. Change the the appropriate directory by entering the following command: cd /home/nms/cfg/options_files/ 4b. Enter the following command to compress each directory: tar –cvf filename.tar source file name or directory
Where: filename.tar = is the name of the compressed file that contains the component name. (for example: tar -cvf modems.tar modems). source file name or directory = is the original name of the file, as displayed in Step 3. (For example, the first line shows the directory and file name of /home/nms/cfg/options_files/modems. The source file name is modems). 5. Repeat Step 4 for each component directory, using the commands as follows: 5a. Enter tar –cvf modems.tar modem 5b. Enter tar –cvf rmtdefs.tar rmtdefs 5c. Enter tar –cvf chassis.tar chassis 5d. Enter tar –cvf pp_globals.tar pp_globals 5e. Enter tar –cvf netdefs.tar netdefs You are now ready to copy the files you compressed from the Primary NMS Serer to the Backup NMS Server.
Copying the Component Options File Directories from the Primary NMS Server to Backup NMS Server 1. Copy the tar file to the Backup NMS Server using the SCP command. For example,
viii
NMS Redundancy and Failover
Procedures for Moving Component Files to the Backup NMS Server
scp ~/ filename.tar root@ipaddr:/home/nms/cfg/options_files/
Where: filename.tar = the name of the compressed file that you created in Step 4. (for example: modems.tar) ipaddr = the IP address of the Backup NMS Server. 2. Repeat Step 1 for each component options file, using the commands as follows (you can refer to Step 1 to understand the elements of this command): 2a. Enter: scp –/ modems.tar
[email protected]:/home/nms/cfg/options_files/ 2b. Enter: scp –/ rmtdefs.tar
[email protected]:/home/nms/cfg/options_files/ 2c. Enter: scp –/ chassis.tar
[email protected]:/home/nms/cfg/options_files/ 2d. Enter: scp –/ pp_globals.tar
[email protected]:/home/nms/cfg/options_files/ 2e. Enter: scp –/ netdefs.tar
[email protected]:/home/nms/cfg/options_files/ Now that you have compressed and copied all of the required files, you are ready to extract these files onto the Backup NMS Server.
Extracting Component File Directories Onto Backup NMS Server To extract the component files onto the Backup NMS Server, perform the following: 1. Login to the Backup NMS Server as root. 2. Optionally, if you want to display the contents of this directory so that you can list all of the component files you need to extract, enter the following command to change to the appropriate directory: cd /home/nms/cfg/options_files/ 3. Enter the following command to list the contents: ls An example of the output that displays is as follows: chassis
NMS Redundancy and Failover
modems
netdefs
pp_globals
rmtdefs
ix
Procedures for Moving pp_controller Files To the Backup NMS Server
4. Enter the following command: tar –xvf filename.tar
Where: filename.tar = the name of the compressed file that you created in Step 4. (for example: modems.tar) 5. Repeat Step 4 for each component file, using the commands as follows: 5a. Enter tar –xvf modems.tar 5b. Enter tar –xvf rmtdefs.tar 5c. Enter tar –xvf chassis.tar 5d. Enter tar –xvf pp_globals.tar 5e. Enter tar –xvf netdefs.tar Now that you have moved the component files to the Backup NMS Server successfully, you are ready to move the pp_controller files.
5.2
Procedures for Moving pp_controller Files To the Backup NMS Server The required procedures for moving the pp_controller files to the Backup NMS Server consist of: • Logging Into The Primary NMS Server • Compressing the pp_controller Files • Copying the pp_controller Files from the Primary NMS Server to Backup NMS Server\ • Extracting the pp_controller Files Onto Backup NMS Server
Logging Into The Primary NMS Server Before you can begin to move from your Primary NMS Server to the Backup NMS Server, logon to the Primary NMS root. Now you are ready to begin moving files. Perform the procedures in the order shown for the best results.
Compressing the pp_controller Files To compress the pp_controller files that reside on the Primary NMS Server, perform the following: 1. Record any static routes that are configured on the Primary NMS Server. 2. On the primary NMS, run the backup script to ensure that you have an up-to-date database on the Backup NMS Server. If the database is not up-to-date, copy any necessary files over as required.
x
NMS Redundancy and Failover
Procedures for Moving pp_controller Files To the Backup NMS Server
3. Change to the appropriate directory by entering the following command: cd /etc/idirect/
4. Display the contents of this directory by entering the following command: ls -la
An example of the output that displays is as follows: total 28 drwxr-xr-x
6 root
root
4096 Sep
4 23:18 .
drwxr-xr-x
82 root
root
8192 Sep
5 14:17 ..
drwx------
3 root
root
4096 Jun 19 11:45 ca
drwxr-xr-x
2 root
root
4096 Sep
4 23:18 map
drwxr-xr-x
3 root
root
4096 Sep
5 20:35 pp_controller_15001
drwxr-xr-x
3 root
root
4096 May 29 21:56 pp_controller_47767
5. Compress all of the pp_controller files (as displayed in previous step) in the 15000 range by entering the following command for each pp_controller that is listed above tar -cvf pp_controller_1500n.tar pp_controller_1500n Where: n = the number of the pp_controller for which you are extracting files, and is listed when you list the contents of the /etc/idirect directory. Note:
Referring to the above output display as an example, the command for compressing a pp_controller file listed is: pp_controller_15001.tar
6. Repeat Step 5 for each pp_controller on your network. You have successfully compressed all of the pp_controller files and are now ready to copy the files you compressed from the Primary NMS Serer to the Backup NMS Server.
Copying the pp_controller Files from the Primary NMS Server to Backup NMS Server 1. Copy the compressed pp_controller tar file to the Backup NMS Server by entering the following command: scp ~/ filename.tar root@ipaddr:/etc/idirect
Where: filename.tar = the name of the compressed file that you created in the previous procedure, Step . (for example: pp_controller_15002.tar) ipaddr = the IP address of the Backup NMS Server.
NMS Redundancy and Failover
xi
Procedures for Moving pp_controller Files To the Backup NMS Server
Now that you have compressed and copied all of the required files, you are ready to extract these files onto the Backup NMS Server. filename.tar = the name of the compressed file that you created in Step 4. (for example: pp_controller_15002.tar)
Extracting the pp_controller Files Onto Backup NMS Server The following procedure assumes you are still logged into the Backup NMS Server from performing the previous procedure. If not, do so now. Extract the pp_controller files onto the Backup NMS Server by performing the following: 1. Change to the appropriate directory into which the pp_controller files being extracted by entering the following command: cd /etc/idirect/ 2. Display the contents of this directory by entering the following command: ls -la
An example of the output that displays is as follows: total 104 drwxr-xr-x
6 root
root
4096 Sep 12 15:31 .
drwxr-xr-x
82 root
root
8192 Sep
drwx------
3 root
root
4096 Jun 19 11:45 ca
drwxr-xr-x
2 root
root
4096 Sep
5 14:17 ..
4 23:18 map
-rwxr--r-1 root pp_controller_15001.tar
root
71680 Sep 12 15:27
drwxr-xr-x
3 root
root
4096 May 29 21:56 pp_controller_47767
drwxr-xr-x
3 root
root
4096 Sep
5 20:35 pp_controller_test
3. Extract the pp_controller files by entering the following command: tar -xvf filename_1500n.tar Where: filename= pp_controller n= the number of the pp_controller for which you are extracting files, and is listed when you execute the ps -ef | grep pp_ as shown in Step 1 results. (for example: pp_controller_15002.tar) An example of the output that displays is as follows: pp_controller_15001/ pp_controller_15001/networks/ pp_controller_15001/networks/1.net/
xii
NMS Redundancy and Failover
Bringing The Backup NMS Online
pp_controller_15001/networks/1.net/netdef.opt pp_controller_15001/networks/1.net/remote_5035562.opt pp_controller_15001/networks/1.net/remote_5034935.opt pp_controller_15001/networks/2.net/ pp_controller_15001/networks/2.net/netdef.opt pp_controller_15001/networks/2.net/remote_6078612.opt pp_controller_15001/networks/2.net/remote_12857401.opt pp_controller_15001/networks/6.net/ pp_controller_15001/networks/6.net/netdef.opt pp_controller_15001/networks/6.net/remote_10491435.opt pp_controller_15001/networks/10.net/ pp_controller_15001/networks/10.net/netdef.opt pp_controller_15001/networks/10.net/remote_6345822.opt pp_controller_15001/controller.conf pp_controller_15001/global.opt pp_controller_15001/x509_local_key.txt pp_controller_15001/x509_local_cert.txt
4. Repeat Step 3 for each port number. You have successfully moved all of the required files to your Backup NMS Server.
5.3
Bringing The Backup NMS Online Now that you have moved all of the files to the Backup NMS Server, you need to bring this server into your network as your Primary NMS Server. This requires several tasks, as indicated in the following procedure. Perform the following steps: 1. If the primary NMS is still accessible, reconfigure the primary NMS eth0 interface with another unused IP address or preferably, physically unplug the eth0 interface. If the interface has been reconfigured, the NMS server must be rebooted. You can also restart the network services by entering the follow command: service network restart
2. If the primary NMS remains connected to the network, the iDirect NMS services MUST be shutdown with the command below and the IP address must be different. Additionally, these services MUST remain down while the other new (backup) NMS server is online. service idirect_nms stop
3. Verify that the routing configuration on the Backup NMS Server is correct. If RIP v2 was running on the Primary NMS Server, it has to be running on the backup. Any static routes that were configured on the Primary NMS Server need to be configured on the Backup NMS Server. Make sure they are configured as persistent routes so they remain in effect after a reboot.
NMS Redundancy and Failover
xiii
Bringing The Backup NMS Online
4. Configure Backup NMS Server eth0 interface with the primary NMS original eth0 interface address. The reason for this is to match the original primary NMS IP address that is configured in the configuration (options) files of the iDirect network elements. 5. Reboot the Backup NMS Server machine or restart the network services entering the following command: service network restart
6. Verify the new Primary NMS Server routing table by entering the following command: netstat –r -n
7. If network has secure mobile and or encrypted remotes, make sure /home/nms/cfg/nms_cfg.opt file has the correct parameters (applies to iDS version 6.0.x). 8. Start the NMS services by entering the following command: service idirect_nms start
9. The network should now be running on the Backup NMS Server. The status on the Line Cards and the Remotes should be normal in iMonitor. 10. If needed, apply the Protocol Processor level and network level configuration to all networks via iBuilder.
Changing The Backup NMS Server To The Primary NMS Server If the Backup NMS Server will remain as the new Primary NMS Server, the crontab file and the config table must be modified in order for the new NMS to act as the Primary NMS Server. The modification of the crontab is to allow the new Primary NMS Server to execute the daily backup and consolidation scripts. To change the Backup NMS Server to be your Primary NMS Server, perform the following: 1. Enter the command to open the crontab file for editing: crontab –e
2. Make the command lines in the crontab file comments by adding a # in front of the each command line. 3. Adding the following lines: 30 0 * * * /home/nms/utils/db_maint/dbBackup >> /home/nms/utils/db_maint/dbbackup.output 2>&1 30 1 * * * /home/nms/utils/db_maint/cons.pl >> /home/nms/utils/db_maint/cons.output 2>&1 4. Press Esc to exit Edit Mode. 5. Save and close the crontab file by entering the following command:
xiv
NMS Redundancy and Failover
Bringing The Backup NMS Online
wq!
6. To configure the new (Backup) NMS Server to start the NMS services every time at bootup, enter the following command: chkconfig --level 2345 idirect_nms on
The backup is a success if there is no data output after entering this command.
NMS Redundancy and Failover
xv
