Netbackup Troubleshooting

S.NO

Status Code

Status

1

58

The server was unable to connect to the client.

2

59

access to the client was not allowed(59)

3

191

duplicate exited with status 191

4

50

import failure-phase 2

5

96

unable to allocate new media for backup Check for available medias in goodies-available

Frozen state reason--Backup exec tapes was loaded,NBU reads it have a BE data & mark it as FROZE 6

1

Partially success

7

74

The bpstart_notify script on the client takes too long.

8

2

none of the requested files were backed up(2)

9

156

snapshot error encountered

10

13

network issue

11

41

network issue

12

25

network issue

13

24

network issue

14

14

file write failed

status 2009 All compatible drive paths are down but media is available 129

(Disk storage unit is full)

status 29/141

catalog job failed

Status 14

file write failed)

Issue can't connect to client(58)--Network issue

or the media server tries to access the client, but the client does not recognize the server as a valid server. duplicate images was not happened due to lack of tapes in the librray http://www.symantec.com/business/support/index?page=content&id=TECH28499 There is no tapes for backup

as loaded,NBU reads it have a BE data & mark it as FROZEN state

ob detail & proceed as per the error.always check for the removable storage services should be disabled. client timed out waiting for bpstart_notify to complete catalog job failed due to lack of tape

Found some Events 1049 and 1284 on server AURHOM0002 event viewer

due to network issue

All compatible drive paths are down but media is available

We have modified the registry entry “HKEY_LOCAL_MACHINE\SOFTWARE\Veritas\NetBackup\CurrentVersion\Config\AUTHE We have removed all the Trusted Domain entry leaving only 2 , One with Master Server Name & Nestle kept as it is. UKYORM0015 "ADDED AUTOMATICALLY" WINDOWS ukyorm0015.nestle.com 0 NESTLE "ADDED AUTOMATICALLY" WINDOWS ukyorm0015.nestle.com 0 I\O error for files & folders

Troubleshooting -solution Check the Network connection & check the server is reachable or not Need to authorise the client server & then check Overwritable tapes & should monitor the duplication jobs Don’t logoff your session until the import job completes send market to load the tapes NA NA NA Change the value in bp start notify attributes Overwritable tapes & should monitor the catalog jobs

network issue

HINE\SOFTWARE\Veritas\NetBackup\CurrentVersion\Config\AUTHENTICATION_DOMAIN”

nly 2 , One with Master Server Name & Nestle kept as it is.

orm0015.nestle.com 0

15.nestle.com 0 go to host properties/matsre server/client attrubutes/use media server for duplication.

IF the status code is not available then write a short description of the problem such as Robot Down, Service Down, Server D Please follow the below ticket title update for NBU tickets

EUR SCOM HQVEVM0024 CHLSNM0004 CHLSNA0101 STATUS 14

bot Down, Service Down, Server Down, Dedup Down, GWAN Down etc. etc. If you have any doubt please check with your shift lead.

e check with your shift lead.

Please find simple useful commands for SLP monitoring on NBU, Refer attached doc as it has more information. Action To check the ctime from backup id To inactivate all further duplication jobs To activate all duplication jobs To inactive further duplications for specific SLP To activate all further duplication jobs to view the status of all duplication jobs to view the list of incomplete duplication jobs to inactivate the STU for duplications only to activate the STU for duplications To cancel a specific duplication job cd "Program Files\VERITAS\NetBackup\bin" cd "Program Files\VERITAS\NetBackup\bin\goodies" To check the available medias in the tape library C:\Program Files\VERITAS\NetBackup\bin\goodies>available_media.cmd

to view the list of incomplete duplication jobs C:\Program Files\VERITAS\NetBackup\bin\admincmd>nbstlutil stlilist image_incomplete -U

To check the client connectivity between the master & the client C:\Program Files\VERITAS\NetBackup\bin\admincmd>bptestbpcd -client ptlism0002.nestle.com

To know abot the ctime -use below cmd C:\Program Files\VERITAS\NetBackup\bin>bpdbm -ctime 1338150733 1338150733 = Sun May 27 22:32:13 2012

C:\Program Files\VERITAS\Volmgr\bin>vmoprcmd.exe -d

C:\Program Files\VERITAS\Volmgr\bin>vmdelete.exe -m (Media ID)

C:\Program Files\VERITAS\NetBackup\bin\admincmd>nbstlutil stlilist -image_incomplete C:\Program Files\VERITAS\NetBackup\bin\admincmd>hostname FRBVRM0001 C:\Program Files\VERITAS\NetBackup\bin\admincmd>

Reauthenticate on NBU C:\Program Files\VERITAS\NetBackup\bin>bpnbat.exe Stop the duplication C:\Program Files\VERITAS\NetBackup\bin\admincmd>nbstlutil inactive -lifecycle DKKOLM0000_DKCPHM0004_LP01_AIR c:\Program Files\VERITAS\NetBackup\bin\admincmd>nbstlutil inactive -lifecycle all To list all SLP jobs, nbstlutil stlilist dkkolw0001.nestle.com_1337623200 nbstlutil -wait inactive dkkolw0001.nestle.com_1337623200 nbstlutil inactive -lifecycle $DKKOLM0000_DKCPHM0004_LP01_AIR

nbstlutil -wait inactive -lifecycle SLP_DKKOLM0000_DKCPHM0004_LP01_AIR -backupid dkkolw0001.nestle.com_1337623200 nbstlutil -wait inactive -backupid dkkolw0001.nestle.com_1337364000 nbstlutil stlilist -U nbstlutil stlilist nbstlutil stlilist -image_incomplete -U to find ctime : C:\Program Files\VERITAS\NetBackup\bin>bpdbm -ctime 1337374479

FOR ROB test C:\Program Files\VERITAS\Volmgr\bin>vmoprcmd C:\Program Files\VERITAS\Volmgr\bin>robtest

Robot Selection --------------1) TLD 0 2) none/quit Enter choice: 1 1 Robot selected: TLD(0) robotic path = {3,0,0,1} Invoking robotic test utility: C:\Program Files\VERITAS\Volmgr\bin\tldtest.exe -rn 0 -r {3,0,0,1} Opening {3,0,0,1} MODE_SENSE complete Enter tld commands (? returns help information) sd drive 1 (addr 1) access = 1 Contains Cartridge = no drive 2 (addr 2) access = 1 Contains Cartridge = no drive 3 (addr 3) access = 1 Contains Cartridge = no drive 4 (addr 4) access = 1 Contains Cartridge = yes Source address = 1005 (slot 5) Barcode = BDC916L5

READ_ELEMENT_STATUS complete ss slot 1 (addr 1001) contains Cartridge = no

slot 2 (addr 1002) contains Cartridge = no slot 3 (addr 1003) contains Cartridge = no slot 4 (addr 1004) contains Cartridge = no slot 5 (addr 1005) contains Cartridge = no slot 6 (addr 1006) contains Cartridge = yes Source address = 1006 Barcode = BDC917L5 slot 7 (addr 1007) contains Cartridge = yes Source address = 1007 Barcode = BDC918L5 slot 8 (addr 1008) contains Cartridge = yes Source address = 1008 Barcode = BDC919L5 slot 9 (addr 1009) contains Cartridge = yes Source address = 1009

Barcode = BDC912L5 slot 10 (addr 1010) contains Cartridge = yes Source address = 1010 Barcode = BDC913L5 slot 11 (addr 1011) contains Cartridge = yes Source address = 1011 Barcode = BDC914L5 > q READ_ELEMENT_STATUS complete q Robot Selection --------------1) TLD 0 2) none/quit Enter choice: 2 2 C:\Program Files\VERITAS\Volmgr\bin> The command to move a tape from drive 1 to slot 15 is : m d1 s15

To check the logs: nbrbutil -dump To release the jobs: nbrbutil -releaseMDS 10911

10.196.208.195 administrator 20121964 Stuck media in the Tape drive Library is offline please to apply troubleshooting and provide feedback.

Remove the stuck media ·

To power library off

·

To check cable are properly connected from server to library

·

To turn on the Library.

·

To load scratch tapes

To check the status of the SLP (Storage Lifecycle Policy) if it is in Active or Inactive mode. Please run command under: \Installation_path ...\Netbackup\bin\admincmd>Nbstl -L Something like this mentioned below will be the output of the command. It’ll show you complete details of the SLP.

In few servers, “Nbstl –L” is not working showing error that no entity found. In such case please run the command with the storage Lifecycle policy name, \Installation_path ...\Netbackup\bin\admincmd>Nbstl EGCAIM001_LP01 –L

tached doc as it has more information. command bpdbm -ctime nbstlutil inactive -lifecycle all nbstlutil active -lifecycle all nbstlutil -wait inactive -lifecycle DKKOLM0000_DKCPHM0004_LP01_AIR nbstlutil active -lifecycle DKKOLM0000_DKCPHM0004_LP01_AIR nbstlutil stlilist -U nbstlutil stlilist -image_incomplete nbstlutil inactive -lifecycle nbstlutil active -destination nbstlutil cancel -backupid

to check the communication-bw cl & Mas

to check the duplication jobs

ptlism0002.nestle.com

To activate particular backup id C:\Program Files\VERITAS\NetBackup\bin\admincmd>nbstlutil.exe active -backupid n lnunw0001.nestle.com_1346436015 -force

For deleting the media from EMM

age_incomplete

Rob Test C:\Program Files\VERITAS\Volmgr\bin> vmoprcmd robtest

lifecycle DKKOLM0000_DKCPHM0004_LP01_AIR

AIR -backupid dkkolw0001.nestle.com_1337623200

1337374479

To check the incomplete Job list

Check for orphaned device allocation. Run this command from cmd: nbrbutil -dump Check the 'MDS Allocation' output at the bottom. Orphaned allocations can be released as follows: nbrbutil -releaseMDS or, if no backups are running, reset all: nbrbutil -resetAll (command is in \veritas\netbackup\bin\admincmd)

Inactive mode.

Path to be executed in C:\Program Files\VERITAS\NetBackup\bin> c:\Program Files\VERITAS\NetBackup\bin\admincmd> c:\Program Files\VERITAS\NetBackup\bin\admincmd> c:\Program Files\VERITAS\NetBackup\bin\admincmd> c:\Program Files\VERITAS\NetBackup\bin\admincmd> c:\Program Files\VERITAS\NetBackup\bin\admincmd> c:\Program Files\VERITAS\NetBackup\bin\admincmd> c:\Program Files\VERITAS\NetBackup\bin\admincmd> c:\Program Files\VERITAS\NetBackup\bin\admincmd> c:\Program Files\VERITAS\NetBackup\bin\admincmd>

telnet czprgd0002 bpcd bpclntcmd -hn czprgd0002

Checked & found the last night daily backup job nlnunl0010 was completed successfull.Hence we close this IM as complete. APPLICATION, INFRASTRUCTURE AND SYSTEM MANAGEMENT/BACKUP - WINDOWS/NETBACKUP nbstlutil stlilist -image_incomplete -U

bpimmedia.exe -mediaid IRD705 -l

n\admincmd>nbstlutil.exe active -backupid n

To check the status of the Drive To move the voume from one slot to another

Hence we close this IM as complete.

Approximate calculationof overwritable tapes--Calculate the data size & ask market for tapes capacity LTO4 1TB LTO5 2TB

Backup completed

ZONE EUR EUR EUR EUR EUR EUR EUR EUR EUR EUR EUR

MasterServer ITMILM0007 ITMILM0007 ITMILM0008 ITPORM0001 MTVALM0001 NLAMNW0000 PLWARM0005 PLWAWM0001 PLWAWM0001 PLWAWM0001 PLWAWM0001

Server ITBERW0000 ITUDIW0000 ITCBUW0001 ITPORL0002 MTVALH0002 NLAMNM0000 PLWARD0018 PLWAWW0004 PLWAWB0000 PLWAWA0020 PLWAWH0001

-F-F--F F-FFF-FFF -F-FFFF F--

Backup completed UKCROM0015 ESBCNM0000

C:\WINNT\system32\NtmsData C:\Temp\Eicar

Status Siddesh is working on the issue Siddesh is working on the issue Siddesh is working on the issue Backup completed Working on the ticket IM0006540293 Backup completed Backup completed IM0007341781- PLWAWM0001-Tape-Drive offline Backup completed Backup completed Backup completed

160.213.34.90

8 zeros

193.148.192.64

8 Zeros

\system32\NtmsData

Netbackup Tips Glossary Term CLI GUI Media Server Master Server

Starting and Stopping Netbackup Stopping Netbackup /usr/openv/netbackup/bin/K77netbackup --> graceful shutdown /usr/openv/netbackup/bin/bpps -a --> check for any remaining processes /usr/openv/netbackup/bin/goodies/bp.kill_all ---> kills all remaining netbackup processes, not necessarily graceful /usr/openv/netbackup/bin/bpps -a --> check for any remaining processes kill -9 for any remaining. NOTE: unkillable processes may require a reboot Starting Netbackup /usr/openv/netbackup/bin/S77netbackup --> after bp.kill_all, to restart

Common Tasks Starting the Administration GUI java from the windows client x-windows from the server - /usr/openv/netbackup/bin/xnb & Checking Backup Status Activity Monitor or /usr/openv/netbackup/bin/admincmd/bpdbjobs -report Cleaning a tape manually Identify the drive name to be cleaned tpclean -L Manually clean the drive: tpclean -C Determining what tapes were used for a backup

GUI Backup and Restore --> Find the file system --> Preview Media Button CLI Find the correct backup images bpimagelist -U -client -d -e Find the media used for those images bpimagelist -U -client -d -e -media Listing the files in a backup Find the tape(s) used (above procedure using bpimagelist) cd /usr/openv/netbackup/db/jobs/done Run the following script and redirect it's output to a text file: for file in `grep MOUNTING *|grep |awk '{print $1}'|sed 's/:MOUNTING//'` do echo $file grep PATH_WRITTEN $file|awk '{print $3}' echo " " echo "==========================================End of Image======================================" echo " " done

This process works for NBU V3.4: cd /usr/openv/netbackup/db/images/ ls -ltr --> this will identify the directory with the proper date verify directory with "bpdbm -ctime cd ls -ltr --> lists all of the backups for this client on this date cat __.f | awk '{print $10}' --> this prints out the files in the backup

For NBU > V3.4 bpflist --help --> undocumented netbackup command to list files from a binary .f file

Inventory the Robot

Inventory Robot --> /opt/openv/volmgr/bin/vmcheckxxx -rt robot_type -rn robot_number -list (where robot_type is tld, a Inventory Robot and Update Configuration --> /opt/openv/volmgr/bin/vmupdate -rt robot_type -rn robot_number -list (w Listing Properties of the Volume Pools vmpool -listall

Scratch Tapes Count scratch tapes: /usr/openv/volmgr/bin/vmquery -pn Scratch | grep -c "robot slot" Moving tapes to the scratch pool If Needed - Expire the tape bpexpdate -ev -d 0 -force -host Move the tape vmchange -p 2 -m Checking Drive Usage /usr/openv/volmgr/bin/vmoprcmd Taking a drive down or up /usr/openv/volmgr/vmoprcmd -down /usr/openv/volmgr/vmoprcmd -up

Performing a Restore From the GUI user backup & restore --> configuration --> client user backup & restore --> configuration --> client to restore directory to search directory depth date range file --> browse backups for restore Adding New Tapes to the Library Using the GUI Media Management --> Actions --> New --> Single Volume . . -->

Media Type (ie DLT) Robot Type (ie TLD) Media ID (from Inventory) Slot Number (from Inventory) Robot Number (ie 0) Volume Group Volume Pool (ie Scratch) Using the CLI vmadd -m -mt -verbose -rt -b -rn -rc1 -p lists all pools, both name and number For example: vmadd -m 000151 -mt dlt -verbose -rt tld -b 000151 -rn 0 -rc1 8 -p 2 -mm 0

Re-using Tapes from other systems or older Netbackups Expire the media bpexpdate -ev MEDIA_ID -d 0 -force -host HOST Deassign the media vmquery -deassignbyid MEDIA_ID 4 0 Move to the scratch pool vmchange -m MEDIA_ID -p POOL# Relabel the media bplabel -ev CIM572 -d dlt -p Scratch Changing the attributes of media Changing the barcode vmchange -barcode CYM100D -m CYM100 Changing the Volume Pool vmchange -m MEDIA_ID -p POOL#

To expire media bpexpdate -ev -d 0 -force -host To unfreeze media List the frozen media /usr/openv/netbackup/bin/goodies/available_media | grep -i FROZEN Unfreeze the media bpmedia -unfreeze -ev -h To relabel a tape bplabel -ev -d -p bplabel -ev 000687 -d dlt -p TriVrgt_OFFSITE To remove media from the Netbackup database Verify that there are no images on the tape bpimmedia -mediaid 000687 -L Expire the tape bpexpdate -ev 000687 -d 0 -host scorpius -force Get the status and pool number of the tape vmquery -m 000687 Deassign the tape vmquery -deassignbyid

vmquery -deassignbyid 000687 4 0x0 Delete the tape vmdelete -m 000687

Installing the Netbackup Client /update_clients -ForceInstall -ClientList /tmp/clients.lst requires that TMPDIR and TEMPDIR be set correctly Excludng files from backup on a client Create /usr/openv/netbackup/exclude_list Put the file specifications of the files/directories to be excluded /mnt/directory/* Displaying Information about a Tape vmquery -m --> Displays attributes about a particular tape bpmedialist -U -mcontents -ev 000687 --> Displays media contents bpmedialist -U -mlist --> List of all media bpmedialist -U -mlist -ev CYM966 --> Listing of a particular media id bpimmedia -mediaid 000687 -L --> Listing of images on a tape

Robtest Commands Starting robtest robtest 1 --> to select TLD 0 Getting help ? Looking at contents of the tape drives sd Looking at the contents of the library ss Moving a tape from a drive to a library slot s d --> to identify drive number that has tape (Contains Cartridge = yes, Barcode=XXXXXX) s s --> to identify an empty slot in the tape library (Netbackup will need to be re-inventoried) m d# s# --> from from drive # to slot # s d --> verify the tape drive is empty s s --> verify the library slot has the tape

Configuration Files /usr/openv/netbackup/bp.conf configuration file, sets backup server and backup clients force statement must be correct client to browse from client to restore to /usr/openv/volmgr/vmconf

Logfiles To utilize logfiles, create the corresponding directory in /usr/openv/netbackup/logs Server Logfile directories: admin - adminstrative commands bpbrm - backup and restore manager bpcd - client daemon bpdbjobs - database manager program process bpdm - disk manager process bpjava-msvc - Java application server authentication service bpjava-usvc - process that services Java requests bprd - request daemon process bpsched - scheduler process that runs on master servers bptm - tape/optical media management process user-ops - required directory for use by Java programs xbpadm - X based administration utility xbpmon - X based job monitor process Client Logfile directories: bp - client user interface process bparchive - archive program bpbackup - backup program bpbkar - program that generates golden images bpcd - client daemon bpjava-msvc - Java application server authentication service bpjava-usvc - process that services Java requests bplist - program that lists backed up and archived files bpmount - program that determines local mountpoints and wildcard expansion for multiple streams bphdb - Oracle database backup program start process

db_log - database specific extension log tar - tar process log during restores user_ops Media Manager logging automatically goes to the system log using syslogd logging facility

.Logging will only occur if these directories are created. These directories will generate a lot of data and should be deleted w To increase the amount of logging information set VERBOSE=2 in /usr/open/netbackup/bp.conf (default is VERBOSE=1)

Processes ltid acsd vmd

Useful Commands bpcllist - list classes bpclinfo -L --> displays info about a class vmpool - volume pools vmpool -listall vmpool -listscratch bplabel -ev -d hcart bpbackup db --> backs up the catalog bpclclients --> lists the clients for a particular policy (class)

Troubleshooting bperror -statuscode this will bring up drive 0 if it's control shows as down Look for pending requests /usr/openv/volmgr/bin/vmoprcmd or gui --> device management If there is a pending request either re-assign it to a drive, or deny the request Downed drive does not come back up or does not stay up

Check for a hardware problem by looking for messages on the tape library Make sure there is not a tape stuck in the drive Use robtest (described above) to look at the drives If there is a tape stuck in the drive, try to remove it using robtest If robtest fails, then you must manually remove it. Verify the Client is communicating properly: bpclncmd -ip --> from both client and server bpclntcmd -hn --> from both client and server bpclntcmd -pn --> from client only

Device Actions Device Management --> info about tape drives dlt hcart (ultrium)

Media Actions Media id must agree with # of the tape Create a media id actions -->new-->single volume-->dlt cart (not dlt2) put it into the "netbackup" volume pool

Netbackup Client To check things out do this: It could be a couple things. Mostly DNS, bp.conf, or something stupid. On the client run this command /usr/openv/netbackup/bin/bpclntcmd -pn /usr/openv/netbackup/bin/bpclntcmd -server "server name" /usr/openv/netbackup/bin/bpclntcmd ip "ip_address"

One of these usually fails and your able to fix it right off

1074 ./bpclntcmd -hn corpbu1 1075 ./bpclntcmd -ip 10.194.1.129 1076 ping 10.194.1.129 1077 ./bpclntcmd -hn corpldv1 1078 ./bpclntcmd -hn corpbu1.corporate.vox.net 1079 ping corpldv1 1080 ./bpclntcmd -ip 10.194.1.120

Must be able to resolve correctly from the master server and the client or it will not work!!!

Definition Command Line Interface Graphical User Interface

essarily graceful

================="

here robot_type is tld, acs, . . .) robot_number -list (where robot_type is tld, acs, . . .)

-rc1 -p -mm

nd should be deleted when no longer necessary.

ult is VERBOSE=1)