Svc Srm Setup Guide(Ga)

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Proven Practice: IBM® System Storage™ SAN Volume Controller & VMware® vCenter Site Recovery Manager Setup Guide

Cormac Hogan VMware Global Support Services September 2009 Rev 1.0 GA

Page 1 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Introduction........................................................................... 3 Disclaimer.............................................................................. 3 Step 1: Zoning........................................................................ 4 Step 2: Initial SVC Configuration – Node Front Panel.................... 5 Step 3: Installation of IBM SVC Console Software........................ 6 Step 4: Launching the SVC Console.......................................... 13 Step 5: Create an SVC Cluster ................................................ 14 Step 6: Add a new node to the cluster...................................... 25 Step 7: Create an MDISK........................................................ 26 Step 8: Create a Managed Disk Group...................................... 27 Step 9: Register a Host.......................................................... 30 Step 10: Create a VDISK........................................................ 31 Step 11: Map the VDISK to a HOST.......................................... 35 Step 12: Discover SVC Storage from the ESX............................ 36 Step 13: VMFS creation on SVC LUN........................................ 37 Step 14: Modify I/O Groups Copy Services Space....................... 38 Step 15: Setup Metro & Global Mirror Relationship..................... 39 Step 16: Install SRM.............................................................. 44 Step 17: Install the IBM SVC SRA............................................ 49 Step 18: Install the SRM Plugin............................................... 53 Step 19: Pairing the SRM sites................................................. 55 Step 20: Array Manager & Inventory........................................ 58 Step 21: Create a Protection Group.......................................... 66 Step 22: Create a Recovery Plan.............................................. 68 Step 23: Test Failover............................................................ 70 Step 24: Troubleshooting SRM using the SVC CLI....................... 74 Trademarks ......................................................................... 85 Summary............................................................................. 85 About the Author................................................................... 86

Page 2 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Introduction This document is part of a series of how-to documents which describe how to configure storage arrays with VMware® vCenter Site Recovery Manager. These documents are hosted on the VMware VI-OPS site at http://viops.vmware.com/home/community/availability. This guide will take you through the deployment of an IBM® System Storage™ SAN Volume Controller (SVC) configuration for use with VMware Site Recovery Manager. It will detail how to configure your SVC environment to create and present storage to your ESX server, how to configure replication, how to setup up VMware vCenter Site Recovery Manager to work with SVC and finally how to test that your environment is working correctly by performing a test failover. The SVC environments that I setup for the purposes of this proven practice are minimal, and are simply used to show the integration with SRM, and demonstrate a test failover scenario. This guide also provides you with details on how to troubleshoot issues with SVC & SRM, as well as highlighting a number of common ‘gotchas’. For the latest IBM support information on VMware vCenter Site Recovery Manager, & the Storage Replication Adapter for SVC, please go to: http://www01.ibm.com/support/docview.wss?rs=1114&context=HW2C2&dc=D400&q1=ssg1 *&uid=ssg1S4000818&loc=en_US&cs=utf-8〈=en.

Disclaimer This guide is by no means comprehensive. It is literally a step-by-step guide to get the SVC cluster to a point where it can be used with VMware vCenter Site Recovery Manager for POC, demonstrations and training purposes. For production deployments, you are urged to refer to the detailed documentation provided by both IBM and VMware. You use this proven practice at your discretion. VMware and the author do not guarantee any results from the use of this proven practice. This proven practice is provided on an as-is basis and is for demonstration purposes only.

Page 3 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Step 1: Zoning Before you begin with the SVC configuration, particular attention must be paid to the zoning of the SAN. IBM recommend that you have a zone for the SVC nodes within a cluster, another zone between the SVC nodes and the disk subsystems, and a zone between each front-end ESX host HBA and the SVC. In a nutshell, you must: 1. Zone the nodes which participate in a cluster to each other. 2. Zone the clusters which are going to do Global/Metro mirroring to each other. If there are two SAN Volume Controller clusters with Metro Mirror (formerly referred to as Peer-to-Peer Remote Copy (PPRC)) and Global Mirror services between them, zoning must be set so that all the nodes of both clusters can see each other. 3. Zone the cluster nodes to the back-end storage arrays. All SVC nodes (in an SVC cluster) must be able to see the same set of disk subsystem ports on each disk subsystem controller. 4. Zone the cluster nodes to the front-end ESX servers IBM SVC documentation provides considerable detail on best practices for implementing zoning in an SVC environment. Further discussion on the zoning configuration is beyond the scope of this document, though we will reference the zoning configuration from time to time during this guide.

Page 4 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Step 2: Initial SVC Configuration – Node Front Panel The environment on which I was working on contained 4 nodes. I split the nodes up into 2 x 2 node clusters, and setup replication between the clusters. To do the initial setup, you need to select which nodes in the cluster will be your primary nodes, and configure them accordingly via the front panel of the nodes. You only need to do this front panel setup on the node that is going to be the ‘primary’ node in the cluster. This is called the Config Node. The other nodes or nodes can be left un-configured from a front panel perspective. When you create a cluster from the front panel, be sure to save off the PASSWORD value that is displayed in the front panel. You will need this later on for cluster setup and the addition of nodes to the cluster via IBM San Volume Controller Console. There is no way to retrieve this password once it has gone from the front panel. You will need to delete and recreate the cluster if you forget to make a note of it. To reset the cluster settings via the front panel, (in case you make a mistake when putting in the details) go to ‘Delete cluster?’ view and whilst keeping the up arrow key pressed and hit the select key. The IBM System Storage SAN Volume Controller Installation Guide has a whole chapter dedicated to using the front panel of the SAN Volume Controller. Refer to this document for additional details on the initial configuration of the primary nodes from the front panel. Once you have your two primary nodes of the protected and recovery cluster created, and having noted down the IP address information as well as the password information, you can now proceed to step 2 with the deployment of the IBM SVC Console software to complete the SVC configuration.

Page 5 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Step 3: Installation of IBM SVC Console Software When deploying IBM SVC Console Software, it is recommended that you deploy it twice. One SVCC deployment will manage the SVC cluster at the protected site – the other SVCC deployment will manage the SVC cluster at the recovery site. Double click on the IBM SVCC exe. This can be retrieved from: http://www-01.ibm.com/support/docview.wss?rs=591&uid=ssg1S1002888 Click in Installation Wizard:

Click Next:

Accept the EULA:

Page 6 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

You should now receive a window regarding a requirement for the PuTTY utility:

So we need to install PuTTY … you will find the binary on the IBM SVCC install media as shown below…

Double click on the PuTTY installer, and then click Next at the welcome screen:

Leave the install location at the default & click Next:

Page 7 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Click Next at the Start Menu Folder:

Click Next at the additional tasks screen:

Click Install:

Click Finish:

Page 8 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

After PuTTY has been installed, click back and then Next in the main SVC Console installation wizard so that it now detects the PuTTY installation…

Click OK to acknowledge the IPv6 popup message:

Click Next to keep the default install location for the software. What now appears are instructions on how to create an SSH2 RSA private key file for connecting to the SVC. Leave the PuTTY private key file name at the default as shown below. Leave this window open while you set up the keys:

Page 9 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Launch PuttyGen via Start -> Programs -> PuTTY -> PuTTYgen. Click on the Generate button.

This launches the PuTTY Key Generator. You need to save off both the public key and the private key by clicking on the buttons highlighted below:

Save Public Key - C:\Program Files\PuTTY\icat.pubk Save Private Key - C:\Program Files\PuTTY\icat.ppk Click Yes to indicate that you want to save the keys without a passphrase. Once the keys are saved off, you can quit PuttyGen via File > Exit and continue with the SVCC installer..

Click Next on the IBM SVC controller Installer – PuTTY configuration window.

Page 10 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

The next window defines the CIM ports. These are the ports to which the IBM SVC SRA will use to communicate to the SVC. By default this port is 5989. Leave it at the default. Click Next:

Click Next:

Click Install:

When the install is complete, click Next.

Page 11 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

As mentioned as the beginning, we have found that it is less problematic to install SVCC twice for SRM, once for management of the protected SVC and again for management of the recovery SVC. Click Finish, leaving the restart my computer button checked.

SVCC is now successfully installed.

Page 12 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Step 4: Launching the SVC Console Now we can begin to configure our SVC clusters. At this point, I currently have 2 primary nodes, one for the protected site and one for the recovery site. However my other nodes are as yet un-configured. My next step is to create a cluster at both the protected and recovery site, and then to add the un-configured nodes to the clusters so that both clusters contain 2 nodes. My first task is to launch my newly installed SVC console as shown here:

The SVC console launches and prompts for logon credentials:

Initial login is superuser/passw0rd. However you need to change the password on initial login. I changed my credentials to superuser/vmware12.

Page 13 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Step 5: Create an SVC Cluster Once logged in, you are prompted to add an SVC cluster. Click on the button to initiate this process:

Click on Add SAN volume Controller Cluster:

Since this is the first cluster that we are about to initialize, check Create (Initialize) Cluster. Click OK. If you have an already existing cluster, you do not need to check Create (Initialize) Cluster, but you can go straight in and add the IP address of the cluster.

Page 14 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

You will be met with a security alert the first time you do this:

Click Yes. You will now be prompted with a login for the cluster. User is admin. You need to use the password that was noted from the SVC front panel to connect to the SVC node.

Next you are prompted to setup the SSH Public Key for the SVC console. This is the key that you created via PuTTYGen in an earlier step.

Page 15 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

In the Add Key section (file upload), browse to the PuTTYGen public key. In the Access Level, for full authority, you need to have "administrator" privilege so you should select that radio button.

Click on Add Key and view the resulting output:

Once completed, click on Clusters in the SVCC menu on the left hand side of the screen:

Page 16 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

From the pull down menu associated with clusters, select the option to Add a Cluster:

Click on Go to the right of the task:

Now you need to login to the node in question:

The password here is that was displayed on the front panel when the cluster was created.

Page 17 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

This first screen tells you the tasks that need to be complete to create the cluster. This next screen asks you to set passwords for the cluster as well as the cluster name and IP address of the cluster. Note – I set all the passwords to vmware12

Acknowledge the message regarding cluster creation by clicking the Continue button:

Page 18 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Acknowledge the message regarding password modification by clicking the Continue button:

Populate error notification information as desired. As I was setting this up in my lab, I simply left this screen blank. Click the Update Settings button:

Click Continue:

Page 19 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

We now come to the License settings. As you’ll see from the screenshots below, I went with Capacity Licensing:

And I set my limits to 1TB. Note that your licensing may be different:

Now the final step is the SSH key configuration. Click on Continue:

Page 20 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Populate the credentials popup window with the information that you provided for this cluster earlier.

Find the SSH Key and use a label to identify it. Again, this screen shot is showing "Monitor" role instead of "administrator" role. You should choose administrator for the Access Level.

Click Add Key, and then Continue to acknowledge that an administrator role has been associated with that key.

Page 21 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Your cluster should now be created:

The cluster should now be visible in the Cluster View.

Repeat this process for the recovery side cluster. While multiple clusters can be managed by a single SVCC, we have found that having a single SVCC managing a single cluster works best with SRM.

We will now verify that the SSH key setup is working. From the host that runs SVCC, you can connect using putty as long as putty is pointing to the ssh private key – the login is admin.

Page 22 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

First, PuTTY must be setup to use the local Private key file for authentication:

Go back to Session, select Default Settings and click Save … Now open a PuTTY session to the IP address of your cluster node. Login as admin. There should be no need to provide a password if this is working correctly.

Page 23 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Some useful commands are: svcinfo lsnode

svcinfo lsnode

There are obviously very many other commands which can be run, but we will be focusing on those to do with replication and storage presentation. We will look at the CLI in more detail later on in this proven practice.

Page 24 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Step 6: Add a new node to the cluster. Now we will add our second node to the cluster. In SVCC, select Work with Nodes > Nodes and then the task Add a Node:

The Available Candidate Nodes should list those nodes which have been configured/zoned correctly for this SVC cluster. you can also choose which i/o group (0-3) to place the node, as well as providing a Node Name. If you do not see any Available Candidate nodes listed, check your zoning configuration.

Click OK when this information is populated:

It may take a moment or two before the node comes online. Once it does, information such as the WWPN and the UPS unique ID will populate. The CLI command svcinfo lsnode can also be used to view this information.

Page 25 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Step 7: Create an MDISK The next step is to present MDISKS from the back-end storage. In this case we have presented 2 x 36GB LUNs to each cluster from an EMC Symmetrix storage array, although many other different storage arrays are supported by SVC. The LUN presentation steps at the array end are beyond the scope of this document. Go to SVCC > Clusters > Work with Managed Disks > Managed Disks. Before doing the LUN presentation from the backend storage, you won’t see any disks.

After presenting the back-end storage, run the Discover MDISK task

If the LUNs have been presented properly, then will show up after this task is run. The svcinfo lsmdisk command can be run at the CLI to report the same information.

If no MDisks are discovered, then once again you will have to revisit your zoning configuration.

Page 26 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Step 8: Create a Managed Disk Group In the SVC world, VDISKS are what are presented to the outside world (in our case, ESX hosts). VDISKS are built using chunks of underlying MDISKS. In order to create VDISKs, the MDISKs need to be placed in a Managed Disk Group. This can be done by following the steps here from the SVCC:

Go to Work with Managed Disks > Managed Disk Groups & select Create an MDISK Group:

This launches the Create a Managed Disk Group Wizard. Click Next.

Page 27 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Provide a name for an MDISK Group, and select which MDISKS you wish to place in the group. In my case, I have chosen to put both of my MDISKS in my MDISK GROUP as shown below. Click Next.

Select an Extent size for the VDISKS. I left this at the default size of 16MB. This may not be the most suitable extent size for your environment as it may be too small. You are advised to refer to the SVC Best Practice Guide for Vdisk extent sizing recommendations at http://www.redbooks.ibm.com/redbooks/pdfs/sg247521.pdf

Review the configuration and click Finish:

Page 28 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Your MDISK Group should now be visible in the Managed Disk Groups section:

The managed disk groups are also visible from the CLI via the command svcinfo lsmdiskgrp:

From the SRA release notes: Important: When creating target volumes on the recovery site SAN Volume Controller make sure that the MDisk group that the volumes are created from has double the amount of space that is needed for the target volumes. This space will be used by the Storage Replication Adapter during TestFailover operations. So with ~ 72GB of MDisks available in my configuration, the maximum amount of VDisk storage that I should create is 36GB if I plan to use this configuration with VMware vCenter Site Recovery Manager.

Page 29 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Step 9: Register a Host Go to Work with Hosts > Hosts & select Create a Host:

Give a name to the host. Note that for the IBM SRA to successfully discover LUNs on IBM SVC, you need to include the string ‘vmware’ in the name of the host you are registering. You also need to select the appropriate WWNs of the HBAs on the ESX server and move them from available ports to selected ports. If no WWNs are visible, then revisit your zoning configuration and ensure your ESX is correctly zoned to the SVC. Click OK.

Your Host is now registered, and you can now map LUNs to it.

Repeat this step for the recovery site.

Page 30 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Step 10: Create a VDISK Navigate to Work with Virtual Disks > Virtual Disks & select Create VDISKs & click Go:

This launches the Create Virtual Disks wizard shown below. Click Next:

Choose an I/O Group and a Preferred Node. The default is to let the system choose. Click Next:

Page 31 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Accept the default attributes by clicking Next:

Note however that If you select the "Space-efficient" attribute for the MM/GM sources and targets, then the snapshots created during TEST FAILOVER task will also be space-efficient. In which case, you don't need the double capacity on the Mdisk group on the recovery site SVC as stated in the SRA release notes. Select the preferred MDisk Group. Since we have only created one, we select that one by clicking on its associated radio button. Also, you need to add the size of the VDISK that you wish to create. In this example I set it to 5GB. Click Next:

Provide a name for your VDISK:

Page 32 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Verify the Attributes, and if correct, click Finish:

The process of creating a VDISK now begins:

And hopefully the creation was successful. Click Close:

Page 33 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Your VDISK(s) are now visible in the Work with Virtual Disks > Virtual Disks:

Repeat this step on the recovery site, as you will eventually be setting up replication between the VDISK on the protected site and the VDISK on the recovery site.

Page 34 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Step 11: Map the VDISK to a HOST Go to Work with Virtual Disks > Virtual Disks and select Map VDISKs to a Host, and click Go:

Select a host and click OK. This is the host that you registered earlier. Again, ensure that the registered host name contains the string ‘vmware’. At this point you can also choose which LUN ID to present to the ESX. I chose a LUN Id 10.

Verify that the mapping was successful & click Close.

You do not need to repeat this step on the recovery. This part is done automatically by the SRA during a FAILOVER.

Page 35 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Step 12: Discover SVC Storage from the ESX Login to your ESX server as root and run an esxcfg-rescan & esxcfg-mpath commands:

[root@cs-tse-f114 root]# Rescanning vmhba1 ... On scsi1, removing: 1:0. On scsi1, adding: 0:10. Done. [root@cs-tse-f114 root]# Disk vmhba2:0:0 /dev/sdb Local 22:0.0 vmhba2:0:0

esxcfg-rescan vmhba1

esxcfg-mpath -l (20000MB) has 1 paths and policy of Fixed On active preferred

Disk vmhba1:0:10 /dev/sda (5120MB) has 4 paths and policy FC 4:0.0 10000000c96dc09f500507680130010e vmhba1:0:10 FC 4:0.0 10000000c96dc09f500507680140010e vmhba1:1:10 FC 4:0.0 10000000c96dc09f5005076801300048 vmhba1:2:10 FC 4:0.0 10000000c96dc09f5005076801400048 vmhba1:3:10

of Most Recently Used On active preferred Standby Standby Standby

[root@cs-tse-f114 root]#

Notice that the LUN ID that has been discovered is 10, which is what we set when we did the LUN Mapping previously. Repeat this step on the recovery site before moving onto the VMFS volume creation.

Page 36 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Step 13: VMFS creation on SVC LUN On the protected ESX, build a VMFS file system on the presented SVC LUN. Once this VMFS file system is created, you then need to create a Virtual Machine on this newly created VMFS for the discoverLUNs process of SRM to work. An alternative would be to use this LUN as an RDM and map it to the ESX. Note that this step only needs to be done on the protected site. You do not need to concern yourself with this task on the recovery site. Since this is pretty basic ESX administration, these steps are not covered in any detail. You are now ready to setup replication between the protected SVC cluster and the recovery SVC cluster.

Page 37 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Step 14: Modify I/O Groups Copy Services Space I am not sure if this is a requirement or not for SVC, but I did modify these parameters for my SVC environments. Go to Work with Nodes > I/O Groups, select your I/O group and from the pull down menu, select Modify Copy Services Space and click Go:

There is 512MB of memory to play with which can be divided up between Global/Metro Mirror, FlashCopy and VDISK Mirroring. I split the memory up as evenly as possible between the 3 services.

This is a tuneable parameter & is dependent on the environment. Please reference document http://www.redbooks.ibm.com/redbooks/pdfs/sg247574.pdf, page 15, for further details on this setting. Keep the setting consistent across both clusters.

Page 38 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Step 15: Setup Metro & Global Mirror Relationship We are now ready to create the partnership between both clusters. This must be done before we can initiate a replication of our VDISKs. Navigate to Manage Copy Services > Metro & Global Mirror Cluster Partnerships. Note the first message/warning – a partnership must be created on both clusters in order for VDISK replication to work. Click Create:

Remote Cluster Candidates should be displayed. If this is not populated, check that all the nodes in the cluster are zoned to each other. No other values need to be modified:

Click OK. The status of the partnership will be displayed.

Page 39 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

You need to repeat the same steps on the recovery site SVC to have the partnership in the FULLY CONFIGURED state.

Now the state of the cluster shows Fully Configured.

Now that the partnership is in place between the clusters, we can go ahead and put in place the VDISK replication. Make sure that the MM/GM source and target are of identical sizes. Once done, launch the relationship wizard via Managed Copy Services > Metro & Global Mirror Relationships. Note that this is done on the protected site only. At the first screen, click Next:

This first screen prompts you for details about the Auxiliary cluster. Until a partnership is created with a remote cluster, you will only see an option for an intra-cluster relationship. Once you have configured the partnership, an option for inter-cluster is also available. In this example, I have chosen a Metro Mirror relationship and an inter-cluster relationship, as well as providing a name for the relationship. Click Next:

Page 40 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

At the master (protected) VDISK filter screen, click Next.

Select the Master (protected) VDISK:

Select the Auxiliary (recovery) VDISK:

Leave the following screen at the defaults, i.e. Synchronized unchecked and no consistency groups selected.

Page 41 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

From the SRA 1.0 release notes: Note: This release of IBM SAN Volume Controller Storage Replication Adapter for VMware Site Recovery Manager does not support Consistency Groups. Verify the relationship settings, and if all ok, click Finish:

The relationship should now be visible in the Manage Copy Services > Metro & Global Mirror Relationships. Note the state is Inconsistent Stopped:

Select the relationship radio button, and choose Start Copy Process for the task list and click Go:

Page 42 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

This presents you with a bunch of options related to the Copy Process. If you are starting the MM/GM relationship for the first time, then you don't need to use FORCE. The first time through, I chose Forced start and Master -> Auxiliary direction. Click OK.

Note that before initiating the copy, I had already created a VMFS with a Virtual Machine on the SVC LUN, or alternatively the SVC LUN could be mapped to a Virtual Machine as an RDM. Note that you can keep track of the synchronization progress via the following Manage Progress > View Progress screens via the Refresh button:

Once the synchronization reaches 100%, you have now completed the setup required on the SVC. The next steps are to setup the SRM environment, create the site pairings, build the protection group and recovery plan, and finally do a test failover to ensure that the virtual machine using the SVC LUN can be recovered.

Page 43 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Step 16: Install SRM This guide was written with SRM version 4.0, although all of the steps done previously are identical for earlier versions of SRM. You will have noticed that the ESX versions are 3.x. SRM 4.0 can be used with ESX 3.x, so long as the version of vCenter server that is managing the hosts in also 4.0. I am also utilizing a new feature of SRM 4.0 called Shared Recovery Site. You do not need to do this for your installation; you can simply follow the defaults. This new feature of SRM 4.0 allows multiple instances of SRM can be managed by the same vCenter server. In this case I am going to setup an SRM environment to manage just the ESX hosts that are using the SVC. In order to use the shared recovery site feature, an additional option has to be passed at the command line called CUSTOM_SETUP=1. Here is an example script which will launch the installer with the CUSTOM_SETUP=1 option: "C:\Documents and Settings\Administrator\My Documents\Software\ SRM\VMware-srm-4.0.0-178984.exe" CUSTOM_SETUP=1" Double click on this batch file to launch the installation. At the welcome screen, click Next:

Accept the EULA and click Next:

Page 44 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Choose a destination folder for the install and click Next:

Populate with the local vCenter credentials & click Next:

Acknowledge the security warning by clicking Yes:

Page 45 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Leave the certificate source at automatic unless you wish to install a PKCS#12 certificate. Click Next:

Populate organization and unit fields for the certificate, click Next:

Populate Local Site Name and Administrator email, click Next:

Page 46 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Unless you have included a CUSTOM_SETUP=1, you will not see these next screens. Only when you choose to do a setup for Shared Recovery Site will you see this pop-up. Here I chose a Custom SRM plug-in ID:

I give the custom SRM ID a value of SVC. I also populate the organization and description fields. Click Next to continue:

Next populate the database information appropriately. This guide assumes that the database is already configured, and that you have created an ODBC DSN (in this case called SRM_SVC_PROT). Click Next:

Page 47 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Click Install:

When the installation is complete, click on the Finish button.

Ensure that the VMware vCenter Site Recovery Manager service is running:

Repeat the installation of SRM at the recovery site. Remember that if you wish to use shared recovery sites, you must ensure that the SRM ID at the remote site is identical to the SRM plug-in ID chosen at the protected site, e.g. SVC.

Page 48 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Step 17: Install the IBM SVC SRA Before starting the install of the IBM SRA for SVC, you first of all need to add the path to the perl binaries to the environment of the SRM user on the SRM server. In my case it is the administrator user. To set the path environment variable, right click on My Computer > Properties, and select the Advanced tab:

Click on the Environment Variables button, and in the System variables window, scroll down to Path. Click on the Edit button when Path is selected:

In the Variable value, append the string ‘;C:\Program Files\VMware\VMware vCenter Site Recovery Manager\external\perl-5.8.8\bin’ remembering to include the ‘;’ to separate the unique paths. Click OK to exit properties.

Page 49 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Now we can start the installation of the IBM SVC SRA. At the time of writing, the recommended IBM SVC SRA version is 1.0.1.09902, released on September 2nd, 2009. This version can be found at the IBM ftp site - ftp.software.ibm.com in the subdirectory /storage/ds_open_api/VMWARE/SVC_SRA. The file is called IBMSVCSRA10109902.zip. Extract this file and double click on the EXE to begin the installation. The first window prompts you for a language. Select English and click Next:

At the Welcome screen, click Next:

Accept the EULA:

Accept or Change the customer information:

Page 50 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Choose a Destination Location for the installation. Note that with the version of the SRA that I was using, I need to change this location from the default. By default, the SRA is being installed in ‘C:\Program Files\VMware\VMware Site Recovery Manager\scripts\SAN\IBMSVC’ which is the correct location for SRM 1.0 & vCenter 2.x. For SRM 4.0 & vCenter 4.0, the correct location is C:\Program Files\VMware\VMware vCenter Site Recovery Manager\scripts\SAN\IBMSVC. Modify the installation folder accordingly via the Change button:

Click OK, followed by Next to complete the installation. At the install complete window, click Finish.

Page 51 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

A final step is to restart the SRM service:

Now repeat this exercise on the SRM server at the recovery site.

Page 52 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Step 18: Install the SRM Plugin Log onto your vCenter server, launch your vSphere client locally and connect to your vCenter server. You should notice that there are no icons in the Solutions and Applications area.

Double click the SRM plugin install exe/msi file. You could also go to the vSphere client > Plug-ins > Manage Plugins & choose the download & install option for the SMR plug-in. At the Welcome screen for the SRM plug-in, click Next.

Accept the EULA:

Page 53 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Click Install

Click Finish when the install completes:

Log out of your vSphere client and log back in. The SRM icon should now be visible in the Solutions and Applications section.

Repeat the installation of the plug-in on vCenter at the other site.

Page 54 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Step 19: Pairing the SRM sites This is only done on the protected site. Click on the Site Recovery icon, then click on Connect to VMware vCenter Site Recovery Manager:

If you have opted to do the Shared Recovery site deployment option, you will be prompted with a list of SRM IDs to connect to. Since I called mine SVC, I select that SRM ID and click Open:

The first step is to configure the connection between the protected and remote sites. At the summary screen, click on the configure link next to Connection in the Protection Setup window:

Page 55 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

The first window which pops up is a request to provide information about the remote vCenter server. Populate it accordingly with either IP or hostname of the remote vCenter and click Next:

Acknowledge the remote server certificate error by clicking OK:

Provide credentials for the remote vCenter server. Note that if you are using active directory, the domain should be included in the username. Click Next:

Page 56 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Acknowledge the remote server certificate error by clicking OK:

While reciprocity is being established, populate any credentials pop-ups that might appear on the protected and recovery vCenters for SRM:

If all is well, reciprocity will be established. Click Finish.

The Connection should now appear as Connected in SRM:

Page 57 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Step 20: Array Manager & Inventory This step only needs to be done at the protected site. Click on the Configure option against Array Managers in the Protection Setup window:

This launches the Configure Array Managers utility. The first step is to add a protected array by clicking on the Add button.

If you get the message pop-up about unable to find any array script files, this most likely means that you did not install the SRA in the correct location on the SRM server.

Remember that we had to modify the install location from: ‘C:\Program Files\VMware\VMware Site Recovery Manager \scripts\SAN\IBMSVC’ To: ‘C:\Program Files\VMware\VMware vCenter Site Recovery Manager \scripts\SAN\IBMSVC’. Recheck your SRA install. If you have installed it in the incorrect location, remove it and install it again, and restart the SRM service and proceed once again with the Array Manager configuration.

Page 58 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

The Add button should now display a window into which you can add Array Manager information. Populate with a display name for the array, The IP address with CIM port (default 5989) of your IBM SVCC, along with the username and password of your SVCC, and click the Connect button:

If you get an error stating that the XML document is empty, check the log file to see what the exact failure is:

The log file is located on the SRM server in the following location:

Page 59 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Examining the most recent log file (in this case vmware-dr-24.log), we see the following error: [2009-08-25 11:35:24.915 01540 verbose 'SysCommandLineWin32'] Starting process: "C:\\Program Files\\VMware\\VMware vCenter Site Recovery Manager\\external\\perl5.8.8\\bin\\perl.exe" "C:/Program Files/VMware/VMware vCenter Site Recovery Manager/scripts/SAN/IBMSVC/command.pl" [2009-08-25 11:35:25.227 01540 trivia 'PrimarySanProvider'] discoverArrays's output: [#1] C:/Program Files/VMware/VMware vCenter Site Recovery Manager/scripts/SAN/IBMSVC [2009-08-25 11:35:25.227 01540 error 'PrimarySanProvider'] discoverArrays's errors: [#1] 'perl' is not recognized as an internal or external command, [#1] operable program or batch file. [#1] [2009-08-25 11:35:25.227 01540 info 'PrimarySanProvider'] discoverArrays exited with exit code 0 [2009-08-25 11:35:25.227 01540 trivia 'PrimarySanProvider'] 'discoverArrays' returned [2009-08-25 11:35:25.227 01540 error 'PrimarySanProvider'] Failed to retrieve script results [2009-08-25 11:35:25.227 01540 info 'ArrayManagerImpl.QueryInfoTask-Task'] Work function threw std::exception: XML document is empty [2009-08-25 11:35:25.227 01540 info 'ArrayManagerImpl.QueryInfoTask-Task'] Fault: [#1] (dr.fault.InternalError) { [#1] dynamicType = , [#1] faultCause = (vmodl.MethodFault) null, [#1] reason = "XML document is empty", [#1] msg = "", [#1] }

This suggests that the perl binary is not being found during a discoverArrays task, suggesting that we have not configured our ‘path’ environment variable correctly on the SRM server to include the path to the perl binaries. Revisit your environment variable configuration on your SRM servers and ensure that the path variable has been configured as described in earlier steps. Once the path environment variable has been modified correctly, restart the SRM service. Reconnect to SRM, and retry once again to configure Array Managers. This time the connect should be successful. If so, the array manager information should populate with the protected SVC array id. Ensure the array is selected before clicking OK:

Page 60 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

This should take you to the next step of array managers and show you the peer array if replication is working. Click Next to proceed to the configuration of the recovery site array.

Add the Recovery Site Array Managers configuration window, click Add:

Populate this window with the attributes of your recovery site array, and click Connect:

Page 61 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

If the IBM SVC SRA has been installed in the correct location on the recovery site SRM server, and the path environment variable has been setup properly, then the connect should do a discoverArrays successfully and display the following information about the recovery array. Click OK:

The SRA now updates the Replicated Array Pairs part of the window. If there is a valid replication between the protected and recovery arrays, we should see a green check mark in the window as follows:

The green check mark icon is very important as this suggests that the protected and recovery sites are in sync and functioning properly. If this does not appear, you need to revisit the replication configuration and verify that it is operational. Click Next to initiate a discoverLUNs task whereby we find out if the LUN in question is presented to our primary ESX server, and contains a VMFS and a virtual machine, or is mapped as an RDM to a virtual machine.

Page 62 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

If the discoverLUNs is successful, we will be shown the Datastore Group which contains our virtual machines:

Click Finish to complete the array manager configuration. Note that the discoverLUNs task will fail if the initiator records created when registering the hosts on the SVC cluster do not include the string ‘vmware’. No devices will be displayed when doing the discovery. The vmware-dr-X.log file will contain a discoverLUNs failure similar to the following: [2009-08-24 13:42:38.037 02660 trivia 'PrimarySanProvider'] discoverLuns's output: C:/Program Files/VMware/VMware vCenter Site Recovery Manager/scripts/SAN/IBMSVC[ Mon Aug 24 13:42:35.475 discoverLuns trivia ] Begin getArrayTypeAndLuns() [#1] [ Mon Aug 24 13:42:35.490 discoverLuns trivia ] CIM_ObjectManager [#1] [ Mon Aug 24 13:42:36.709 discoverLuns trivia ] Begin getSVCLuns() [#1] [ Mon Aug 24 13:42:36.709 discoverLuns trivia ] IBMTSSVC_StorageHardwareID [#1] [ Mon Aug 24 13:42:36.834 discoverLuns trivia ] IBMTSSVC_RemoteStorageSynchronized [#1] [ Mon Aug 24 13:42:37.662 discoverLuns trivia ] IBMTSSVC_HardwareIDStorageVolumeView [2009-08-24 13:42:38.037 02660 info 'PrimarySanProvider'] discoverLuns exited with exit code 0 [2009-08-24 13:42:38.037 02660 trivia 'PrimarySanProvider'] 'discoverLuns' returned [#1] [#1] 4 [#1] [#1] [2009-08-24 13:42:38.037 02660 info 'PrimarySanProvider'] Return code for discoverLuns: 4 [2009-08-24 13:42:38.037 02660 error 'PrimarySanProvider'] The scripts returned an error, leaving the temporary file 'C:\WINDOWS\TEMP\vmware-SYSTEM\dr-sanprovider2660-0' [2009-08-24 13:42:38.037 02660 error 'PrimarySanProvider'] Unknown error encountered by the script [2009-08-24 13:42:38.037 02660 verbose 'PropertyProvider'] RecordOp ASSIGN: status, array-5037 [2009-08-24 13:42:38.037 02660 trivia 'PrimarySanProvider'] Status of array 'array5037' is set to 'syncFailed' [2009-08-24 13:42:38.037 02660 warning 'PrimarySanProvider'] Failed to re-sync with storage array: (dr.san.fault.ArrayUnknownFault) { [#1] dynamicType = , [#1] faultCause = (vmodl.MethodFault) null, [#1] command = "discoverLuns", [#1] msg = "", [#1] }

Ensure that the hostnames registered on the SVC cluster for your ESX servers contain the string ‘vmware’.

Page 63 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

The next step is to configure inventory mappings. This is done at the protected site only. Click on the configure link against Inventory Mappings in the Protection Setup window.

In the Recovery Site Resources column, right click on the ‘None selected’ field and select Configure.

Map the resources appropriately from the protection site to the recovery site:

Page 64 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

That completes the array manager setup and the inventory mapping setup.

Proceed to the next step to create a protection group.

Page 65 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Step 21: Create a Protection Group The protection group creation is done at the recovery site only. In the Summary screen for Site Recovery on the protected site, click on the Create link against Protection Groups in the Protection Setup window:

Give the Protection Group a name and click Next:

Select a datastore group to protect. In this case it is called SVC_Disk. When you select the datastore group in the top window, a list of VM on the datastore(s) will be displayed in the bottom half of the window. In this example, there is only a single VM called dummy on the datastore. Click Next:

Page 66 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Select a datastore on which to place your place-holder VM on the recovery site. Remember that in this configuration, my SVC storage is mapped to ESX 3.x hosts. Therefore my place-holder datastore must be mapped to an ESX 3.x host on the recovery site. If you try to map it to an ESX 4.0 host, you will receive an error which tells you that the place-holder VM cannot be created because of compatibility issues. Unable to create placeholder virtual machine at the recovery site: Host, resource pool and datastore are not compatible. Once the correct place-holder datastore is selected, the protection group should be created. Check the status of the protection group before continuing:

Proceed to the next step where we will be creating a recovery plan.

Page 67 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Step 22: Create a Recovery Plan This step is done on the recovery site only. Connect to your recovery site vCenter server, connect to the SRM instance, and in the Recovery Setup window, click on the create link for Recovery Plans:

Provide a name for the Recovery Plan and click Next:

Select a Protection Group for this Recovery Plan. Select the PG created in the previous step:

Page 68 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Decide how long you wish to wait for VM heartbeats to begin. By default this is 600 seconds, but this can be reduced to 60 to speed things up for the purposes of testing. However for production environments, the recommendation is not to change these values.

The next window asks about networking. I will leave this at the default of Auto for the purposes of testing. Auto will create a bubble network when testing failovers so it does not interfere with the production network. Click Next:

We are not going to suspend any VMs as part of the recovery plan so click Finish at the final window:

The recovery plan is now complete, so we can proceed with a test Failover.

Page 69 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Step 23: Test Failover Staying on the recovery site, select the Recovery Plan followed by the Recovery Steps tab. Break out the various steps such as prepare storage and recover virtual machines as shown below:

In the top left hand corner, there is a green icon which is labelled test. Click this to proceed with a test failover. Answer Yes to the continue popup. Note: Earlier we made the recommendation about having separate SVC Consoles for managing the protected site and the recovery site. If you did not follow this guideline, this is where you will have difficulty. If you use the same SVCC to manage both sites, then a testFailover operation will fail with the following error: Non-fatal error information reported during execution of SRA: Failed to create lun snapshots. The vmware-dr-X.log file contains additional errors: [#3] [ Thu Aug 27 10:45:10.143 testFailover Error ] The CIMOM may not be functioning correctly.CIM_ERR_FAILED [2009-08-27 10:45:10.237 01836 error 'SecondarySanProvider'] testFailover's errors: org.sblim.wbem.cim.CIMException: CIM_ERR_FAILED; nested exception is: [#3] java.lang.NullPointerException [#3] at org.sblim.wbem.client.CIMClientXML.invokeMethod(CIMClientXML.java:778) [#3] at org.sblim.wbem.client.CIMClient.invokeMethod(CIMClient.java:2142) [#3] at testFailover.deleteSVCFC(testFailover.java:1218) [#3] at testFailover.getArrayTypeAndInfo(testFailover.java:210) [#3] at testFailover.main(testFailover.java:163) [#3] Caused by: java.lang.NullPointerException [#3] at org.sblim.wbem.xml.CIMXMLBuilderImpl.createINSTANCENAME(CIMXMLBuilderImpl.java:828) [#3] at org.sblim.wbem.xml.CIMClientXML_HelperImpl.invokeMethod_request(CIMClientXML_HelperImp l.java:552) [#3] at org.sblim.wbem.client.CIMClientXML.invokeMethod(CIMClientXML.java:734) [#3] ... 4 more

Page 70 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

If you experience this error, you will have to deploy a second SVCC, and remove one of the clusters from the SVCC (which is currently managing two clusters) and put it under the control of the new SVCC so that each instance managed a single SVC. IBM inform me that if the code level on the SVC is 4.3.1.7 or greater, then you don't have to connect to a master console. Instead you can directly connect to the SVC cluster (eliminating the need for master consoles on both sides for SRM). Another issue observed was similar to the above, where a testFailover failed after 20 seconds or so in the prepare storage phase. The error reported in the Recovery Plan is the same as before: Non-fatal error information reported during execution of SRA: Failed to create lun snapshots. The vmware-dr-X.log reports the following events: [#5] [ Mon Sep 28 13:38:23.348 testFailover trivia ] Prepare FC for"root/ibm:IBMTSSVC_StorageVolume.CreationClassName=\"IBMTSSVC_StorageVolume\",Devic eID=\"0\",SystemCreationClassName=\"IBMTSSVC_IOGroup\",SystemName=\"10.21.64.234:0\""r eturned: 0 [#5] [ Mon Sep 28 13:38:24.317 testFailover trivia ] testFailover::ModifySynchronization [#5] [ Mon Sep 28 13:38:24.489 testFailover trivia ] Start FC for "root/ibm:IBMTSSVC_StorageVolume.CreationClassName=\"IBMTSSVC_StorageVolume\",DeviceID =\"0\",SystemCreationClassName=\"IBMTSSVC_IOGroup\",SystemName=\"10.21.64.234:0\""retu rned: 4096 [#5] [ Mon Sep 28 13:38:24.630 testFailover trivia ] Acquiring lock... [#5] [ Mon Sep 28 13:38:24.692 testFailover trivia ] testFailover::ControllerConfigurationService [#5] [ Mon Sep 28 13:38:24.739 testFailover trivia ] testFailover::exposePaths() [#5] 1 unmask successful [#5] [ Mon Sep 28 13:38:24.833 testFailover trivia ] Attach Volumes returned: 0 [#5] [ Mon Sep 28 13:38:24.833 testFailover trivia ] Releasing lock... [#5] [ Mon Sep 28 13:38:24.848 testFailover Error ] testFailover::createSVCFC(): 203 [#5] [ Mon Sep 28 13:38:24.848 testFailover trivia ] Exit failoverTest End(): 203 [#5] [ Mon Sep 28 13:38:24.848 testFailover trivia ] End testFailover::createSVCFC() [#5] [ Mon Sep 28 13:38:24.848 testFailover trivia ] End testFailover::getArrayTypeAndInfo() [2009-09-28 13:38:24.895 00568 error 'SecondarySanProvider'] testFailover's errors: java.lang.StringIndexOutOfBoundsException [#5] at java.lang.String.substring(String.java:1088) [#5] at testFailover.createSVCFC(testFailover.java:900) [#5] at testFailover.getArrayTypeAndInfo(testFailover.java:207) [#5] at testFailover.main(testFailover.java:163)

This is a known issue which is addressed in a hot-fix version of the IBM SRA v1.0. The version is 1.0.1.09902, released on September 2nd, 2009. This hot-fix version can be found at the IBM ftp site - ftp.software.ibm.com in the subdirectory /storage/ds_open_api/VMWARE/SVC_SRA. The file is called IBMSVCSRA10109902.zip.

Page 71 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

If the test failover is successful, the plan should wait after the virtual machine has been brought online at the recovery site, and the following message displayed in the recovery plan:

You can now take this opportunity to examine the failed over virtual machine, and its storage and network resources. Here is the virtual machine in the Hosts and Clusters view. Note the datastore and network resources:

Page 72 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Here is the network view of the ESX showing the bubble network:

Here is the storage view of the ESX showing the replicated LUN presented to the recovery ESX. Since the VMFS file system is replicated, it is treated as a snapshot by the recovery side ESX, and so it has to be resignatured – thus, the snap label:

Return to the SRM view, and click on the Continue link in the Recovery Steps to end the test failover. This should undo the VM registration, and storage/network mappings.

Page 73 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Step 24: Troubleshooting SRM using the SVC CLI If you have successfully configured your ssh public/private keys for the admin user on the SVC Console, then you can login as the admin user to each of your clusters and look at the status of your MDisks, VDisks, FlashCopy, replication and host mappings. This can be very useful for troubleshooting why certain operations are not working as expected from SRM on SVC. The main command which will help you is svcinfo. To get a list of options that you can run with svcinfo, use svcinfo -? IBM_2145:SVC_VMW_2:admin>svcinfo -? The following actions are available with this command : catauditlog caterrlog caterrlogbyseqnum ls2145dumps lsauditlogdumps lsauth lscimomdumps lscluster lsclustercandidate lscontroller lscontrollerdependentvdisks lscopystatus lsdiscoverystatus lsemailuser lserrlogbyfcconsistgrp lserrlogbyfcmap lserrlogbyhost lserrlogbyiogrp lserrlogbymdisk lserrlogbymdiskgrp lserrlogbynode lserrlogbyrcconsistgrp lserrlogbyrcrelationship lserrlogbyvdisk lserrlogdumps lsfabric lsfcconsistgrp lsfcmap lsfcmapcandidate lsfcmapdependentmaps lsfcmapprogress lsfeaturedumps lsfreeextents lshbaportcandidate lshost lshostiogrp lshostvdiskmap lsiogrp lsiogrpcandidate lsiogrphost lsiostatsdumps lsiotracedumps lslicense lsmdisk

Page 74 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

lsmdiskcandidate lsmdiskextent lsmdiskgrp lsmdisklba lsmdiskmember lsmigrate lsnode lsnodecandidate lsnodevpd lsrcconsistgrp lsrcrelationship lsrcrelationshipcandidate lsrcrelationshipprogress lsrepairsevdiskcopyprogress lsrepairvdiskcopyprogress lssevdiskcopy lssoftwaredumps lssoftwareupgradestatus lssshkeys lstimezones lsvdisk lsvdiskcopy lsvdiskdependentmaps lsvdiskextent lsvdiskfcmappings lsvdiskhostmap lsvdisklba lsvdiskmember lsvdiskprogress lsvdisksyncprogress showtimezone There are obviously too many commands to cover in great detail here, so please refer to the IBM SVC CLI guide which will give you further details. We’ll concentrate on looking at this from the storage perspective. First, lets take a look at any managed disk groups using the svcinfo lsmdiskgroup: IBM_2145:SVC_VMW_2:admin>svcinfo lsmdiskgrp id name status capacity extent_size free_capacity real_capacity overallocation warning 0 MDISKGRP_SRM online 71.5GB 16 66.5GB 5.00GB 6 0

mdisk_count vdisk_count virtual_capacity used_capacity 2 5.00GB

Let’s look at the managed disk group MDISK_SRM in more detail: IBM_2145:SVC_VMW_2:admin>svcinfo lsmdiskgrp MDISKGRP_SRM id 0 name MDISKGRP_SRM status online mdisk_count 2 vdisk_count 1 capacity 71.5GB extent_size 16 free_capacity 66.5GB virtual_capacity 5.00GB used_capacity 5.00GB real_capacity 5.00GB overallocation 6 warning 0 IBM_2145:SVC_VMW_2:admin>

Page 75 of 86

1 5.00GB

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Remember that one of the pre-requisites of the MDISK group is to have double the amount of space required for volumes. This is because FlashCopy needs this additional space to clone VDisks during test failovers. Since we are using only 5GB of a possible 71.5GB, we mote than meet this requirement. Next let’s have a look at the managed disks using the svcinfo lsmdisk command: IBM_2145:SVC_VMW_2:admin>svcinfo lsmdisk id name status mode mdisk_grp_id mdisk_grp_name capacity ctrl_LUN_# controller_name UID 0 mdisk0 online managed 0 MDISKGRP_SRM 36.0GB 0000000000000052 controller0 0103001060060480000290301014533030304234000000000000000000000000 1 mdisk1 online managed 0 MDISKGRP_SRM 36.0GB 0000000000000053 controller1 0103001060060480000290301014533030304235000000000000000000000000

Looking at one of my MDisks in more detail: IBM_2145:SVC_VMW_2:admin>svcinfo lsmdisk mdisk0 id 0 name mdisk0 status online mode managed mdisk_grp_id 0 mdisk_grp_name MDISKGRP_SRM capacity 36.0GB quorum_index 0 block_size 512 controller_name controller0 ctrl_type 4 ctrl_WWNN 5006048C5369158F controller_id 0 path_count 2 max_path_count 2 ctrl_LUN_# 0000000000000052 UID 0103001060060480000290301014533030304234000000000000000000000000 preferred_WWPN 5006048C5369158F active_WWPN 5006048C5369158F IBM_2145:SVC_VMW_2:admin>

And finally let’s take a look at my VDisks using the svcinfo lsvdisk command. This is the device which is presented as a LUN to my ESX: IBM_2145:SVC_VMW_2:admin>svcinfo lsvdisk id name IO_group_id IO_group_name status mdisk_grp_id mdisk_grp_name capacity type FC_id FC_name RC_id RC_name vdisk_UID fc_map_count copy_count fast_write_state 0 SRM_VDISK 0 io_grp0 online 0 MDISKGRP_SRM 5.00GB striped 0 MM_for_SRM 6005076801FA0007A000000000000004 0 1 empty IBM_2145:SVC_VMW_2:admin>

Page 76 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Note first that the RC_id & RC_name field are populated, indicating that this volume is involved in some sort of Remote Copy. However the FC_id and FC_name fields are blank, meaning that it is not involved in any FlashCopy operations. Let’s look at this VDisk in more detail: IBM_2145:SVC_VMW_2:admin>svcinfo lsvdisk SRM_VDISK id 0 name SRM_VDISK IO_group_id 0 IO_group_name io_grp0 status online mdisk_grp_id 0 mdisk_grp_name MDISKGRP_SRM capacity 5.00GB type striped formatted yes mdisk_id mdisk_name FC_id FC_name RC_id 0 RC_name MM_for_SRM vdisk_UID 6005076801FA0007A000000000000004 throttling 0 preferred_node_id 2 fast_write_state empty cache readwrite udid 0 fc_map_count 0 sync_rate 50 copy_count 1 copy_id 0 status online sync yes primary yes mdisk_grp_id 0 mdisk_grp_name MDISKGRP_SRM type striped mdisk_id mdisk_name fast_write_state empty used_capacity 5.00GB real_capacity 5.00GB free_capacity 0.00MB overallocation 100 autoexpand warning grainsize IBM_2145:SVC_VMW_2:admin>

Since we can see that it is involved in remote copy, lets have a look at the remote copy relationship information using the lsrcrelationship option: IBM_2145:SVC_VMW_2:admin>svcinfo lsrcrelationship id name master_cluster_id master_cluster_name master_vdisk_id master_vdisk_name aux_cluster_id aux_cluster_name aux_vdisk_id aux_vdisk_name primary consistency_group_id consistency_group_name state bg_copy_priority progress copy_type 0 MM_for_SRM 000002007F800090 SVC_VMW_1 0 SRM_VDISK 000002007E8001E8 SVC_VMW_2 0 SRM_VDISK master consistent_synchronized 50 metro

Page 77 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

The name of the relationship is MM_for_SRM. Display further details using: IBM_2145:SVC_VMW_2:admin>svcinfo lsrcrelationship MM_for_SRM id 0 name MM_for_SRM master_cluster_id 000002007F800090 master_cluster_name SVC_VMW_1 master_vdisk_id 0 master_vdisk_name SRM_VDISK aux_cluster_id 000002007E8001E8 aux_cluster_name SVC_VMW_2 aux_vdisk_id 0 aux_vdisk_name SRM_VDISK primary master consistency_group_id consistency_group_name state consistent_synchronized bg_copy_priority 50 progress freeze_time status online sync copy_type metro IBM_2145:SVC_VMW_2:admin>

Here you can see the master cluster information and the auxiliary cluster information. You will also notice that no consistency groups are involved. At the time of writing, consistency groups were not supported on SVC when used with SRM. Finally you can see that the copy_type is metro. Lastly lets see if there are any FlashCopy replications going on at all – use the command svcinfo lsfcmap to look at this. IBM_2145:SVC_VMW_2:admin>svcinfo lsfcmap IBM_2145:SVC_VMW_2:admin>

No FlashCopy operations going on. Finally let’s take a look at the host mappings. We can do this using the following commands. First we want to see which hosts are registered against this SVC. IBM_2145:SVC_VMW_2:admin>svcinfo lshost id name port_count 0 f115_vmware 1

iogrp_count 1

IBM_2145:SVC_VMW_2:admin>svcinfo lshost f115_vmware id 0 name f115_vmware port_count 1 type generic mask 1111 iogrp_count 1 WWPN 10000000C96DC38A node_logged_in_count 2 state active

Here we can see that the WWN associated with hostname f115_vmware is active and logged into our cluster on both nodes. Also note that the string ‘vmware’ needs to appear in the name of the host if the current version of IBM’s SRA is to work with SRM.

Page 78 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Now let’s see which VDisks are mapped to this host: IBM_2145:SVC_VMW_2:admin>svcinfo lshostvdiskmap id name SCSI_id vdisk_id wwpn vdisk_UID 0 f115_vmware 10 0 10000000C96DC38A 6005076801FA0007A000000000000004

vdisk_name SRM_VDISK

The VDisk SRM_VDISK is mapped to this host. So everything looks good. Let’s now initiate a test failover and see what changes occur on the recovery site SVC cluster. One of the first things that happens during the prepare storage stage is that a FlashCopy operation is initiated which clones the original replicated VDisk at the recovery site SVC. We can see this using a command we saw earlier: IBM_2145:SVC_VMW_2:admin>svcinfo lsfcmap id name source_vdisk_id source_vdisk_name target_vdisk_id target_vdisk_name group_id group_name status progress copy_rate clean_progress incremental 0 fcmap0 0 SRM_VDISK 1 vdisk1 copying 0 0 100 off

Now we can see the FlashCopy operation has been initiated. Looking at the FlashCopy operation in more detail: IBM_2145:SVC_VMW_2:admin>svcinfo lsfcmap fcmap0 id 0 name fcmap0 source_vdisk_id 0 source_vdisk_name SRM_VDISK target_vdisk_id 1 target_vdisk_name vdisk1 group_id group_name status copying progress 0 copy_rate 0 start_time 090826121005 dependent_mappings 0 autodelete off clean_progress 100 clean_rate 50 incremental off difference 100 grain_size 256 IO_group_id 0 IO_group_name io_grp0 IBM_2145:SVC_VMW_2:admin>

Now we can see that the source disk, SRM_DISK, is being replicated to a target disk, vdisk1 – vdisk1 will be the disk that is presented to our recovery site ESX as part of the recovery plan.

Page 79 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Let’s take a closer look at the VDisks: IBM_2145:SVC_VMW_2:admin>svcinfo lsvdisk id name IO_group_id mdisk_grp_id mdisk_grp_name capacity FC_name RC_id RC_name fc_map_count copy_count fast_write_state 0 SRM_VDISK 0 0 MDISKGRP_SRM 5.00GB fcmap0 0 MM_for_SRM 6005076801FA0007A000000000000004 1 1 vdisk1 0 0 MDISKGRP_SRM 5.00GB fcmap0 6005076801FA0007A000000000000007 1 IBM_2145:SVC_VMW_2:admin>

IO_group_name status type FC_id vdisk_UID io_grp0 striped 1

online

0 empty

io_grp0 striped 1

online

0 empty

We can now clearly see that there are two VDisks, my original SRM_VDISK and the FlashCopy VDisk called vdisk1 which has been created by the IBM SVC SRA during the test Failover operation. And looking at some further details about the VDisk: IBM_2145:SVC_VMW_2:admin>svcinfo lsvdisk SRM_VDISK id 0 name SRM_VDISK IO_group_id 0 IO_group_name io_grp0 status online mdisk_grp_id 0 mdisk_grp_name MDISKGRP_SRM capacity 5.00GB type striped formatted yes mdisk_id mdisk_name FC_id 0 FC_name fcmap0 RC_id 0 RC_name MM_for_SRM vdisk_UID 6005076801FA0007A000000000000004 throttling 0 preferred_node_id 2 fast_write_state empty cache readwrite udid 0 fc_map_count 1 sync_rate 50 copy_count 1 copy_id 0 status online sync yes primary yes mdisk_grp_id 0 mdisk_grp_name MDISKGRP_SRM type striped mdisk_id mdisk_name fast_write_state empty used_capacity 5.00GB real_capacity 5.00GB free_capacity 0.00MB overallocation 100 autoexpand warning grainsize

Page 80 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

IBM_2145:SVC_VMW_2:admin>svcinfo lsvdisk vdisk1 id 1 name vdisk1 IO_group_id 0 IO_group_name io_grp0 status online mdisk_grp_id 0 mdisk_grp_name MDISKGRP_SRM capacity 5.00GB type striped formatted no mdisk_id mdisk_name FC_id 0 FC_name fcmap0 RC_id RC_name vdisk_UID 6005076801FA0007A000000000000007 throttling 0 preferred_node_id 1 fast_write_state empty cache none udid fc_map_count 1 sync_rate 50 copy_count 1 copy_id 0 status online sync yes primary yes mdisk_grp_id 0 mdisk_grp_name MDISKGRP_SRM type striped mdisk_id mdisk_name fast_write_state empty used_capacity 5.00GB real_capacity 5.00GB free_capacity 0.00MB overallocation 100 autoexpand warning grainsize IBM_2145:SVC_VMW_2:admin>

What we can see is that both SRM_VDISK and vdisk1 are involved in FlashCopy operations – the same FlashCopy operation in fact, where one of the VDisk’s is the source and the other VDisk is the destination. You can also see that SRM_VDISK is still involved in a Remote Copy (RC) operation but vdisk1 is not. And of course another part of the test failover operation is to present the cloned disk to the recovery ESX server so that the protected VM can be brought online at the recovery site. Let’s see if vdisk1 is presented to the recovery site ESX: IBM_2145:SVC_VMW_2:admin>svcinfo lshostvdiskmap f115_vmware id name SCSI_id vdisk_id wwpn vdisk_UID 0 f115_vmware 0 2 10000000C96DC38A 6005076801FA0007A000000000000008 0 f115_vmware 10 0 10000000C96DC38A 6005076801FA0007A000000000000004

Page 81 of 86

vdisk_name vdisk1 SRM_VDISK

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

And in more detail: IBM_2145:SVC_VMW_2:admin>svcinfo lsvdiskhostmap SRM_VDISK id name SCSI_id host_id wwpn vdisk_UID 0 SRM_VDISK 10 0 10000000C96DC38A 6005076801FA0007A000000000000004 IBM_2145:SVC_VMW_2:admin>svcinfo lsvdiskhostmap vdisk1 id name SCSI_id host_id wwpn vdisk_UID 1 vdisk1 0 0 10000000C96DC38A 6005076801FA0007A000000000000008 IBM_2145:SVC_VMW_2:admin>

host_name f115_vmware

host_name f115_vmware

What is noticeable here is that the original SRM_VDISK has a SCSI ID of 10, but the newly presented FlashCopy VDisk has a SCSI ID of 0 when presented to the recovery site ESX. This all looks pretty good, and now SRM can resignature this volume, create a test network and bring up any Virtual Machine that may have been on this volume. But what is something goes wrong? Well, SVC has logging and auditing which can be reviewed. First, let’s look at the auditing. This tracks all the operations which get run on the cluster: IBM_2145:SVC_VMW_2:admin>svcinfo catauditlog 124 090826122508 admin putty_pub_key_2 10.21.68.52 superuser 0 2 svctask mkvdisk -iogrp 0 -mdiskgrp 0 -size 5368709120 -unit b -vtype striped -cache none 125 090826122509 admin putty_pub_key_2 10.21.68.52 superuser 0 0 svctask mkfcmap -source 0 -target 2 -copyrate 0 126 090826122509 admin putty_pub_key_2 10.21.68.52 superuser 0 svctask prestartfcmap 0 127 090826122510 admin putty_pub_key_2 10.21.68.52 superuser 0 svctask startfcmap 0 128 090826122510 admin putty_pub_key_2 10.21.68.52 superuser 0 svctask mkvdiskhostmap -host 0 -scsi 0 -force 2

So here in plain text, you can see exactly the commands that the IBM SVC SRA is running on the SVC array in order to create a FlashCopy of the replicated VDisk and present it to the recovery side ESX. You can’t really ask for much better than that. The commands that the SRA runs can be run manually by you to see which one is not working. In other words, you can simulate a test failover operation by running the above commands and verifying that a LUN becomes visible on the recovery site ESX.

Page 82 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

But how do you know if the above commands succeeded or not. Well, you can look at the error log for that: IBM_2145:SVC_VMW_2:admin>svcinfo caterrlog -first 10 id type fixed SNMP_trap_raised error_type node_name sequence_number root_sequence_number first_timestamp last_timestamp number_of_errors error_code copy_id 1 vdisk no no 5 SVC_VMW_2_Node1 0 0 090826122939 090826122939 1 00990140 2 vdisk no no 5 SVC_VMW_2_Node1 0 0 090826122510 090826122510 1 00990182 0 flash no no 5 SVC_VMW_2_Node1 0 0 090826122510 090826122510 1 00990189 0 flash no no 5 SVC_VMW_2_Node1 0 0 090826122509 090826122509 1 00990187 0 flash no no 5 SVC_VMW_2_Node1 0 0 090826122509 090826122509 1 00990184 2 vdisk no no 5 SVC_VMW_2_Node1 0 0 090826122509 090826122509 1 00990169 0 flash no no 6 n/a 5319 5319 090826122508 090826122508 1 00983001 0 flash no no 5 SVC_VMW_2_Node1 0 0 090826122337 090826122337 1 00990186 0 flash no no 5 SVC_VMW_2_Node1 0 0 090826122307 090826122307 1 00990186 0 flash no no 5 SVC_VMW_2_Node1 0 0 090826122305 090826122305 1 00990191 IBM_2145:SVC_VMW_2:admin>

First you would check the time stamp, which appears in YYMMDDHHmmss format, and match it up to the time that the SRA was running its various commands. IBM can help with the decoding of the error codes. And what happens when we finish the test Failover. Well obviously the FlashCopy VDisk is removed from the host and destroyed. IBM_2145:SVC_VMW_2:admin>svcinfo catauditlog 130 putty_pub_key_2 superuser 0 rmvdiskhostmap -host 0 2 131 putty_pub_key_2 superuser 0 stopfcmap -force 0 132 putty_pub_key_2 superuser 0 rmfcmap -force 0 133 putty_pub_key_2 superuser 0 rmvdisk -force 2 IBM_2145:SVC_VMW_2:admin>

090826123912

admin 10.21.68.52 svctask

090826123912

admin 10.21.68.52 svctask

090826123915

admin 10.21.68.52 svctask

090826123916

admin 10.21.68.52 svctask

Again you can see the exact commands that are run to achieve this.

Page 83 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

And the VDisks presented to the recovery site host is updated, and only my original replicated LUN (SRM_VDISK) is visible. IBM_2145:SVC_VMW_2:admin>svcinfo lshostvdiskmap id name SCSI_id wwpn vdisk_UID 0 f115_vmware 10 10000000C96DC38A 6005076801FA0007A000000000000004 IBM_2145:SVC_VMW_2:admin>

vdisk_id 0

vdisk_name SRM_VDISK

My replicated VDisk is no longer involved in any FlashCopy operations: IBM_2145:SVC_VMW_2:admin>svcinfo lsvdisk SRM_VDISK id 0 name SRM_VDISK IO_group_id 0 IO_group_name io_grp0 status online mdisk_grp_id 0 mdisk_grp_name MDISKGRP_SRM capacity 5.00GB type striped formatted yes mdisk_id mdisk_name FC_id FC_name RC_id 0 RC_name MM_for_SRM vdisk_UID 6005076801FA0007A000000000000004 throttling 0 preferred_node_id 2 fast_write_state empty cache readwrite udid 0 fc_map_count 0 sync_rate 50 copy_count 1 copy_id 0 status online sync yes primary yes mdisk_grp_id 0 mdisk_grp_name MDISKGRP_SRM type striped mdisk_id mdisk_name fast_write_state empty used_capacity 5.00GB real_capacity 5.00GB free_capacity 0.00MB overallocation 100 autoexpand warning grainsize IBM_2145:SVC_VMW_2:admin>

And in fact there are no FlashCopy operations happening on the cluster: IBM_2145:SVC_VMW_2:admin>svcinfo lsfcmap IBM_2145:SVC_VMW_2:admin>

Hopefully this will give you some insight as to what is happening on the array during a test failover, and what to look for if something goes wrong.

Page 84 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

Trademarks © 2009 VMware, Inc. All rights reserved. Protected by one or more U.S. Patent Nos. 6,397,242, 6,496,847, 6,704,925, 6,711,672, 6,725,289, 6,735,601, 6,785,886, 6,789,156, 6,795,966, 6,880,022, 6,944,699, 6,961,806, 6,961,941, 7,069,413, 7,082,598, 7,089,377, 7,111,086, 7,111,145, 7,117,481, 7,149,843, 7,155,558, 7,222,221, 7,260,815, 7,260,820, 7,269,683, 7,275,136, 7,277,998, 7,277,999, 7,278,030, 7,281,102, 7,290,253, 7,356,679, 7,409,487, 7,412,492, 7,412,702, 7,424,710, 7,428,636, 7,433,951, 7,434,002, and 7,447,854; patents pending. VMware, the VMware “boxes” logo and design, Virtual SMP, and VMotion are registered trademarks or trademarks of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies. IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at ″Copyright and trademark information″ at www.ibm.com/legal/copytrade.shtml.

Summary As stated in the introduction, the point of this proven practice is to take the reader through the steps of deploying replication between some minimal SVC clusters, and then configure VMware vCenter Site Recovery Manager so that a test failover can be initiated. The main purpose of this document is to assist in setting up such a configuration for demo and training purposes, or Proof-Of-Concept purposes. While it may be of some assistance with deploying a production environment, further reading is essential to determine best practices. There are a number of people I wish to thank for their assistance with getting this guide written: • • • • •

Conor Dolan Paudie O’Riordan Geoff Lane Troy Lee Micheal White

(VMware) (VMware) (IBM) (IBM) (VMware)

Page 85 of 86

Proven Practice: IBM® SVC and VMware® SRM Setup Guide

About the Author Cormac Hogan is a Staff Technical Course Developer for VMware Global Support Services (GSS) based out of Cork, Ireland. He develops and delivers training for the GSS Technical Support Engineers within VMware, primarily on storage and storage related topics.

Page 86 of 86