Linux Client DM-Multipath Best Practice for CDP-NSS

Linux DM-Multipath Configuration for CDP/NSS  Best Practice Guide

FalconStor Software, Inc. 2 Huntington Quadrangle, Suite 2S01 Melville, NY 11747 Phone: 631-777-5188 Fax: 631-501-7633 Website: www.falconstor.com

Copyright © 2011 FalconStor Software. All Rights Reserved. FalconStor Software, FalconStor, and IPStor are registered trademarks of FalconStor Software, Inc. in the United States and other countries. Windows is a registered trademark of Microsoft Corporation. All other brand and product names are trademarks or registered trademarks of their respective owners. FalconStor Software reserves the right to make changes in the information contained in this publication without prior notice. The reader should in all cases consult FalconStor to determine whether any such changes have been made. 6.17.2011

Contents  Introduction ............................................................................................... 1 Check DM-Multipath package ................................................................................................. 1

Configure DM-Multipath for FALCON Disks ........................................... 3 Create the DM-Multipath configuration file for FALCON disks ............................................. 3 Run the script ...................................................................................................................... 3 Add parameters for a storage device ..................................................................................... 9 Reload multipath.conf ......................................................................................................... 9

Add FALCON Disks to the DM-Multipath Configuration ...................... 10 1. 2. 3. 4. 5.

Configure devices ............................................................................................................10 Restart the multipath daemon .........................................................................................11 Reload the configuration file ...........................................................................................11 Check configuration ........................................................................................................11 Start the DM-Multipath daemon at boot time .................................................................12

Configure HBAs ...................................................................................... 13 Tune HBAs for fast path switch .............................................................................................13 Tune HBAs for a single-path host..........................................................................................13

Linux DM-Multipath Configuration for CDP/NSS 

i 

Introduction  Linux Device Mapper Multipathing (DM-Multipath) is a native path-failover solution available with Red Hat Enterprise Linux 4.0 U2, SUSE SLES 9.0 PS2, and above. DM-Multipath makes it possible to configure multiple I/O paths between the storage server and storage in order to provide transparent failover and failback. DM-Multipath supports path redundancy and throughput aggregation. This document describes best practices for configuring Linux DM-Multipath to manage multipathing for storage provisioned by CDP/ NSS. Two options are included: 

If DM-Multipath is being configured for use with FALCON IPStor Disks, follow instructions in Configure DM-Multipath for FALCON Disks to run the configuration script. You can add disks other than FALCON disks at a later time.



If DM-Multipath is being configured/has been configured for disks other than  FALCON IPStor Disks, follow instructions in Add FALCON Disks to the DM-Multipath Configuration to include FALCON IPStor Disks in the devices section of the configuration file.

For more information on the DM-Multipath configuration file, refer to documentation for your operating system.

Check DM-Multipath package DM-Multipath includes packages as shown below. Make sure you have the latest version of each package and update if necessary. 





Red Hat Enterprise Linux 

device-mapper



device-mapper-multipath

SUSE Linux Enterprise Server 

device-mapper



multipath-tools

Debian Linux Enterprise Server (Ubuntu) 

multipath-tools

OPERATING SYSTEM DOCUMENTATION 

Red Hat Enterprise Linux 4 DM Multipath http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/4.8/pdf/DM_Multipath.pdf



Red Hat Enterprise Linux 5.1 DM Multipath http://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/enUS/RHEL510/pdf/DM_Multipath.pdf



How to set up/use multipathing on SUSE Linux Enterprise Server 9 http://support.novell.com/techcenter/sdb/en/2005/04/sles_multipathing.html Linux DM-Multipath Configuration for CDP/NSS 

1

Contents 

Storage Administration Guide - SUSE Linux Enterprise Server 10 http://www.novell.com/documentation/sles10/pdfdoc/stor_evms/stor_evms.pdf



Storage Administration Guide - SUSE Linux Enterprise Server 11 http://www.novell.com/documentation/sles11/pdfdoc/stor_admin/stor_admin.pdf

ADDITIONAL REFERENCES 

Using Device-Mapper Multipath http://www.redhat.com/docs/manuals/csgfs/browse/4.6/DM_Multipath/index.html



Online Storage Guide for Red Hat Enterprise Linux 5 http://www.redhat.com/docs/enUS/Red_Hat_Enterprise_Linux/html/Online_Storage_Reconfiguration_Guide/index.html

Linux DM-Multipath Configuration for CDP/NSS 

2 

Configure DM-Multipath  for FALCON Disks  Perform the following operations on any Linux client machine that is using DM-Multipath.

Create the DM-Multipath configuration file for FALCON disks The script updates the configuration file and restarts the daemon to initiate a new configuration. It accepts an input parameter to identify disks for which you do not want multipathing enabled, such as a local SCSI drive such as /dev/sda . In addition, the script includes default values and a multipath section for storage devices; you can replace these defaults in the Devices  section of /etc/multipath.conf and also remove comments.

Run the script  DM-Multipath also supports group paths based on Asymmetric Logical Unit Access (ALUA) state. To run the script without ALUA, enter the following: sh multipath.sh nonalua “^sda[0-9] ^sdb[0-9]”

To run the script with ALUA support, enter the following: sh multipath.sh alua “^sda[0-9] ^sdb[0-9]”

This section of the command represents the devices to be blacklisted: ^sda[0-9] or ^sdb[0-9]

USE THIS SCRIPT: #!/bin/sh configure() { touch multipath.conf # defaults for user_friendly_names echo "defaults {" > multipath.conf echo " user_friendly_names yes" >> multipath.conf echo "}" >> multipath.conf echo >> multipath.conf echo >> multipath.conf echo >> multipath.conf

Linux DM-Multipath Configuration for CDP/NSS 

3 

Configure DM-Multipath for FALCON Disks # blacklisting devices echo "blacklist {" >> multipath.conf for i in $* do if [ $i != alua -a $i != nonalua ] then echo " devnode \"$i\"" >> multipath.conf fi done echo "}" >> multipath.conf echo >> multipath.conf echo >> multipath.conf echo >> multipath.conf # uncomment the defaults section to override default values echo "#defaults {" >> multipath.conf echo "# udev_dir echo "# polling_interval echo "# selector multipath.conf echo "# path_grouping_policy echo "# path_checker echo "# rr_min_io echo "# max_fds echo "# rr_weight echo "# failback echo "# no_path_retry echo "# user_friendly_names echo "# features multipath.conf echo "#}" >> multipath.conf

/dev" >> multipath.conf 10" >> multipath.conf \"round-robin 0\"" >> multibus" >> multipath.conf tur" >> multipath.conf 100" >> multipath.conf 8192" >> multipath.conf priorities" >> multipath.conf immediate" >> multipath.conf queue" >> multipath.conf yes" >> multipath.conf \"1 queue_if_no_path\"" >>

echo >> multipath.conf echo >> multipath.conf echo >> multipath.conf # multipaths section to override devices section echo "#multipaths {" >> multipath.conf echo "# multipath {" >> multipath.conf echo "# wwid 3600508b4000156d70001200000b0000" >> multipath.conf echo "# alias yellow" >> multipath.conf echo "# path_grouping_policy multibus" >> multipath.conf echo "# path_checker readsector0" >> multipath.conf echo "# path_selector \"round-robin 0\"" >> multipath.conf echo "# failback manual" >> multipath.conf echo "# rr_weight priorities" >> multipath.conf echo "# no_path_retry 5" >> multipath.conf

Linux DM-Multipath Configuration for CDP/NSS 

4 

Configure DM-Multipath for FALCON Disks echo "# }" >> multipath.conf echo "# multipath {" >> multipath.conf echo "# wwid 1DEC_____321816758474" >> multipath.conf echo "# alias red" >> multipath.conf echo "# rr_weight priorities" >> multipath.conf echo "# }" >> multipath.conf echo "#}" >> multipath.conf echo >> multipath.conf echo >> multipath.conf echo >> multipath.conf # values per storage device override the defaults section echo "devices {" >> multipath.conf echo " device {" >> multipath.conf echo " vendor echo " product multipath.conf echo " path_selector multipath.conf echo " path_grouping_policy

\"FALCON\"" >> multipath.conf \"IPSTOR DISK\"" >> \"round-robin 0\"" >> multibus" >> multipath.conf

if [ -f /etc/redhat-release ] then if [ `cat /etc/redhat-release | awk '{print $1}'` = "Red" ] then if [ `cat /etc/redhat-release | awk '{print $7}' | awk -F"." '{print $1}'` -eq 6 ] then if [ $1 = alua ] then echo " prio alua" >> multipath.conf else echo " prio const" >> multipath.conf fi else if [ $1 = nonalua ] then echo " prio_callout \"/bin/true\"" >> multipath.conf else echo " prio_callout \"/sbin/mpath_prio_alua %d\"" >> multipath.conf fi fi else if [ $1 = nonalua ] then echo " \"/bin/true\"" >> multipath.conf else

prio_callout

Linux DM-Multipath Configuration for CDP/NSS 

5 

Configure DM-Multipath for FALCON Disks echo " prio_callout \"/sbin/mpath_prio_alua %d\"" >> multipath.conf fi fi fi if [ -f /etc/SuSE-release ] then if [ `cat /etc/SuSE-release | awk '{print $3}' | grep 11` -eq 11 ] then if [ $1 = alua ] then echo "

prio

alua"

prio

const"

>> multipath.conf else echo " >> multipath.conf fi else if [ $1 = nonalua ] then echo " prio_callout \"/bin/true\"" >> multipath.conf else echo " prio_callout \"/sbin/mpath_prio_alua %d\"" >> multipath.conf fi fi fi if [ -f /etc/lsb-release ] then if [ `cat /etc/lsb-release | grep -c Ubuntu` -ge 1 ] then if [ $1 = nonalua ] then echo " prio_callout \"/bin/true\"" >> multipath.conf else echo " prio_callout \"/sbin/mpath_prio_alua %d\"" >> multipath.conf fi fi fi

echo " echo " echo " echo " echo " multipath.conf echo "

path_checker rr_min_io rr_weight failback features

tur" >> multipath.conf 100" >> multipath.conf priorities" >> multipath.conf immediate" >> multipath.conf \"1 queue_if_no_path\"" >>

}" >> multipath.conf

Linux DM-Multipath Configuration for CDP/NSS 

6 

Configure DM-Multipath for FALCON Disks echo >> multipath.conf echo >> multipath.conf echo >> multipath.conf device return }

# restart the multipathd service

restart() { if [ -f /etc/redhat-release ] then if [ `cat /etc/redhat-release | awk '{print $1}'` = "Red" ] then if [ `cat /etc/redhat-release | awk '{print $7}' | awk F"." '{print $1}'` -eq 6 ] then mpathconf --enable fi fi service multipathd restart chkconfig multipathd on echo "=======================================================================" echo "

Reloading dm-multipath"

echo "=======================================================================" multipath -F multipath -r multipath -ll elif [ -f /etc/SuSE-release ] then service multipathd restart chkconfig multipathd on echo "=======================================================================" echo " Reloading dm-multipath" echo "=======================================================================" multipath -F multipath -r multipath -ll else

Linux DM-Multipath Configuration for CDP/NSS 

7 

Configure DM-Multipath for FALCON Disks sudo multipathd restart echo "=======================================================================" echo " Reloading dm-multipath" echo "=======================================================================" sudo multipath -F sudo multipath sudo multipath -ll fi return } device() { echo "}" >> multipath.conf if [ -f /etc/lsb-release ] then sudo mv multipath.conf /etc/. else mv multipath.conf /etc/. fi restart } usage() { echo echo echo exit

"multipath.sh alua|nonalua \"blacklist devices\"" "example:" "multipath.sh alua \"^sda[0-9]\"" 0

} if [ $# -lt 2 ]; then usage fi if [ $# -eq 2 ]; then if [ $1 != alua -a $1 != nonalua ]; then usage fi fi if [ $# -gt 2 ]; then if [ $1 != alua -a $1 != nonalua ]; then usage fi fi if [ -f "/etc/multipath.conf" ] then if [ -f /etc/lsb-release ] then sudo mv /etc/multipath.conf /etc/multipath.conf.old else mv /etc/multipath.conf /etc/multipath.conf.old fi

Linux DM-Multipath Configuration for CDP/NSS 

8 

Configure DM-Multipath for FALCON Disks fi configure $1 $2 exit 0

Add parameters for a storage device After you run the script, you can edit the devices portion of /etc/multipath.conf to add device parameters for storage devices other than FALCON IPSTOR DISK.

Reload multipath.conf  To ensure the modified configuration file is read and adopted after modifying device parameters, reload multipath.conf by entering the following commands: multipath –F (Flush the impending IO and remove devices from the device-mapper.) multipath –r (“multipath” in Ubuntu) ( Reload the configuration file to apply new

changes.) Note: There may be some outstanding IOs on multipath devices; if “multipath -F” fails, restart the multipath daemon and then reload the configuration file: service multipathd restart (e.g., RHES and SLES)

To list multipath devices, enter the following command: multipath –ll (Display the long list of multipath devices.)

Note: When the user_friendly_names option in the multipath configuration file is set to yes, the name of a multipath device is in the form mpathn. For versions of Red Hat Enterprise Linux earlier than version 6, n is an integer. For Red Hat Enterprise Linux 6, n is an alphabetic character; therefore, the name of a multipath device might be mpatha or mpathb. For more information, refer to http://docs.redhat.com/docs/enUS/Red_Hat_Enterprise_Linux/6/html/DM_Multipath/MPIO_Overview.html .

Linux DM-Multipath Configuration for CDP/NSS 

9 

Add FALCON Disks to  the DM-Multipath  Configuration  You can override the default configuration values for DM-Multipath by editing /etc/multipath.conf to suit your environment. Make a backup of the original file before making any changes.

1. Configure devices When DM-Multipath is being used to support storage devices other than IPSTOR DISK, you must define parameters for IPSTOR DISK in the “ devices” section of the configuration file. Add the section similar to the one shown below to include CDP/NSS-managed disks when generating multipath devices. In addition, you can set the appropriate parameters for a failover environment. There may be slight differences for other operating systems, with and without ALUA. This example shows the “ device” section of the /etc/multipath.conf file including with ALUA for Redhat 5.6 and below: devices { device { vendor

"FALCON"

product

"IPSTOR DISK"

getuid_callout

"/sbin/scsi_id -g -u -s /block/%n"

prio_callout

"/sbin/mpath_prio_alua %d"

features

"1 queue_if_no_path"

hardware_handler

"0"

path_selector

"round-robin 0"

path_grouping_policy

group_by_prio

failback

immediate

rr_weight

prioritiesuniform

rr_min_io

100

path_checker

tur

} }

CDP/NSS v6.15 and above supports asymmetric logical unit access (ALUA). To enable ALUA, set the priority callout program or the priority option as shown below. If you are using the iSCSI protocol, set the path_grouping_policy to multibus for Red Hat Enterprise Linux 4 update 7 and earlier. Linux DM-Multipath Configuration for CDP/NSS 

10 

Add FALCON Disks to the DM-Multipath Configuration The following parameter “priority callout (prio_callout)” must be changed to support specific operating system versions and ALUA: With ALUA: 

SLES 10 SP2 and later  – prio “alua”



RED HAT 6 and above  – prio “alua”



Other - prio_callout “/sbin/mpath_prio_alua %d”

Without ALUA: 

SLES 10 SP2 and later  – prio const



RED HAT 6 and above  – prio const



Other - prio_callout /bin/true

2. Restart the multipath daemon Whenever you modify the configuration file in any way, you must enter this command to restart the multipath daemon when you are done: /etc/init.d/multipathd restart

3. Reload the configuration file multipath –F (Flush all inpending IOs) multipath –r (“multipath” in Ubuntu) (Reload the new configuration)

4. Check configuration Enter this command to display current multipathing status: multipath -ll mpath1 (36000d7780000e5dc65d6877cf8618d14) dm-0 FALCON,IPSTOR DISK [size=9.8G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=200][active] \_ 3:0:0:0 sdb 8:16

[active][ready]

\_ 4:0:0:0 sdc 8:32

[active][ready]

\_ 3:0:1:0 sdd 8:48

[active][ready]

\_ 4:0:1:0 sde 8:64

[active][ready]

Some useful commands are listed below: 

multipath –F  – Flush impending IO and remove devices from the device-mapper.



multipath –r – Reload the configuration file to apply new changes.



multipath-l - List multipath devices.

Linux DM-Multipath Configuration for CDP/NSS 

11

Add FALCON Disks to the DM-Multipath Configuration 

multipath –ll - Display the long list of multipath devices.



dmsetup – Configure DM devices.



pvcreate  – Initialize physical volumes for use by LVM.

5. Start the DM-Multipath daemon at boot time Use the chkconfig command to configure the run level of the DM-Multipath daemon service. 



Red Hat Enterprise Linux 

chkconfig add multipathd



chkconfig multipathd on

SUSE Linux Enterprise Server 

chkconfig add boot.multipath



chkconfig add multipathd



chkconfig boot.multipath on



chkconfig multipathd on

Linux DM-Multipath Configuration for CDP/NSS 

12 

Configure HBAs  Perform the following operations on any client machine that is using DM-Multipath.

Tune HBAs for fast path switch For the multipath environment, path failure must be detected as quickly as possible in order for I/O to be redistributed to a healthy path. It is recommended that you configure the time-out setting to disable failover at the HBA level, which allows the failure of the HBA or cable to be propagated to the multipath layer. For example, setting driver parameters of a QLogic qla2xxx HBA in /etc/modprobe.conf or /etc/modprobe.conf.local is recommended. You may need to recreate the INITRD on the system using the mkinitrd command to reflect the change. options qla2xxx qlport_down_retry=1

Tune HBAs for a single-path host For the single-path environment, path failure should be delayed as long as possible in order for I/O to be resumed by the underlying storage subsystem. It is recommended that you configure the time-out setting to prolong the HBA retry period. For example, setting driver parameters of a QLogic qla2xxx HBA in /etc/modprobe.conf or /etc/modprobe.conf.local is recommended. You may need to recreate the INITRD on the system using the mkinitrd command to reflect the change. options qla2xxx qlport_down_retry=255

For an Emulex HBA, the following is recommended: options lpfc_nodev_tmo=60

Linux DM-Multipath Configuration for CDP/NSS 

13