VCP6-DCV STUDY GUIDE [UNOFFICIAL]
By Vladan SEGET www.vladan.fr
1
Running out of Capacity Again?
It’s Time to Hyperconverge 90% Capacity Savings – Guaranteed.
E
HY
P
ER
MPL I VI TY
TE
SI
GUARAN
SimpliVity HyperGuarantee The Industry’s Most Complete Guarante
www.simplivity.com/vcp6
Contents VCP6-DCV Objective 1.1 – Configure and Administer Role-based Access Control .................................................................... 3 VCP6-DCV Objective 1.2 – Secure ESXi, vCenter Server, and vSphere Virtual Machines .......................................................... 9 VCP6-DCV Objective 1.3 - Enable SSO and Active Directory Integration.............................................................................. 17 VCP6-DCV Objective 2.1 - Configure Advanced Policies/Features and Verify Network Virtualization Implementation ................. 26 VCP6-DCV Objective 2.2 - Configure Network I/O Control (NIOC) ...................................................................................... 41 VCP6-DCV Objective 2.3 – Configure vSS and vDS Policies ............................................................................................... 45 VCP6-DCV Objective 3.1 - Manage vSphere Storage Virtualization ..................................................................................... 52 VCP6-DCV Objective 3.2 - Configure Software-defined Storage ......................................................................................... 65 VCP6-DCV Objective 3.3 - Configure vSphere Storage Multi-pathing and Failover ................................................................ 76 VCP6-DCV Objective 3.4 - Perform Advanced VMFS and NFS Configurations and Upgrades ................................................... 83 VCP6-DCV Objective 3.5 - Setup and Configure Storage I/O Control .................................................................................. 93 VCP6-DCV Objective 4.1 - Perform ESXi Host and Virtual Machine Upgrades ....................................................................... 96 VCP6-DCV Objective 4.2 - Perform vCenter Server Upgrade ............................................................................................ 100 VCP6-DCV Objective 5.1 - Configure Advanced/Multilevel Resource Pools ......................................................................... 108 VCP6-DCV Objective 6.1 - Configure and Administer a vSphere Backups/Restore/Replication Solution .................................. 116 VCP6-DCV Objective 7.1 - Troubleshoot vCenter Server, ESXi Hosts, and Virtual Machines ................................................. 132 VCP6-DCV Objective 7.2 - Troubleshoot vSphere Storage and Network Issues................................................................... 139 VCP6-DCV Objective 7.3 - Troubleshoot vSphere Upgrades ............................................................................................. 144 VCP6-DCV Objective 7.4 - Troubleshoot and Monitor vSphere Performance ....................................................................... 149 VCP6-DCV Objective 7.5 - Troubleshoot HA and DRS Configurations and Fault Tolerance .................................................... 156 VCP6-DCV Objective 8.1 - Deploy ESXi Hosts Using Autodeploy ....................................................................................... 166 VCP6-DCV Objective 8.2 - Customize Host Profile Settings .............................................................................................. 172 VCP6-DCV Objective 8.3 - Consolidate Physical Workloads using VMware Converter ........................................................... 177 VCP6-DCV Objective 9.1 - Configure Advanced vSphere HA Features ............................................................................... 181 VCP6-DCV Objective 9.2 - Configure Advanced vSphere DRS Features ............................................................................. 189 VCP6-DCV Objective 10.1 - Configure Advanced vSphere Virtual Machine Settings............................................................. 192 VCP6-DCV Objective 10.2 - Create and Manage Multi-Site Content Library ........................................................................ 200 VCP6-DCV Objective 10.3 - Configure and Maintain a vCloud Air Connection ..................................................................... 205
2
VCP6-DCV OBJECTIVE 1.1 – CONFIGURE AND ADMINISTER ROLE -BASED ACCESS CONTROL Today's VCP6-DCV goal is to talk about - VCP6-DCV Objective 1.1 - Configure and Administer Role-based Access Control. VMware VCP exam is a gold standard of VMware certification exams. VCP exam is the most known VMware exams, even if it's not the highest technical level. But it's most recognized. By a future employer, by industry as a whole. We will cover VCP6-DCV exam certification based on VMware latest VMware VCP6-DCV blueprint. Check VCP6-DCV page for all objectives.
VMware vSphere Knowledge
Identify common vCenter Server privileges and roles Describe how permissions are applied and inherited in vCenter Server View/Sort/Export user and group lists Add/Modify/Remove permissions for users and groups on vCenter Server inventory objects Create/Clone/Edit vCenter Server Roles Determine the correct roles/privileges needed to integrate vCenter Server with other VMware products Determine the appropriate set of privileges for common tasks in vCenter Server
IDENTIFY
COMMON V C ENTER
S ERVER
PRIVILEGES AND ROLES
There are roles and privileges. Role is a collection of privileges assigned to group or a user. There are certain number of Out-of-the-box (predefined) roles when we look at the vSphere client > Roles. You can keep them, clone them, delete or edit.
3
Four different types of permissions Not only vCenter server, like the ones above, but also Local permissions for ESXi. The full list:
Global Permissions – Global permissions are applied to a global root object that spans solutions. To assign permissions via global root allows to propagate them to the other products relying on SSO (vCO, vROPS, vCD..) vCenter Server Permissions – Hierarchical model. Permission gives you a certain number of privileges. Similar like in Microft's AD. You Select object > assign role to a group of users > to give them privileges on that object. Group Membership in vSphere.local Groups – The vsphere.local domain includes several predefined groups. Assign users from AD (if you're using AD) to one of those groups to be able to perform the corresponding actions. For some services that are not managed by vCenter Server directly, privileges are determined by membership to one of the vCenter Single Sign-On groups. For example, a user who is a member of the Administrator group can manage vCenter Single Sign-On. A user who is a member of the CAAdmins group can manage the VMware Certificate Authority, and a user who is in the LicenseService.Administrators group can manage licenses.
Note: to be able to find the AD groups it's necessary to add Identity sources via: Home > Administration > Single Sign-ON > Configuration > Identity sources.
The user
[email protected] can perform tasks that are associated with services included with the Platform Services Controller.
4
ESXi Local Host Permissions – If you are managing a standalone ESXi host that is not managed by a vCenter Server system, you can assign one of the predefined roles to users.
DESCRIBE HOW PERMISSIONS ARE APPLIED AND INHERITED IN VC ENTER SERVER The global permissions are assigned via web client only (SSO), via Home > Administration > Global permissions. If you deselect the propagate to children the objects lying down the road won't be accessible by that particular user/group. (It's like when you manage NTFS permissions on Windows servers and you uncheck the heritage check box). Permissions are applicable directly and propagated to children by default.
If you click the "View Children" link, it'll show you the permission of all the children which permission will apply to (if "Propagate to children is selected).
Inheritance of Multiple Permissions - If user is member of more than one group? Then combined privileges within the roles apply. Example below showing user member of both groups.
Child permissions override Parent permissions - Permissions applied on a child object always override permissions that are applied on a parent object. See examples P. 119 of vSphere Security Guide.
5
Ex. Role 1 can power on VMs and Role 2 can take snapshots. Group A is granted Role 1 on VM folder and permissions propagate to child objects Group B is granted Role 2 on VM B User 1, who belongs to groups A and B, logs on. Because Role 2 is assigned at a lower point in the hierarchy than Role 1, it overrides Role 1 on VM B. User 1 can power on VM A, but not take snapshots. User 1 can take snapshots of VM B, but not power it on.
User role overriding group role - if two permissions are defined on the same object.
Permissions are on the same object. One permission is granted to a group, the other to a user which at the same time is member of the group. Role 1 can power VMs Group A is granted Role 1 on VM folder and at the same time User 1 is granted No Access role on VM folder. User 1, who belongs to group A, logs on. The No Access role granted to User 1 on VM Folder overrides the role assigned to the group. User 1 has no access to VM Folder or VMs A and B.
V IEW /S ORT /E XPORT
USER AND GROUP LISTS
To check Global permissions you have to go and use Web client > Home > Administration > Global permissions. You can be export to a CSV file or copy to the Clipboard selected or All items. You can also use CTRL+Click to copy to the clipboard.
6
A DD /M ODIFY /R EMOVE
PERMISSIONS FOR USERS AND GROUPS ON V C ENTER
S ERVER
INVENTORY
OBJECTS
To modify/add permissions you must Select an object > Manage > Permissions. Than you can use the delete, edit or Add icons there...
CREATE/CLONE/E DIT V CENTER S ERVER R OLES To edit, create or clone vCenter roles it's necessary to use vSphere Web client > Administration > Roles OR Home > Roles. Default roles are:
Administrator Read-Only No Access
To clone role click the icon...
7
vSphere Security Guide (p. 121). DETERMINE THE CORRECT ROLES / PRIVILEGES OTHER VM WARE PRODUCTS
NEEDED TO INTEGRATE V C ENTER
S ERVER
WITH
Global permissions are applied to a global root object that spans solutions, for example, both vCenter Server and vCenter Orchestrator. Use global permissions to give a user or group privileges for all objects in all object hierarchies. Global permissions are applied to a global root object that spans solutions, for example, both vCenter Server and vCenter Orchestrator. Use global permissions to give a user or group privileges for all objects in all object hierarchies. P. 122
DETERMINE
THE APPROPRIATE SET OF PRIVILEGES FOR COMMON TASKS IN V C ENTER
Common tasks Required Privileges - p.127 All privileges - p.229
Tools:
vSphere Installation and Setup Guide vSphere Security Guide What’s New in the VMware vSphere® 6.0 Platform vSphere Administration with the vSphere Client Guide vSphere Client / vSphere Web Client
8
S ERVER
VCP6-DCV OBJECTIVE 1.2 – SECURE ESX I, VCENTER SERVER, AND VSPHERE VIRTUAL M ACHINES This post covers VCP6-DCV Objective 1.2 - Secure ESXi, vCenter Server, and vSphere Virtual Machines. A very interesting chapter indeed, where we cover all the "locks" which an admin can put in place to secure his/here environment. And you don't have to be Linux expert as all this is done without much difficulty! For whole exam coverage I created a dedicated VCP6-DCV page. Or if you're not preparing to pass a VCP6-DCV, you might just want to look on some how-to, news, videos about vSphere 6 - check out my vSphere 6 page. If you find out that I missed something, don't hesitate to comment.
Knowledge
Enable/Configure/Disable services in the ESXi firewall Enable Lockdown Mode Configure network security policies Add an ESXi Host to a directory service Apply permissions to ESXi Hosts using Host Profiles Configure virtual machine security policies Create/Manage vCenter Server Security Certificates
E NABLE/CONFIGURE/DISABLE
SERVICES IN THE
ESX I
FIREWALL
HOW TO ENABLE / DISABLE SERVICES IN THE ESX I FIREWALL - THE HARD WAY ( VIA CLI)
CHECK
WHIH SERVICES ARE ACTIVE
esxcli network firewall ruleset list
O PEN
FIREWALL PORT VIA
CLI:
esxcli network firewall ruleset set -e true -r httpClient
HOW TO ENABLE / DISABLE SERVICES IN THE ESX I FIREWALL - THE EASY WAY ( VIA VS PHERE CLIENT ) Note that you can do the same by selecting the host through vSphere client > configuration > security profile > Firewall
9
Services can be Started, Stopped, or Restarted. Services can be configured to Start and stop with host, Start and stop manually, or Start and stop with port usage. ESXi Shell and SSH are disabled (Set to Start and stop manually) by default. ESXi Shell and SSH can be enabled/disabled in the DCUI from the Troubleshooting Mode Options menu.
10
E NABLE LOCKDOWN M ODE When you enable lockdown mode, you can't connect directly from the console. the host is accessible only through the vSphere client directly or via vCenter server.
Lockdown Modes:
Disabled - Lockdown mode is disabled. Normal - Lockdown mode is enabled. The host can only be accessed from vCenter or from the console (DCUI). Strict - Lockdown mode is enabled. The DCUI service is stopped. The host can not be accessed from the console (DCUI).
[TIP]: You can activate DCUI from within SSH session Type this after login in with Putty or other SSH client. dcui There you see the DCUI screen
11
vSphere 6 introduced "Exception users" which are users with local accounts or Microsoft Active Directory accounts with permissions defined locally on the host where these users have host access. You can define those exception locally on the host, but it’s not recommended for normal user accounts, but rather for service accounts. You should set permissions on these accounts to strict minimum and only what’s required for the application to do its task and with an account that needs only read-only permissions to the ESXi host. This is basically the same principle of local server accounts on Windows member server, where you can create local accounts, but as a best practice to give them only the permissions they need… Smart Card Authentication to DCUI – There is new function, but apparently it is for U.S. federal customers only. It allows DCUI login access using a Common Access Card (CAC) and Personal Identity Verification (PIV). In this case the ESXi host must be part of Microsoft AD.
CONFIGURE
NETWORK SECURITY POLICIES
Network security policies are defined on two places:
vSwitch level Portgroup level
Three different policies:
Promiscuous mode – If set to Accept then it allows the guest OS to receive all traffic observed on the connected vSwitch or PortGroup (the switch becames a HUB basically - with all the inconveniences, packet colisions, performance degradation etc... ). By default it's Reject MAC address changes – A host is able to accepts requests to change the effective MAC address to a different address than the initial MAC address. By default it's Accept Forged transmits – A host does not compare source and effective MAC addresses transmitted from a virtual machine. By default it's Accept
Or via vSphere client (more convenient)
12
MAC address changes and Forged transmits if set to Reject, than it protects against MAC address spoofing. If changing the settings at the Portgroup level there is an Override checkbox allowing you to set the policy on a portgroup rather than on the vSwitch.
A DD
AN
ESX I H OST
TO A DIRECTORY SERVICE
Using Active Directory for user authentication simplifies the ESXi host configuration and reduces the risk for configuration issues that could lead to unauthorized access. You can join or leave domain by selecting a host > configuration > authentication services > properties. You can also join standalone ESXi hosts to AD. By using AD you eliminate to manage locally users on ESXi hosts.
A special AD group named "ESX Admins" shall be manually created before host is joined to AD. Why? Because like this All members of this group (ESX admins) are automatically assigned with the Administrator role on the host when this host is joined to AD. If not the permissions has to be applied manually.
13
vSphere web client > Hosts and clusters > Select ESXi host > Manage > Settings > Authentication services.
A PPLY
PERMISSIONS TO
ESX I H OSTS
USING
H OST P ROFILES
Host profiles are very cool feature allowing to homogenize configuration across ESXi hosts and automate compliance. In some cases, host profiles can be also useful when for example you need to reset esxi root password on a host. Check vSphere Security guide (PDF) on p. 133, but basically this procedure apply: 1. Set up the reference host to specification and create a host profile. 2. Attach the profile to a host or cluster. 3. Apply the host profile of the reference host to other hosts or clusters.
If you haven't done yet, go to Home > Host profiles > Extract profile from host. Once you have that profile you can apply it to a host...
Select the host profile > Click Actions > Edit Host Profile (or right click > edit settings) Expand Security and Services Select the Permission Rules folder > click the Plus Sign
14
Root password is encrypted within the host profile, however by joining hosts to AD via Host profiles leaves password in plain text... -:(. Configure virtual machine security policies VMs are fragile. The same for Guest OS. Treat them accordingly ... -:). Seriously, you should patch to the latest release for the OS patches, Antivirus patches and/or Malware patches.... That's a bare minimum to prevent system corruption.
Be organized - Use templates to deploy virtual machines Minimize use of virtual machine console 15
Prevent virtual machines from taking over resources Disable unnecessary functions inside virtual machines - usually Windows/Linux services can be stopped, to put them on manual instead of automatic startup, etc.. Remove unnecessary hardware devices - floppy, printers, sound devices... All you don't need you can remove to have lower overhead. Disable unused display features Disable unexposed features Disable HGFS file transfers Disable copy and past operations between guest operating system and remote console (by default is disabled - on per host level, but you can add an advanced settings:)
isolation.tools.copy.disable isolation.tools.paste.disable = true
=
true
Limiting exposure of sensitive data copied to the Clipboard Restrict users from running commands within a virtual machine
1. Click Administration and select Roles > click create role > NO Guest Access > select all privileges 2. Deselect All Privileges >Virtual machine > Guest Operations to remove the Guest Operations set of privileges > validate OK.
Prevent a virtual machine user or process from disconnecting devices Modify guest operating system variable memory limit Prevent guest operating system process from sending configuration messages to the host Avoid using Independent Nonpersistent Disks - keep in mind non persistent disks are not affected by snapshots. If you use snapshots. A redo log is created to capture all subsequent writes to that disk. However, if the snapshot is deleted, or the virtual machine is powered off, the changes captured in that redo log are discarded for that Independent Non-persistent VMDK.
CREATE/M ANAGE V C ENTER S ERVER S ECURITY CERTIFICATES Certificates got easier with vSphere 6 as those can be viewed and renewed within vSphere Web client.
There are two operations modes:
Root CA - (by default) Issuer CA – possibility integrate Microsoft Certification authority. In this case you’ll create the CSR (request) > Go to Microsoft Cert Server and get certificate.
To view certificates:
16
The VMware Certificate Authority (VMCA) provisions vCenter Server components and ESXi hosts with certificates that use VMCA as the root certificate authority by default. The vSphere Certificate Manager utility allows you to perform most certificate management tasks interactively from the command line. Example. On Windows you must go to this directory: C:\Program Files\VMware\vCenter Server\vmcad\certificate-manager.bat Link to Online documentation for using vSphere Certificate manager utility. vSphere Certificate Manager prompts you for the task to perform, for certificate locations and other information as needed, and then stops and starts services and replaces certificates for you. vCenter Certificate Utilities:
vSphere Certificate Manager utility – certificate replacement tasks from a command line utility. Certificate management CLIs – dir-cli, certool, and vecs-cli command line utilities. o certool can Generate and manage certificates and keys. Part of VMCA. o dir-cli is a able to create and update certificates in VMware Directory Service. Part of VMAFD. o ves-cli can manage the contents of VMware Certificate Store instances. Part of VMAFD
vSphere Web Client certificate management – view certificate information in the Web Client
Tools
vSphere Installation and Setup Guide vSphere Security Guide What’s New in the VMware vSphere® 6.0 Platform Security of the VMware vSphere® Hypervisor vSphere Administration with the vSphere Client Guide VMware Hardened Virtual Appliance Operations Guide added to Tech Resource Directory vSphere Client / vSphere Web Client
VCP6-DCV OBJECTIVE 1.3 - ENABLE SSO AND ACTIVE DIRECTORY INTEGRATION In no particular order I'll start covering VCP6-DCV sections to help out folks learning towards VCP6-DCV VMware certification exam. Due to VMware recertification policy the VCP exam has now an expiration date. You can renew by passing delta exam while still holding current VCP or pass VCAP. The topic today - VCP6-DCV Objective 1.3 - Enable SSO and Active Directory Integration. For whole exam coverage I created a dedicated VCP6-DCV Wordpress page. If you just look on some how-to, news, videos about vSphere 6 check out my vSphere 6 page. vSphere 6 grew up quite big compared to vSphere 5.5 release, but simplified the deployment and management. vSphere Web client is more present and used in this release as the legacy C# client does not allow to configure advanced configuration options and functions like SSO, FT, VSAN
You'll need certain knowledge that we'll try to cover today:
Configure/Manage Active Directory Authentication Configure/Manage Platform Services Controller (PSC) Configure/Manage VMware Certificate Authority (VMCA) Enable/Disable Single Sign-On (SSO) Users 17
Identify available authentication methods with VMware vCenter
CONFIGURE/M ANAGE A CTIVE DIRECTORY A UTHENTICATION Step 1: Connect to your vCenter server by entering the ip address you have entered during the deployment process: https://vCenter Server IP/vsphere-client and by using the
[email protected] as a user name and your password you have used during the deployment.
Step 2: Click the Administration button on the left and
And then go to Single Sign-On > Configuration > Identity Sources > Click the "+" sign to add your AD as an identity source. Normally it will populate your local AD automatically, so you just have to click the OK button... 18
You can also click the globe icon to make the AD as the default while you're there... Screenshot showing the Identity source where we added our AD - lab.local
N EXT S TEP : P ERMISSIONS You'll need to assign permissions to users which will administer the vSphere infrastructure. Usually it's domain admin, but not always..... Also keep in mind where you assign those permissions. If it's at the Datacenter level, vCenter level or at the cluster level... Usually you'll want to do it at the vCenter Level. Go to Home > vCenter Inventory Lists > vCenter Servers > vCenter.lab.local (in my case) > Click the Manage Tab > Permissions There you click the "+" sign > Add button > make sure that you select the drop-down for your Microsoft Ad to make appear the Domain admin user...
19
Click OK to validate. You can disconnect and connect as domain admin now... Note that in case your workstation is part of Microsoft AD, you just have to check the box and no need to enter your domain user password... -:)
Some of you might wonder why there is this Single Sign-On. The vCenter Single Sign On is an authentication service which allows the different vSphere software components present in the vCloud suite, to communicate between each other via a secure token exchange mechanism.
CONFIGURE/M ANAGE P LATFORM S ERVICES CONTROLLER (PSC) The Platform Services Controller (PSC) provides:
Single Sign-On (SSO) Licensing 20
Certificate Authority (VMCA)
You can deploy it on at the same time or a part and you can deploy it as Windows based or Appliance based (VCSA). It's important to know that PSO is completely transparent working with Windows or VCSA based vCenter! PSC Deployment Options - A two different type installation are allowed:
Embedded (in the same VM) External
The embedded PSC is meant to be used for standalone sites where vCenter server will be the only SSO integrated solution. In this case a replication to another PSC is not necessary. External PSC shall be deployed in anvironments where there is more then one SSO enabled solution (vCenter Server, vRealize Automation, etc…) OR where replication to another PSC (another site) is necessary. Here is the screenshot from the installation process (VCSA) showing the different options and changing the options also changes the different phases of the deployment (on the left).
PSC features:
Manages and generates SSL certificates for your vSphere environment. Stores and replicates VMware License Keys Stores and replicates permissions via the Global Permissions layer. Manages the storage and replication of TAGS and CATEGORIES. There is a Built-in automatic replication between different, logical SSO sites. (if any) There is only one single default domain for the identity sources.
D EPLOYMENT O PTIONS:
21
Embedded Platform Service Controller All services bundled with the Platform Services Controller are deployed on the same virtual machine or physical server as vCenter Server. External Platform Service Controller The services bundled with the Platform Services Controller and vCenter Server are deployed on different virtual machines or physical servers.
Recommended reads: VMware vSphere Blog - vCenter Server 6 Deployment Topologies and High Availability. VMware KB - Recommended topologies for vSphere 6.0.x (2108548). Configure/Manage VMware Certificate Authority (VMCA) When you first install vSphere, the default certificates are deployed with 10 years of life span. The VMCA generates those self-signed certs during the installation process, and provisions each of the ESXi host with a signed certificate by this root certificate authority. Earlier versions of vSphere with self-signed certificates are automatically replaced by new self-signed certificates by VMCA. There are different ESXi Certificate replacement modes:
Default - VMCA as cert authority where VMCA issues certs for your hosts. Custom - you can override and do and issue certs manually via VMCA Thumbprint mode - this way you keep certs from vSphere 5.5
To check this go to the View Support Information after logging to your ESXi host:
22
W HERE TO CHECK THE CERTIFICATES IN W EB CLIENT ?
Home -> System Configuration -> Nodes -> Node -> Manage -> Certificate Authority Note: If you're not a member of SystemConfiguration.Administrators group than you might want to add yourself there. If of course you're connecting as an domain administrator....
Back to where to check the certificates on vSphere Web Client: Home > System Configuration > Nodes > Node > Manage > Certificate Authority
E NABLE/DISABLE S INGLE S IGN-O N (SSO) USERS The VMware SSO uses different configuration policy which can be found via vSphere Web client only:
Administration > Single Sign-On > Configuration Policies 23
Password Policy Lockout Policy Token Policy
P ASSWORD P OLICY
You can configure the following parameters:
Description – Password policy description. Required. Maximum lifetime – Maximum number of days that a password can exist before it has to be changed. Restrict re-use – Number of the user’s previous passwords that cannot be set again. Maximum length – Maximum number of characters that are allowed in the password. Minimum length – Minimum number of characters required in the password. Character requirements – Minimum number of different character types required in the password. Identical adjacent characters – Maximum number of identical adjacent characters allowed in the password.
To get to this screen You must click Administration > Single Sign-On > Configuration
By clicking the Edit button you are able to change values there…
24
If you leave the default values and after 90 days you will want to log-in you might end up with messages saying that:
User Account is locked. User Account is disabled.
Those SSO policies are pretty much the same as in vSphere 5.5, but with a difference that in vSphere 5.5 we also had an administrator password expiry on the vCenter server appliance (VCSA). The VCSA 6.0 is pretty much locked out and the GUI we use to manage VCSA accessible via the port 5480 is no longer available. Lockout Policy Specifies the condition under which a vCenter SSO account is locked when the user attempts to log in with incorrect credentials. Five login attempts and three minutes between failures are set by default. This policy also specifies the time that must elapse before the account is automatically unlocked.
Description – Description of the lockout policy. Required. Max. number of failed login attempts – Maximum number of failed login attempts that are allowed before the account is locked. Time interval between failures (seconds) – Time period in which failed login attempts must occur to trigger a lockout. Unlock time (seconds) – Amount of time that the account remains locked. If you enter 0, the account must be explicitly unlocked by an administrator.
To see the lockout policy parameters, click on the Policies tab and select Lockout Policy:
Token Policy - also interesting as for example the Clock tolerance shows time difference, in milliseconds, that vCenter Single Sign-On tolerates between a client clock and the domain controller clock. If the time difference is greater than the specified value, vCenter Single Sign-On declares the token invalid.
25
Other configuration options:
Maximum token renewal count – Maximum number of times that a token can be renewed. After the maximum number of renewal attempts, a new security token is required. Maximum token delegation count – Holder-of-key tokens can be delegated to services in the vSphere environment. A service that uses a delegated token performs the service on behalf of the principal that provided the token. A token request specifies a DelegateTo identity. The DelegateTo value can either be a solution token or a reference to a solution token. This value specifies how many times a single holder-of-key token can be delegated. Maximum bearer token lifetime – Bearer tokens provide authentication based only on possession of the token. Bearer tokens are intended for short-term, single-operation use. A bearer token does not verify the identity of the user or entity that is sending the request. This value specifies the lifetime value of a bearer token before the token has to be reissued. Maximum holder-of-key token lifetime – Holder-of-key tokens provide authentication based on security artifacts that are embedded in the token. Holder-of-key tokens can be used for delegation. A client can obtain a holder-of-key token and delegate that token to another entity. The token contains the claims to identify the originator and the delegate. In the vSphere environment, a vCenter Server obtains delegated tokens on a user’s behalf and uses those tokens to perform operations. This value determines the lifetime of a holder-ofkey token before the token is marked invalid.
IDENTIFY
AVAILABLE AUTHENTICATION METHODS WITH
VM WARE V CENTER
We have already saw that at the beginning of the post. The possible identity sources can be found via web client > Administration > Single Sign-On > Configuration > Identity Sources And we can see that there are four of them:
AD integrated (preferred) Active Directory LDAP Open LDAP Local OS
Yep, you can obviously use Local OS option only if you don't want to interconnect with your AD (for security reasons or isolation purposes). Check How-to, news, videos and tutorials at my vSphere 6 page too or check Free VMware tools page. Tools to get the knowledge and further reading:
vSphere Installation and Setup Guide vSphere Security Guide What’s New in the VMware vSphere® 6.0 Platform VMware vCenter Server™ 6.0 Deployment Guide Direct Console User Interface (DCUI) vSphere Client / vSphere Web Client
VCP6-DCV OBJECTIVE 2.1 - CONFIGURE ADVANCED POLICIES/FEATURES AND VERIFY NETWORK VIRTUALIZATION IMPLEMENTATION Today's VCP6-DCV topic Objective 2.1: Configure Advanced Policies/Features and Verify Network Virtualization Implementation is the core of virtualization networking. Together with 2 other chapters it covers all vSphere 6 networking.
26
You can follow the VCP6-DCV study guide built through my VCP6-DCV page. When finished, there will be a PDF version which will get its proper formatting for better reading experience. We're more than half way through right now, and the work continues. Let's kick on with this chapter!
vSphere Knowledge
Identify vSphere Distributed Switch (vDS) capabilities Create/Delete a vSphere Distributed Switch Add/Remove ESXi hosts from a vSphere Distributed Switch Add/Configure/Remove dvPort groups Add/Remove uplink adapters to dvUplink groups Configure vSphere Distributed Switch general and dvPort group settings Create/Configure/Remove virtual adapters Migrate virtual machines to/from a vSphere Distributed Switch Configure LACP on Uplink portgroups Describe vDS Security Polices/Settings Configure dvPort group blocking policies Configure load balancing and failover policies Configure VLAN/PVLAN settings Configure traffic shaping policies Enable TCP Segmentation Offload support for a virtual machine Enable Jumbo Frames support on appropriate components Determine appropriate VLAN configuration for a vSphere implementation
IDENTIFY V S PHERE DISTRIBUTED S WITCH ( V DS)
CAPABILITIES
VMware vSphere Distributed Switch (vDS) is in its version 6 and packed in more feature than in previous relase of VDS. If you're upgrading you shall upgrade vDS to version 6.0 as well to benefit the latest features. The vDS separates the data plane and management plane to separate them. The data plane resides on ESXi host, but the management plane moves to vCenter server. The data plane is called host proxy switch.
NetFlow Support - Netflow is used for troubleshooting, it picks a configurable number of samples of network traffic for monitoring.. PVLAN Support - PVLAN is able to get more from VLANs (which are limited in numbers) and you can use these PVLANS to further segregate your traffic and increase security. (Note: Enterprise plus licensing required! Check my detailed post on PVLANs here. Ingress and egress traffic shaping - Inbound/outbound traffic shaping, which allows you throttle bandwidth to the switch. VM Port Blocking - can block VM ports in case of viruses or troubleshooting... Load Based Teaming - LBT is an additional load balancing that works off the amount of traffic a queue is sending Central Management across cluster - vDS can create the config once and push it to all attached hosts...so you don't have to go to each host one-by-one... Per Port Policy Settings - It's possible to override policies at a port level which gives you more controll Port State Monitoring - This feature allows each port to be monitored separately from other ports LLDP - Allows supports for link layer discovery protocol Network IO Control - possibility to set priority on port groups and reserve bandwidth for VMs connected to this port group. Check the detailed chapter on NIOC here: Objective 2.2: Configure Network I/O Control (NIOC) LACP Support - LACP (Link aggregation control protocol) ability to aggregate links together into a single link (your physical switch must support it!) Backup/Restore Network config - It's possible to backup/restore network config at the vDS level (Not new! It's here since 5.1! - save and restore network config...) Port Mirroring - Allows monitoring and can send all traffic from one port to another 27
Stats stays at the VM level - statistics move with the VM even after vMotion.
CREATE/DELETE
A V S PHERE
DISTRIBUTED S WITCH
Create a vSphere vDS - Networking Guide on p27. vSphere Web client > Networking > Rigt click datacenter > Distributed switch > New Distributed switch
Put a name and then select the version...
Select how many uplinks, specify if you want to enable Network I/O control and rename the default port group (not mandatory)...
28
A DD /R EMOVE ESX I
HOSTS FROM A VS PHERE
DISTRIBUTED S WITCH
You can add/remove ESXi hosts from vDS to manage their networking (or not) from a central location. The good thing is that you can analyse impact before breaking a connectivity, so you're able to see the impact. The impact can be as follows:
No Impact Important impact Critical Impact
Next...
A DD /CONFIGURE/R EMOVE
DV P ORT GROUPS
Right click on the vDS > New Distributed Port Group.
To remove a port group. Simple. Right click on the port group > delete... 29
A DD /R EMOVE
UPLINK ADAPTERS TO DV U PLINK GROUPS
Again, right click is your friend... -:) If you want to add/remove (increase or decrease) number of uplinks you can do so by going to the properties of the vDS.
Right click on the vDS > Edit settings
And on the next screen you can do that... Note that at the same time you can give a different names to your uplinks...
30
CONFIGURE V S PHERE DISTRIBUTED S WITCH
GENERAL AND DV P ORT GROUP SETTINGS
General properties of vDS can be reached via Right click on the vDS > Settings > Edit settings
Port binding properties (at the dvPortGroup level - Right click port group > Edit Settings)
Static binding - Assigns a port to a VM when the virtual machine is connected to the PortGroup. Dynamic binding - it's kind of deprecated. For best performance use static binding Ephemeral – no binding
Port allocation:
31
Elastic - Increase or decreas on-the-fly..... 8 at the beginning (default). Increases by 8 when needed. Fixed - There is 128 by default.
CREATE/CONFIGURE/R EMOVE
VIRTUAL ADAPTERS
VMkernel adapters can be add/removed at the Networking level vSphere Web Client > Host and Clusters > Select Host > Manage > Networking > VMkernel adapters
Different VMkernel Services, like :
vMotion traffic Provisioning traffic Fault Tolerance (FT) traffic Management traffic vSphere Replication traffic vSphere Replication NFC traffic VSAN traffic
M IGRATE
VIRTUAL MACHINES TO/ FROM A VS PHERE
DISTRIBUTED SWITCH
Migrate VMs to vDS. Right click vDS > Migrate VM to another network
Make sure that you previously created a distributed port group with the same VLAN that the current VM is running... (in my case the VMs run at VLAN 7)
32
Pick a VM...
Done!
CONFIGURE LACP
ON
UPLINK
PORTGROUPS
LACP can be found in the Networking guide on p.65.
vSphere Web Client > Networking > vDS > Manage > Settings > LACP Create Link Aggregation Groups (LAG)
33
LAG Mode can be:
Passive - where the LAG ports respond to LACP packets they receive but do not initiate LACP negotiations. Active - where LAG ports are in active mode and they initiate negotiations with LACP Port Channel.
LAG load balancing mode (LNB mode):
Source and destination IP address, TCP/UDP port and VLAN Source and destination IP address and VLAN Source and destination MAC address Source and destination TCP/UDP port Source port ID VLAN
Note that you must configure the LNB hashing same way on both virtual and physical switch, at the LACP port channel level. Migrate Network Traffic to Link Aggregation Groups (LAG)
34
DESCRIBE V DS S ECURITY P OLICES /S ETTINGS Note that those security policies exists also on standard switches. There are 3 different network security policies:
Promiscuous mode – Reject is by default. In case you set to Accept > the guest OS will receive all traffic observed on the connected vSwitch or PortGroup. MAC address changes – Reject is by default. In case you set to Accept > then the host will accepts requests to change the effective MAC address to a different address than the initial MAC address. Forged transmits – Reject is by default. In case you set to Accept > then the host does not compare source and effective MAC addresses transmitted from a virtual machine.
35
Network security policies can be set on each vDS PortGroup. Configure dvPort group blocking policies Port blocking can be enabled on a port group to block all ports on the port group
or you can configure the vDS or uplink to be blocked at the vDS level...
vSphere Web Client > Networking > vDS > Manage > Ports
And then select the port > edit settings > Miscellaneous > Override check box > set Block port to yes.
36
CONFIGURE
LOAD BALANCING AND FAILOVER POLICIES
Load balancing algos can be found in the Networking Guide on p. 91.
vDS load balancing (LNB):
Route based on IP hash - The virtual switch selects uplinks for virtual machines based on the source and destination IP address of each packet. Route based on source MAC hash - The virtual switch selects an uplink for a virtual machine based on the virtual machine MAC address. To calculate an uplink for a virtual machine, the virtual switch uses the virtual machine MAC address and the number of uplinks in the NIC team. Route based on originating virtual port - Each virtual machine running on an ESXi host has an associated virtual port ID on the virtual switch. To calculate an uplink for a virtual machine, the virtual switch uses the virtual machine port ID and the number of uplinks in the NIC team. After the virtual switch selects an uplink for a virtual machine, it always forwards traffic through the same uplink for this virtual machine as long as the machine runs on the same port. The virtual switch calculates uplinks for virtual machines only once, unless uplinks are added or removed from the NIC team. Use explicit failover order - No actual load balancing is available with this policy. The virtual switch always uses the uplink that stands first in the list of Active adapters from the failover order and that passes failover detection criteria. If no uplinks in the Active list are available, the virtual switch uses the uplinks from the Standby list. Route based on physical NIC load (Only available on vDS) - based on Route Based on Originating Virtual Port, where the virtual switch checks the actual load of the uplinks and takes steps to reduce it on overloaded uplinks. Available only for vSphere Distributed Switch. The distributed switch calculates uplinks for virtual machines by taking their port ID and the number of uplinks in the NIC team. The distributed switch tests the uplinks every 30 seconds, and if their load exceeds 75 percent of usage, the port ID of the virtual machine with the highest I/O is moved to a different uplink.
Virtual switch failover order:
Active uplinks Standby uplinks Unused uplinks 37
CONFIGURE VLAN/PVLAN
SETTINGS
private VLANs allows further segmentation and creation of private groups inside each of the VLAN. By using private VLANs (PVLANs) you splitting the broadcast domain into multiple isolated broadcast “subdomains”. Private VLANs needs to be configured at the physical switch level (the switch must support PVLANs) and also on the VMware vSphere distributed switch. (Enterprise Plus is required). I’ts more expensive and takes a bit more work to setup.
THERE
ARE DIFFERENT TYPES OF
PVLAN S :
P RIMARY
Promiscuous Primary VLAN – Imagine this VLAN as a kind of a router. All packets from the secondary VLANS go through this VLAN. Packets which also goes downstream and so this type of VLAN is used to forward packets downstream to all Secondary VLANs.
S ECONDARY
Isolated (Secondary) – VMs can communicate with other devices on the Promiscuous VLAN but not with other VMs on the Isolated VLAN. Community (Secondary) – VMs can communicate with other VMs on Promiscuous and also w those on the same community VLAN.
The graphics shows it all…
CONFIGURE
TRAFFIC SHAPING POLICIES
Networking Guide p.105 vDS supports both ingress and egress traffic shaping. 38
Traffic shaping policy is applied to each port in the port group. You can Enable or Disable the Ingress or egress traffic
Average bandwidth in kbits (Kb) per second - Establishes the number of bits per second to allow across a port, averaged over time. This number is the allowed average load.
Peak bandwidth in kbits (Kb) per second - Maximum number of bits per second to allow across a port when it is sending or receiving a burst of traffic. This number limits the bandwidth that a port uses when it is using its burst bonus. Burst size in kbytes (KB) per second - Maximum number of bytes to allow in a burst. If set, a port might gain a burst bonus if it does not use all its allocated bandwidth. When the port needs more bandwidth than specified by the average bandwidth, it might be allowed to temporarily transmit data at a higher speed if a burst bonus is available
E NABLE TCP S EGMENTATION O FFLOAD
SUPPORT FOR A VIRTUAL MACHINE
Use TCP Segmentation Offload (TSO) in VMkernel network adapters and virtual machines to improve the network performance in workloads that have severe latency requirements. When TSO is enabled, the network adapter divides larger data chunks into TCP segments instead of the CPU. The VMkernel and the guest operating system can use more CPU cycles to run applications. By default, TSO is enabled in the VMkernel of the ESXi host , and in the VMXNET 2 and VMXNET 3 virtual machine adapters
E NABLE JUMBO F RAMES
SUPPORT ON APPROPRIATE COMPONENTS
There are many places where you can enable Jumbo frames and you should enable jumbo frames end-to-end. If not the performance will not increase, but rather the opposite. Jumbo Frames can be enabled on a vSwitch, vDS, and VMkernel Adapter.
39
Jumbo frames maximum value = 9000.
DETERMINE
APPROPRIATE
VLAN
CONFIGURATION FOR A V S PHERE IMPLEMENTATION
There are three main places or three different ways to tag frames in vSphere.
External Switch Tagging (EST) - VLAN ID is set to None or 0 and it is the physical switch that does the VLAN tagging. Virtual Switch Tagging (VST) - VLAN set between 1 and 4094 and the virtual switch does the VLAN tagging. Virtual Guest Tagging (VGT) - the tagging happens in the guest OS. VLAN set to 4095 (vSwitch) or VLAN trunking on vDS.
The best to understand this is I guess this document from VMware called Best Practices for Virtual Networking and from there I also "borrowed" this screenshot...
Networking is big chapter. If I missed something, just comment or email me your suggestion. Thanks... vSphere documentation tools
vSphere Installation and Setup Guide vSphere Networking Guide What’s New in the VMware vSphere® 6.0 Platform Leveraging NIC Technology to Improve Network Performance in VMware vSphere 40
VDS Network Health Check vSphere Client / vSphere Web Client
VCP6-DCV OBJECTIVE 2.2 - CONFIGURE NETWORK I/O CONTROL (NIOC) VCP6-DCV Study time... In no particular order I start covering VCP6-DCV section of the VMware blueprint to help out folks learning towards VCP6-DCV VMware certification exam. Due to VMware recertification policy the VCP exam has now an expiration date. You can renew by passing delta exam while still holding current VCP or pass VCAP. If you're new to virtualization and do not have any VMware certification exam, the VCP is the exam to have. Today's topic? VCP6-DCV Objective 2.2 - Configure Network I/O Control (NIOC). For whole exam coverage I created a dedicated VCP6-DCV page. If you just look on some how-to, news, videos about vSphere 6 check out my vSphere 6 page. vSphere 6 grew up quite big compared to vSphere 5.5 release, but simplified the deployment and management. "White boxing" got more complicated as drivers for unsupported hardware not always works. vSphere Web client is more present and used in this release as the legacy C# client does not allow to configure advanced configuration options and functions like SSO, FT, VSAN. Let's get started.
vSphere Knowledge
Identify Network I/O Control requirements Identify Network I/O Control capabilities Enable/Disable Network I/O Control Monitor Network I/O Control
IDENTIFY N ETWORK I/O CONTROL
REQUIREMENTS
What is network I/O control? It's a mechanism which allows to prioritize certain data flows on distributed switch over others. It allows to allocate more network bandwidth to business critical applications/VMs where those have to "fight" for bandwidth. (similarly like SIOC for storage).
THE
REQUIREMENTS :
Licensing - Enterprise + license required because it uses vSphere Distributed Switch. VDS Only - the Network I/O control can be enabled only on VDS Network I/O control v3 possible only on VDS 6.0 SR-IOV is not available for virtual machines configured to use Network I/O Control version 3.
IDENTIFY N ETWORK I/O CONTROL
CAPABILITIES
When enabled NIOC divides the traffic into resource pools. Bandwidth reservations can be used to isolate network resources for a class of traffic, for example in VSAN cluster you'd want to reserve part of the traffic only for VSAN traffic no matter what happens to the other traffic.
E NABLE/DISABLE N ETWORK I/O C ONTROL Where to enable? In vSphere 6 when creating new VDS it gets enabled by default. vSphere Web Client > Networking > vDS > Manage > Resource Allocation > System traffic Note: If you have previous version of vSphere and you upgraded, than you might see previous version of NIOC (version 2) and so there is not the menu "system traffic". Make sure that you upgrade your VDS to v 6.0. 41
So in our case we can see the menu system traffic... The traffic types are all set to 50 shares except the VM traffic. No reservation or limits are set by default.
Management traffic - VM traffic NFS traffic Virtual SAN traffic iSCSI vMotion vSphere Replication (VR) Fault tolerance (FT) vSphere Data protection (VDP) backup traffic
Shares and reservations at their default state. No limits or Reservations.
B ANDWIDTH A LLOCATION FOR V IRTUAL M ACHINE TRAFFIC Version 3 of Network I/O Control lets you configure bandwidth requirements for individual virtual machines. You can also use network resource pools where you can assign a bandwidth quota from the aggregated reservation for the virtual machine traffic and then allocate bandwidth from the pool to individual virtual machines. 42
Individual VMs can be configured according to bandwidth requirements through VM options at the network level...
Shares - The relative priority, from 1 to 100, of the traffic through this VM network adapter against the capacity of the physical adapter that is carrying the VM traffic to the network. Reservation - The minimum bandwidth, in Mbps, that the VM network adapter must receive on the physical adapter. Limit - The maximum bandwidth on the VM network adapter for traffic to other virtual machines on the same or on another host. Enable/Disable Network I/O Control - at the vDS level..
To enable bandwidth allocation for virtual machines by using Network I/O Control, configure the virtual machine system traffic. The bandwidth reservation for virtual machine traffic is also used in admission control. When you power on a virtual machine, admission control verifies that enough bandwidth is available. 43
Check the following requirements:
vSphere Distributed Switch is version 6.0.0 and later. Network I/O Control on the switch is version 3. Network I/O Control is enabled.
Network Resource Pools - You can create new network resource pools to reserve part of the aggregated bandwidth for VMs system trafic on all the physical adapters connected to the VDS. For example, if the virtual machine system traffic has 0.5 Gbps reserved on each 10 GbE uplink on a distributed switch that has 10 uplinks, then the total aggregated bandwidth available for VM reservation on this switch is 5 Gbps. Each network resource pool can reserve a quota of this 5 Gbps capacity. Example from vSphere Networking Guide p.167
Create network resource pool: Distributed switch > Manage > Resource allocation > Network resource pools > Add Once you create network resource pool you can add distributed port group so you an allocate bandwidth to the VMs that are connected to that portgroup. Monitor Network I/O Control You can check and monitor Network I/O Control through vSphere web client. Networking > vDS > Manage > Resource Allocation Concerning the system traffic it's possible to have a look a those metrics and details:
44
Network I/O Control Status (state is Enabled/Disabled) NIOC Version Physical network adapters details Available bandwidth capacity Total bandwidth capacity Maximum reservation allowed Configured reservation Minimum link speed
Documentation and Tools
vSphere Installation and Setup Guide vSphere Networking Guide What’s New in the VMware vSphere® 6.0 Platform Performance Evaluation of Network I/O Control in VMware vSphere 6 vSphere Client / vSphere Web Client
VCP6-DCV OBJECTIVE 2.3 – CONFIGURE VSS AND VDS POLICIES VCP6-DCV Study guide continues today by covering the VCP6-DCV Objective 2.3 - Configure vSS and vDS Policies. vSphere networking is one of the tough parts to know and this part is where any IT admins have difficulties. This chapter works hand in hand with the VCP6-DCV Objective 2.1 – Configure Advanced Policies/Features and Verify Network Virtualization Implementation. You can also check the vSphere 6 page where you'll find many how-to, videos, and tutorials about vSphere 6. Let's get back to our today's objective.
vSphere Knowledge
Identify common vSS and vDS policies Describe vDS Security Polices/Settings Configure dvPort group blocking policies Configure load balancing and failover policies Configure VLAN/PVLAN settings Configure traffic shaping policies Enable TCP Segmentation Offload support for a virtual machine Enable Jumbo Frames support on appropriate components Determine appropriate VLAN configuration for a vSphere implementation
IDENTIFY
COMMON V SS AND V DS POLICIES
Since vSphere 4 we have had vSphere distributed switches. But let's start with virtual standard switches first. The virtual standard switches (vSS) can have following policies and settings:
Traffic shaping (outbound only) VLANs (none, VLAN ID, All) - at the portgroup level config MTU Teaming and failover Security
45
If you set VLAN policy to 4095 (All) it allows you to pass All VLANs, and the tagging is done at the Guest OS level vSphere distributed switches (vDS) policies and settings:
Traffic filtering and marking MTU VLANs (none, VLAN ID, VLAN trunking, PVLANs) Monitoring (netflow) Security Traffic Shaping - inbound and outbount (ingress / egress) LACP Port mirroring Health check for VLAN and MTU, teaming and failover - allows to check the status of the overall config. And Teaming and failiover like on vSS swiches.
DESCRIBE V DS S ECURITY P OLICES /S ETTINGS There are three network security policies on vDS. Those are promiscuous mode, MAC address changes and Forged transmits.
Promiscuous Mode - Default settings are set to reject for both (VSS and VDS). If you change to accept then the guest OS can receive all traffic which passes through the vSwitch or Portgroup. MAC address change - The default setting is reject for VDS but accept on VSS. If set to allow then the host accepts requests to change the effective MAC address to a different one than the original. Forged transmits - The default settings is Reject for VDS but accept on VSS. The host do not compare source and effective MAC addresses which are transmitted from a VM.
Each settings can be set to Accept or Reject and it can be done at the virtual switch level or at the port group level. More granular ist's obviously at the port group level.
CONFIGURE
DV P ORT GROUP BLOCKING POLICIES
Ports can be blocked to prohibit them from sending or receiving data. Only available for distributed switches. The port blocking policy is done at the portgroup level. vSphere web client > Networking > Right click a portgroup > Edit settings.
Then you get the Miscelaneous option
46
You can also block individual distributed switch or uplink port. It can be done by selecting the VDS > Manage > Ports > Select Port > Edit > check the box and select Yes.
CONFIGURE
LOAD BALANCING AND FAILOVER POLICIES
vSphere Networking Guide on p. 93 You can configure various load balancing algorithms on a virtual switch to determine how network traffic is distributed between the physical NICs in a team.
Route Based on Originating Virtual Port - The virtual switch selects uplinks based on the virtual machine port IDs on the vSphere Standard Switch or vSphere Distributed Switch.
47
Route Based on Source MAC Hash - The virtual switch selects an uplink for a virtual machine based on the virtual machine MAC address. To calculate an uplink for a virtual machine, the virtual switch uses the virtual machine MAC address and the number of uplinks in the NIC team. Route Based on IP Hash - The virtual switch selects uplinks for virtual machines based on the source and destination IP address of each packet Route Based on Physical NIC Load - Route Based on Physical NIC Load is based on Route Based on Originating Virtual Port, where the virtual switch checks the actual load of the uplinks and takes steps to reduce it on overloaded uplinks.
And for VDS there is another one called Use Explicit Failover Order.
Use Explicit Failover Order - No actual load balancing is available with this policy. The virtual switch always uses the uplink that stands first in the list of Active adapters from the failover order and that passes failover detection criteria. If no uplinks in the Active list are available, the virtual switch uses the uplinks from the Standby list.
N ETWORK F AILOVER D ETECTION OPTIONS :
Link Status only - check link availability. Is the adapter is physically up or down? Depending on the result it can possibly detects physical switch failures. Beacon Probing - Sends out and listens for beacon probes on all NICs in the team. Can be used together with link status and get better results to determine if there is a link failure. Beacon probing should not be used with IP hash load balancing policy or on vSwitches which has less than 3 uplinks. Unused NICs do not participate in beacon probing. Active/active or active/standby only.
FAILOVER ORDER : It can be specified at the vSwitch level or at the port group level, where you basically override the vSwitch level policy (VSS). If there is a failover, then standby NIC became active in order that they're specified/listed. You must define if during failback the physical adapter is returned to active state (and if it is!).
CONFIGURE VLAN/PVLAN
SETTINGS
3 types of VLAN:
None - no tags. Physical switch ports are configured as an access ports or VLAN is configured as native VLAN on trunk port VLAN - in this case, the VLAN ID Tag is done on the virtual switch level. VLAN Trunking - VLANs are tagged at the guest OS level. PVLAN - private VLANs
Note: Same for vSphere web client. You’ll be doing it at the vDS level, so select and right click the vDS > Edit Settings > Private VLAN tab. Once there you can add some PVLANs. Notice the Secondary Promiscuous was created automatically when you created the Primary private VLAN.
48
So in my example above I created Primary Private VLAN 500 which automatically created secondary PVLAN 500. Then I only could create an Isolated Secondary VLAN 501 and Community VLAN 502. Now we have those PVLANs created and this gives us the possibility to use them for new or existing port groups. Example below I’m creating new port group with some name and after selecting the PVLAN, a new drop-down menu appears which gives the option to choose an entry between the Isolated, or Community.
THERE
ARE DIFFERENT TYPES OF
PVLAN S :
P RIMARY
Promiscuous Primary VLAN – Imagine this VLAN as a kind of a router. All packets from the secondary VLANS go through this VLAN. Packets which also goes downstream and so this type of VLAN is used to forward packets downstream to all Secondary VLANs.
S ECONDARY
Isolated (Secondary) – VMs can communicate with other devices on the Promiscuous VLAN but not with other VMs on the Isolated VLAN. Community (Secondary) – VMs can communicate with other VMs on Promiscuous and also w those on the same community VLAN.
49
CONFIGURE
TRAFFIC SHAPING POLICIES
On vDS there are Ingress and Egress traffic shaping policies.
Average bandwidth in kbits (Kb) per second - Bits per second to allow across a port, averaged over time. Peak bandwidth in kbits (Kb) per second - Maximum number of bits per second to allow across a port when it is sending or receiving a burst of traffic. Burst size in kbytes (KB) per second - Maximum number of bytes to allow in a burst.
At the port group level (both Web client or vSphere client). Home > Networking > right click the port group > traffic shaping.
E NABLE TCP S EGMENTATION O FFLOAD
SUPPORT FOR A VIRTUAL MACHINE
(TSO)
TCP segmentation offload is used for reducing a CPU overhead of TCP/IP on fast networks. TSO breaks down large groups of data sent over a network into smaller segments that pass through all the network elements between the source and destination. Only on enhanced vmxnet adapters. If you using just vmxnet you must replace the adapter by enhanced vmxnet adapter. From VMware KB Enabling TSO in a Windows virtual machine To use TSO, enable it in three places: the VMkernel, the virtual machine, and the guest operating system. 1. TSO is enabled for the VMkernel by default. If it is disabled on your system, you can enable it in the VMware Management Interface Advanced Settings page. Access this page by clicking the Options tab. 2. Enable TSO for the virtual machine by powering off the virtual machine and adding the following line to the configuration file (.vmx):ethernetn.features = "0x2" In this example, n is the number of the virtual Ethernet adapter. How to check If a physical network adapter supports TSO? If yes, then TSO is enabled by default.
50
Via CLI - Run this command to see if TSO is supported on the physical network adapter on a host: esxcli network nic tso get
lab output:
E NABLE JUMBO F RAMES
SUPPORT ON APPROPRIATE COMPONENTS
Jumbo frames (MTU 9000) shall be enabled end-to-end if not they will ont raise the network performance, but the opposite will happens. By defaul the MTU is 1500. Jumbo Frames can be enabled on a vSwitch, vDS, and VMkernel Adapter.
DETERMINE
APPROPRIATE
VLAN
CONFIGURATION FOR A V S PHERE IMPLEMENTATION
You should check further the vSphere Networking guide (p.131) VLAN configuration in a vSphere environment provides certain benefits.
Integrates ESXi hosts into a pre-existing VLAN topology. Isolates and secures network traffic. Reduces congestion of network traffic
Tools
vSphere Installation and Setup Guide vSphere Networking Guide Leveraging NIC Technology to Improve Network Performance in VMware vSphere vSphere Client / vSphere Web Client
51
VCP6-DCV OBJECTIVE 3.1 - M ANAGE VSPHERE STORAGE VIRTUALIZATION VMware VCP certification exam for vSphere 6 is now available and you can register for the exam. We'll start to cover VCP6-DCV sections to help out folks learning towards VCP6-DCV VMware certification exam. Today’s topic is VCP6DCV Objective 3.1 - Manage vSphere Storage Virtualization. It's quite large chapter but it' broken into several sections, always with screenshots. We will use vSphere Web Client only (I know not everyone's favorite, but new features aren't exposed to the old C# client anymore...). Due to VMware re-certification policy the VCP exam has now an expiration date. You can renew by passing delta exam while still holding current VCP or pass VCAP. For whole exam coverage I created a dedicated VCP6-DCV page. Or if you’re not preparing to pass a VCP6-DCV, you might just want to look on some how-to, news, videos about vSphere 6 – check out my vSphere 6 page.
vSphere Knowledge
Identify storage adapters and devices Identify storage naming conventions Identify hardware/dependent hardware/software iSCSI initiator requirements Compare and contrast array thin provisioning and virtual disk thin provisioning Describe zoning and LUN masking practices Scan/Rescan storage Configure FC/iSCSI LUNs as ESXi boot devices Create an NFS share for use with vSphere Enable/Configure/Disable vCenter Server storage filters Configure/Edit hardware/dependent hardware initiators Enable/Disable software iSCSI initiator Configure/Edit software iSCSI initiator settings Configure iSCSI port binding Enable/Configure/Disable iSCSI CHAP Determine use case for hardware/dependent hardware/software iSCSI initiator Determine use case for and configure array thin provisioning
IDENTIFY
STORAGE ADAPTERS AND DEVICES
We will be heavily using one document - vSphere 6 Storage Guide PDF. VMware vSphere 6 supports different classes of adapters: SCSI, iSCSI, RAID, Fibre Channel, Fibre Channel over Ethernet (FCoE), and Ethernet. ESXi accesses adapters directly through device drivers in the VMkernel. Note that you must enable certain adapters (like the software iSCSI), but this isn't new as it's been the case already in previous release. W HERE TO CHECK STORAG E ADAPTERS ?
Web Client > Hosts and clusters > host > manage > storage > storage adapters
52
You can also check storage devices there which shows basically all storage attached to the host...
IDENTIFY
STORAGE NAMING CONVENTIONS
When you select the device tab (as on the image above), you'll see that there is a storage device(s) that are accessible to the host. Depending of the type of storage, ESXi host uses different algorithms and conventions to generate an identifier for each storage device. There are 3 types of identifiers:
SCSI Inquire identifiers - the host query via SCSI INSUIRY command a storage device. The resulting data are being used to generate a unique identifier in different formats (naa.number or t10.number OR eui.number). This is because of the T10 standards. Path-based identifiers - ex. mpx.vmhba1:C0:T1:L3 means in details - vmhbaAdapter is the name of the storage adapter. Channel - Target - LUN. MPX path is generated in case the device does not provide a device identifier itself. Note that the generated identifiers are not persistent across reboots and can change. Legacy identifiers - In addition to the SCSI INQUIRY or mpx. identifiers, for each device, ESXi generates an alternative legacy name. The identifier has the following format:
vml.number The legacy identifier includes a series of digits that are unique to the device. Check via CLI to see all the details: esxcli storage core device list
53
Note that the display name can be changed - web client Select host > Manage > Storage > Storage Devices > select > click rename icon.
54
There are also: Fibre Channel targets which uses World Wide Names (WWN)
World Wide Port Names (WWPN) World Wide Node Names (WWNN)
Check vSphere Storage Guide p.64 for iSCSI naming conventions B ASICALLY SIMILAR TO THE W ORLDW IDE N AME (WWN) FOR FC DEVICES . ISCSI NAMES ARE FORMATTED IN TWO DIFFERENT WAYS . T HE MOST COMMON IS THE IQN FORMAT . iSCSI Qualified Name (IQN) Format
iqn.yyyy-mm.naming-authority:unique name, where:
yyyy-mm is the year and month when the naming authority was established. naming-authority is usually reverse syntax of the Internet domain name of the naming authority. For example, the iscsi.vmware.com naming authority could have the iSCSI qualified name form of iqn. 1998-01.com.vmware.iscsi. The name indicates that the vmware.com domain name was registered in January of 1998, and iscsi is a subdomain, maintained by vmware.com. unique name is any name you want to use, for example, the name of your host. The naming authority must make sure that any names assigned following the colon are unique, such as: o iqn.1998-01.com.vmware.iscsi:name1 o iqn.1998-01.com.vmware.iscsi:name2 o iqn.1998-01.com.vmware.iscsi:name999
OR ENTERPRISE U NIQUE I DENTIFIER (EUI) NAMING FORMAT
eui.16 hex Example: eui.16hexdigits ie eui.0123456789ABCDEF IDENTIFY
digits.
HARDWARE / DEPENDENT HARDWARE / SOFTWARE I SCSI INITIATOR REQUIREMENTS
Two types of iSCSI adapters.
Hardware based - add-On iSCSI cards (can do boot-on-lan). Those types of adapters are also capable of offloading the iSCSI and network processing so the CPU activity is lower. Hardware adapters can be dependent or independent. Compared to Dependent, the Indpendent adapters do not use VMkernel adapters for connections to the storage. Software based - activated after installation (cannot do boot-on-lan). Brings a very light overhead. Software based iSCSI uses VMkernel adapter to connect to iSCSI storage over a storage network.
Dependent adapters can use CHAP, which is not the case of Independent adapters.
COMPARE
AND CONTRAST ARRAY THIN PROVISIONING AND VIRTUAL DISK THIN PROVISIONING
55
Virtual disk thin provisioning allows to allocate only small amount of disk space at the storage level, but the guest OS sees as it had the whole space. The thin disk grows in size when adding more data, installing applications at the VM level. So it's possible to over-allocate the datastore space, but it brings a risks so it's important to monitor actual storage usage to avoid conditions when you run out of physical storage space. Image says thousands words... p.254 of vSphere Storage Guide
Thick Lazy Zeroed - default thick format. Space is allocated at creation, but the physical device is not erased during the creation proces, but zeroed-on-demand instead. Thick Eager Zeroed - Used for FT protected VMs. Space is allocated at creation and zeroed immediately. The Data remaining on the physical device is zeroed out when the virtual disk is created. Takes longer to create Eager Zeroed Thick disks. Thin provission - as on the image above. Starts small and at first, uses only as much datastore space as the disk needs for its initial operations. If the thin disk needs more space later, it can grow to its maximum capacity and occupy the entire datastore space provisioned to it. Thin disk can be inflated (thin > thick) via datastore browser (right click vmdk > inflate).
Check the different VMDK disk provisioning options when creating new VM or adding an additional disk to existing VM
56
Thin-provissioned LUN Array Thin Provisioning and VMFS Datastores on p. 257. ESXi also supports thin-provisioned LUNs. When a LUN is thin-provisioned, the storage array reports the LUN's logical size, which might be larger than the real physical capacity backing that LUN. A VMFS datastore that you deploy on the thin-provisioned LUN can detect only the logical size of the LUN. For example, if the array reports 2TB of storage while in reality the array provides only 1TB, the datastore considers 2TB to be the LUN's size. As the datastore grows, it cannot determine whether the actual amount of physical space is still sufficient for its needs. Via Storage API -Array integration (VAAI) you CAN be aware of underlying thing-provisioned LUNs. VAAI let the array know about datastore space which has been freed when files are deleted or removed to allow the array to reclaim the freed blocks. Check thin provissioned devices via CLI: esxcli storage core device list -d vml.xxxxxxxxxxxxxxxx
57
DESCRIBE
ZONING AND
LUN
MASKING PRACTICES
Zoning is used with FC SAN devices. Allow controlling the SAN topology by defining which HBAs can connect to which targets. We say that we zone a LUN. Allows:
Protecting from access non desired devices the LUN and possibly corrupt data Can be used for separation different environments (clusters) Reduces number of targets and LUN presented to host Controls and isolates paths in a fabric.
Best practice? Single-initiator-single target
58
LUN MASKING
esxcfg-scsidevs -m — the -m esxcfg-mpath -L | grep naa.5000144fd4b74168 esxcli storage core claimrule add -r 500 -t location -A vmhba35 -C 0 -T 1 -L 0 -P MASK_PATH esxcli storage core claimrule load esxcli storage core claiming reclaim -d naa.5000144fd4b74168 U NMASK A LUN
esxcli storage core claimrule remove -r 500 esxcli storage core claimrule load esxcli storage core claiming unclaim -t location -A vmhba35 -C 0 -T 1 -L 0 esxcli storage core adapter rescan -A vmhba35 S CAN/R ESCAN STORAGE
Perform the manual rescan each time you make one of the following changes.
Zone a new disk array on a SAN. Create new LUNs on a SAN. Change the path masking on a host. Reconnect a cable. Change CHAP settings (iSCSI only). Add or remove discovery or static addresses (iSCSI only). Add a single host to the vCenter Server after you have edited or removed from the vCenter Server a datastore shared by the vCenter Server hosts and the single host.
You can scan at the Host level or at the datacenter level (storage > select datacenter > right click > Storage > Rescan storage.
Click host > manage > storage > storage adapters
59
Scan for New Storage Device – Rescans HBAs for new storage devices Scan for New VMFS Volumes – Rescans known storage devices for VMFS volumes
CONFIGURE FC/ ISCSI LUN S AS ESX I BOOT DEVICES Few requirements. As being said, only the hardware iSCSI can boot from LUN. Boot from SAN is supported on FC, iSCSI, and FCoE.
1:1 ratio - Each host must have access to its own boot LUN only, not the boot LUNs of other hosts. Bios Support - Enable the boot adapter in the host BIOS HBA config - Enable and correctly configure the HBA, so it can access the boot LUN.
Docs:
Boot from FC SAN - vSphere Storage Guide on p. 49 Boot from iSCSI SAN - p.107. Boot from Software FCoE - P.55
CREATE
AN
NFS
SHARE FOR USE WITH V S PHERE
An NFS client built into ESXi uses the Network File System (NFS) protocol over TCP/IP to access a designated NFS volume that is located on a NAS server. The ESXi host can mount the volume and use it for its storage needs. vSphere supports versions 3 and 4.1 of the NFS protocol. How? By exporting NFS volume as NFS v3 or v4.1 (latest release). Different storage vendors have different methods of enabling this functionality, but typically this is done on the NAS servers by using the no_root_squash option. If the NAS server does not grant root access, you might still be able to mount the NFS datastore - but read only. NFS uses VMkernel port so you need to configure one. v3 and v4.1 compare:
60
E NABLE/CONFIGURE/DISABLE V CENTER S ERVER
STORAGE FILTERS
When you perform VMFS datastore management operations, vCenter Server uses default storage protection filters. The filters help you to avoid storage corruption by retrieving only the storage devices that can be used for a particular operation. Unsuitable devices are not displayed for selection. p. 167 of vSphere 6 storage guide.
Where? Hosts and clusters > vCenter server > manage > settings > advanced settings
In the value box type False for appropriate key. From the vSphere Storage Guide:
61
CONFIGURE /E DIT HARDWARE / DEPENDENT HARDWARE INITIATORS W HERE ?
Host and Clusters > Host > Manage > Storage > Storage Adapters. It's possible to rename the adapters from the default given name. It's possible to configure the dynamic and static discovery for the initiators.
It's not so easy to find through Web client, as before we use to do it eyes closed through a vSphere client...
E NABLE/DISABLE
SOFTWARE I SCSI INITIATOR
CONFIGURE/E DIT
SOFTWARE I SCSI INITIATOR SETTINGS
As being said above, to configure and Edit Software iSCSI initiator settings, you can use Web client or C# client. Web Client > Host and Clusters > Host > Manage > Storage > Storage Adapters And there you can:
View/Attach/Detach Devices from the Host Enable/Disable Paths Enable/Disable the Adapter Change iSCSI Name and Alias Configure CHAP 62
Configure Dynamic Discovery and (or) Static Discovery Add Network Port Bindings to the adapter Configure iSCSI advanced options
CONFIGURE I SCSI
PORT BINDING
Port binding allows to configure multipathing when :
iSCSI ports of the array target must reside in the same broadcast domain and IP subnet as the VMkernel adapters. All VMkernel adapters used for iSCSI port binding must reside in the same broadcast domain and IP subnet. All VMkernel adapters used for iSCSI connectivity must reside in the same virtual switch. Port binding does not support network routing.
Do not use port binding when any of the following conditions exist:
Array target iSCSI ports are in a different broadcast domain and IP subnet. VMkernel adapters used for iSCSI connectivity exist in different broadcast domains, IP subnets, or use different virtual switches. Routing is required to reach the iSCSI array.
Note: The VMkernel adapters must be configured with single Active uplink. All the others as unused only (not Active/standby). If not they are not listed...
E NABLE/CONFIGURE/DISABLE I SCSI CHAP W HERE ? Web Client > Host and Clusters > Host > Manage > Storage > Storage Adapters > Properties > Authentication (Edit button).
63
p. 98 of vSphere 6 Storage Guide. Challenge Handshake Authentication Protocol (CHAP), which verifies the legitimacy of initiators that access targets on the network. Unidirectional CHAP - target authenticates the initiator, but the initiator does not authenticate the target. Bidirectional CHAP - an additional level of security enables the initiator to authenticate the target. VMware supports this method for software and dependent hardware iSCSI adapters only. CHAP METHODS :
None - CHAP authentication is not used. Use unidirectional CHAP if required by target - Host prefers non-CHAP connection but can use CHAP if required by target. Use unidirectional CHAP unless prohibited by target - Host prefers CHAP, but can use non-CHAP if target does not support CHAP. Use unidirectional CHAP - Requires CHAP authentication. Use bidirectional CHAP - Host and target support bidirectional CHAP.
CHAP does not encrypt, only authenticates the initiator and target. Determine use case for hardware/dependent hardware/software iSCSI initiator It's fairly simple, as we know that if we use the software iSCSI adapter we do not have to buy additional hardware and we're still able to "hook" into iSCSI SAN. The case for Dependent Hardware iSCSI Adapter which is dependant on the VMKernel adapter but offloads iSCSI processing to the adapter, which accelerates the treatment and reduces CPU overhead. On the other hand, the Independent Hardware iSCSI Adapter has its own networking, iSCSI configuration, and management interfaces. So you must go through the BIOS and the device configuration in order to use it.
DETERMINE
USE CASE FOR AND CONFIGURE ARRAY THIN PROVISIONING
64
Some arrays do support thin provissioned LUNs while others do not. The benefit is to offer more capacity (visible) to the ESXi host while consuming only what's needed at the datastore level. (attention however for over-subscribing, so proper monitoring is needed). So at the datastore level it's possible to use thin provisioned virtual disk or on the array using thin provisioned LUNs.
Tools
vSphere Installation and Setup Guide vSphere Storage Guide Best Practices for Running VMware vSphere® on iSCSI vSphere Client / vSphere Web Client
VCP6-DCV OBJECTIVE 3.2 - CONFIGURE SOFTWARE -DEFINED STORAGE VCP6-DCV (datacenter virtualization) VMware certification exam was recently released and the registration will be soon available. The term software-defined, you can love or hate this term, but Software-defined storage is here, and this post covers VCP6-DCV Objective 3.2 - Configure Software-defined Storage. Hopefully it will help you to learn this topic towards the exam... For whole exam coverage I created a dedicated VCP6-DCV page. Or if you're not preparing to pass a VCP6-DCV, you might just want to look on some how-to, news, videos about vSphere 6 - check out my vSphere 6 page. If you find out that I missed something, don't hesitate to comment.
vSphere Knowledge Covered in this post:
Configure/Manage VMware Virtual SAN Create/Modify VMware Virtual Volumes (VVOLs) Configure Storage Policies Enable/Disable Virtual SAN Fault Domains
CONFIGURE/M ANAGE VM WARE V IRTUAL SAN
VMware VSAN (traditional) needs some spinning media (SAS or SATA) and 1 SSD per host (SATA, SAS or PCIe). VMware VSAN (All-Flash) needs some SATA/SAS for capacity tier and 1 SSD hight performance and endurance for caching. HBA which is on the VMware HCL (queue depth > 600) All hardware must be part of HCL (or if you want easy way -> via VSAN ready nodes!) HBA with RAID0 jor direct pass-through so ESXi can see the individual disks, not a raid volume. SSD sizing - 10% of consumed capacity 1Gb Network (10GbE recommended) 1 VMkernel unterface configured (dedicated) for VSAN traffic Multicast activated on the switch IGMP Snooping and an IGMP Querier can be used to filter multicast traffic to a limited to specific port group. Usefull if other non-Virtual SAN network devices exist on the same layer 2 network segment (VLAN). IPv4 only on the switch Minimum 3 hosts in the cluster (4 recommended) - maxi. 64 hosts (vSphere 6)
CREATE VM KERNEL INTERFACE WITH VSAN TRAFFIC ON 65
Host > Manage > Networking > VMkernel Adapters > Add
ENABLE VSAN AT THE CLUSTER LEVEL
Hosts and Clusters > Cluster > Manage > Settings > Virtual SAN > General
Add disk to storage:
Manual – Requires manual claiming of any new disks. Automatic – All empty disks on cluster hosts will be automatically claimed by VSAN
CREATE DISK GROUPS
Hosts and Clusters > Cluster > Manage > Settings > Virtual SAN > Disk Management
66
CLAIM DISKS FOR VSAN
You can do several tasks when managing disk in VSAN cluster.
Claim Disks for VSAN Create a new disk group (when adding more capacity). Remove the disk group Add a disk to the selected disk group Place a host in maintenance mode
S O H OW TO M ARK LOCAL DISK AS SSD DISK ? Connect to your vCenter > Go to Hosts and clusters > Select a Host > Select disk which you want to tag as SSD. You can click to enlarge.
This brings a small warning window saying that you might deteriorate the performance of datastores and services that use them, but if you’re sure on what you’re doing, then go ahead and validate on Yes button.
As a result, after few seconds (without even refreshing the client’s page) the disk turns into a SSD disk… It’s magic, no?
It works also the other way around! SSD to HDD. Note that this works only in VSAN 6.0! TAG D ISKS FOR CAPACITY OR CACHING 67
So let’s demonstrate it in my lab. I use VMware Workstation for the job where I quickly created few ESXi VMs. I configured the ESXi 6 host with 7 hard drives, where each virtual disk is destined to fill different function. Here are the details:
40Gb is local disk where is installed ESXi 20 Gb drives are the ones which I need to tag as capacity 5 Gb drive is the caching tier
The view or our disks…
To check the status of your disks as ESXi sees them you can use the vdq -q command So in our case:
vdq -q gives us this:
We can see that the mpx.vmhba1:C0:T6:L0 is our disk which we need to tag to be able to use is in our disk group. (otherwise the disk won’t appear to be used in VSAN as capacity tier). We need to connect via SSH to our host. If you haven’t enabled yet, please enable SSH by going and selecting your host > Manage > Security Profile > services > Edit After you have identified the disk which you need to tag, just enter this command: esxcli vsan storage tag add -d naa.XYZ -t capacityFlash
68
where naa.XYZ is your hard drive. In my example esxcli vsan storage tag add -d mpx.vmhba1:C0:T5:L0 -t capacityFlash
After tagging all of the 20Gb disks we can create a disk group where those disks will appear as data disks below… (You can see that our mpx.vmhba1:C0:T6:L0 device can now be selected to be used data disk)…
Note: You can not only tag but also untag! Check this: esxcli vsan storage tag remove -d naa.XYZ -t capacityFlash the above command will simply remove the “capacityFlash” tag from the storage device. How to check if SSD is participating as capacity tier or not? So if you just want to check which tag does your storage has you can use this command: vdq -q See the output here…
69
You should get this VSAN Troubleshooting Reference Manual which is great resource VSAN AND M AINTENANCE M ODE Maintenance mode for each ESXi participating in VSAN cluster has new options depending what you want to do with the data located on the particular host (the object's locations are on the local storage of each host) So, Virtual SAN host's when you want to put them in maintenance mode thay allows 3 options:
Ensure accessibility - Virtual SAN ensures that all virtual machines on this host will remain accessible if the host is shut down or removed from the cluster. Full data migration - Virtual SAN migrates all data that resides on this host. No data migration - Virtual SAN will not migrate any data from this host. Some virtual machines might become inaccessible if the host is shut down or removed from the cluster.
CREATE/M ODIFY VM WARE V IRTUAL V OLUMES (VVOLS)
70
VVOls are new in vSphere 6. By using a special set of APIs called vSphere APIs for Storage Awareness (VASA), the storage system becomes aware of the virtual volumes and their associations with the relevant virtual machines. Through VASA, vSphere and the underlying storage system establish a two-way out-ofband communication to perform data services and offload certain virtual machine operations to the storage system. For example, such operations as snapshots, storage DRS and clones can be offloaded.
VVOLs are supported on SANs compatible with VAAI (vSphere APIs for Array Integration). VVOLs supports vMotion, sVMotion, Snapshots, Linked-clones, vFRC, DRS VVOLs supports backup products which uses VADP (vSphere APIs for Data Protection) VVOLs supports FC, FCoE, iSCSI and NFS
Image courtesy VMware
71
VVOL S L IMITATIONS
VVOLs Does not works with standalone ESXi hosts (needs vCenter) VVOLs do not support RDMs VVOLs wih the virtual datastores are tighten to vCenter sor if used with Host profiles, than only within this particular vCenter as the extracted host profile can be attached only to the hosts withing the same vCenter as the reference host is located. No IPv6 support NFS v3 only (v4.1 isn't supported) Multipathing only on SCSI-based endpoints, not on NFS-based protocol endpoint.
VVOLs vSphere Storage Guide p211. Virtual volumes are encapsulations of virtual machine files, virtual disks, and their derivatives. Virtual volumes are not preprovisioned, but created automatically when you perform virtual machine management operations. These operations include a VM creation, cloning, and snapshotting. ESXi and vCenter Server associate one or more virtual volumes to a virtual machine.
Storage Provider - A Virtual Volumes storage provider, also called a VASA provider, is a software component that acts as dastorage awareness service for vSphere.
Storage Container - A storage container is a part of the logical storage fabric and is a logical unit of the underlying hardware. The storage container logically groups virtual volumes based on management and administrative needs. Protocol Endpoints -ESXi hosts use a logical I/O proxy, called the protocol endpoint, to communicate with virtual volumes and virtual disk files that virtual volumes encapsulate. ESXi uses protocol endpoints to establish a data path on demand from virtual machines to their respective virtual volumes. Virtual Datastores - A virtual datastore represents a storage container in vCenter Server and the vSphere Web Client.
Steps to Enable VVOLs (p.218):
Step 1: Register Storage Providers for VVOLs
vCenter Inventory Lists > vCenter Servers > vCenter Server > Manage > Storage Providers
Step 2: Create a Virtual Datastore 72
vCenter Inventory Lists > Datastores
Step 3: Review and manage protocol endpoints
vCenter Inventory Lists > Hosts > Host > Manage > Storage > Protocol Endpoints
(optional) Change the path selection policy (psp) for protocol endpoint.
Manage > Storage > Protocol Endpoints > select the protocol endpoint you want to change and click Properties > Under multipathing Policies click Edit Multipathing
CONFIGURE S TORAGE P OLICIES (VM
STORAGE POLICIES )
Virtual Machine Storage policies are covered vSphere Storage Guide on p. 225. Virtual machine storage policies are essential to virtual machine provisioning. These policies help youdefine storage requirements for the virtual machine
73
and control which type of storage is provided for the virtual machine, how the virtual machine is placed within the storage, and which data services are offered for the virtual machine. SP contains storage rule or collection of storage rules. define a storage policy, you specify storage requirements for applications that run on virtual machines. After you apply this storage policy to a virtual machine, the virtual machine is placed in a specific datastore that can satisfy the storage requirements. In case of VSAN and VVOLs, the SP determines how the VM storage objects are handled and allocated within the datastore to guarantee the SLA.
Rules based on storage-specific data service - VSAN and VVOLs uses VASA to surface the storage capability to VMstorage policies's interface Rules based on TAGs - by tagging a specific datastore. More than One tag can be applied per datastore
V IEW VM S AND DISKS IF THEY COMPLY WITH VM STORAGE POLICIES VM Storage Policies > Click a particular Storage Policy > Monitor
74
E NABLE/DISABLE V IRTUAL SAN F AULT DOMAINS VSAN fault domains allows to create an environment where the in case of failure 2 hosts for example, which are in the same rack. Failure of all hosts within a single fault domain is treated as one failure. VSAN will not store more than one replica in this group (domain). VSAN Storage Guide p.22 Requirements: 2*n+1 fault domains in a cluster. In order to leverage fault domain you need at least 6 hosts (3 fault domains). Using a three domains does not allow the use of certain evacuation modes, nor is Virtual SAN able to reprotect data after a failure. VMware recommends 4 Fault domains. (the same for vSAN clusters - 4 hosts in a VSAN cluster). On the pic below you see my hosts are down, but VSAN still works and provide storage for my VM... (nested environment). Hosts and Clusters > Cluster > Manage > Settings > Virtual SAN > Fault Domains
If a host is not a member of a fault domain, Virtual SAN interprets it as a separate domain.
Tools
Administering VMware Virtual SAN vSphere Storage Guide What's New: VMware Virtual SAN 6.0 What’s New in the VMware vSphere® 6.0 Platform Virtual SAN 6.0 Performance: Scalability and Best Practices vSphere Client / vSphere Web Client
vSphere how-to, news, videos on my Dedicated vSphere 6 page!
75
VCP6-DCV OBJECTIVE 3.3 - CONFIGURE VSPHERE STORAGE MULTI-PATHING AND FAILOVER Today’s VCP6-DCV goal is to talk about VCP6-DCV Objective 3.3 - Configure vSphere Storage Multi-pathing and Failover. VMware VCP exam is a gold standard of VMware certification exams. VMware vSphere 6 brings new certification exam. VCP exam is the most known VMware exams, even if it’s not the highest technical level. But it’s most recognized. By a future employer, by industry as a whole. We will cover VCP6-DCV exam certification based on VMware latest VMware VCP6-DCV blueprint. Check VCP6-DCV page for all objectives.
vSphere knowledge
Configure/Manage Storage Load Balancing Identify available Storage Load Balancing options Identify available Storage Multi-pathing Policies Identify features of Pluggable Storage Architecture (PSA) Configure Storage Policies Enable/Disable Virtual SAN Fault Domains
CONFIGURE/M ANAGE S TORAGE LOAD B ALANCING The goal of load balancing policy is to give equal "chance" to each storage processors and the host server paths by distributing the IO requests equally. Using the load balancing methods allows to optimize Response time, IOPs or MBPs for VMs performance. To get started, if you're using block storage - check the Storage > Datastore > Manage > Settings > Connectivity and Multipathing
76
IDENTIFY
AVAILABLE
S TORAGE LOAD B ALANCING
OPTIONS
You can manage multipathing using the vSphere Client, the esxcli command, or using the following commands. Use the HostStorageSystem.multipathStateInfo property to access the HostMultipathStateInfo. SAN storage systems require continual redesign and tuning to ensure that I/O is load balanced across all storage system paths. To meet this requirement, distribute the paths to the LUNs among all the SPs toprovide optimal load balancing. Multipathing allows you to have more than one physical path from the ESXi host to a LUN on a storage system. Generally, a single path from a host to a LUN consists of an iSCSI adapter or NIC, switch ports, connecting cables, and the storage controller port. If any component of the path fails, the host selects another available path for I/O. The process of detecting a failed path and switching to another is called path failover. Path information:
Active - Paths available for issuing I/O to a LUN. A single or multiple working paths currently used for transferring data are marked as Active (I/O). Standby - If active paths fail, the path can quickly become operational and can be used for I/O Disabled - path disabled, no transfer possible. Dead - impossible to connect to the disk via this path.
IDENTIFY
AVAILABLE
S TORAGE M ULTI - PATHING P OLICIES
You can select different path selection policy from the default ones, or if you have installed a third party product which has added its own PSP: Fixed - (VMW_PSP_FIXED) the host uses designated preferred path if configured. If not it uses first working path discovered. Prefered path needs to be configured manually.
77
Most Recently Used - (VMW_PSP_MRU) The host selects the path that it used most recently. When the path becomes unavailable, the host selects an alternative path. The host does not revert back to the original path when that path becomes available again. There is no preferred path setting with the MRU policy. MRU is the default policy for most active-passive arrays.
Round Robin (RR) - VMW_PSP_RR - The host uses an automatic path selection algorithm rotating through all active paths when connecting to active-passive arrays, or through all available paths when connecting to active-active arrays. RR is the default for a number of arrays and can be used with both active-active and active-passive arrays to implement load balancing across paths for different LUNs.
78
IDENTIFY
FEATURES OF
P LUGGABLE S TORAGE A RCHITECTURE (PSA)
VMware NMP - default multipathing module (Native Multipathing Plugin). Nmp plays a role when associating the set of physical paths with particular storage device or LUN, but delegates the details to SATP plugin. On the other hand the choice of path used when IO comes is is handled by PSP (Path Selection Plugin) VMware SATP - Storage Array Type Plugins runs hand in hand with NMP and are responsible for array based operations. ESXi has SATP for every supported SAN, It also provides default SATPs that support non-specific active-active and ALUA storage arrays, and the local SATP for direct-attached devices. VMware PSPs - Path Selection Plugins are sub plugins of VMware NMP and they choose a physical path for IO requests.
The multipathing modules perform the following operations:
Manage physical path claiming and unclaiming. Manage creation, registration, and deregistration of logical devices. Associate physical paths with logical devices. Support path failure detection and remediation. Process I/O requests to logical devices: o Select an optimal physical path for the request. o Depending on a storage device, perform specific actions necessary to handle path failures and I/O command retries. Support management tasks, such as reset of logical devices.
CONFIGURE S TORAGE P OLICIES A storage policy can include multiple rule sets. Storage-Specific Data Service rules and Tag based rules can be combined in the same storage policy. VM Storage Policies, where? Home > VM Storage Policies Guide: vSphere Storage Guide on p. 225 Storage rules based on: Rules based on storage-specific data service – VSAN and VVOLs uses VASA to surface the storage capability to VMstorage policies’s interface. To supply information about underlying storage to vCenter Server, Virtual SAN and Virtual Volumes use storage providers, also called VASA providers. Storage information and datastore characteristics appear in the VM Storage Policies interface of the vSphere Web Client as data services offered by the specific datastore type.
79
Rules based on TAGs – by tagging a specific datastore. More than One tag can be applied per datastore.
First you must tag a datastore
Then you go back to a VM storage policy > Add new policy icon > put some meaningful name > click Add tag-based rule > choose your rule from the category drop down menu > click Next > choose a compatible datastore
80
Check compliance via VM storage Policies > Storage policy > monitor
If you want to change from default storage policy to newly created one, you must first change it at the VM level and then check back at VM storage Policies > Storage policy > monitor
81
E NABLE/DISABLE V IRTUAL SAN F AULT DOMAINS VMware fault domains in VSAN environment allows to spread the replicas over different locations (different racks) in order to "not to put all eggs in the same basket" - literarly. Let's say you have 4 hosts per rack and you want to achieve a redundancy in case of failure multiple components within single rack. VSAN considers each fault domain as single host. Virtual SAN Fault Domains ensures replicas of VM data is spread across the defined failure domains. Fault domains provide the ability to tolerate:
Rack failures Storage controller Network failures Power failure
Image courtesy of VMware
Where to manage VSAN fault domains? Hosts and Clusters > Cluster > Manage > Settings > Virtual SAN > Fault Domains
82
If a host is not a member of a fault domain, Virtual SAN interprets it as a separate domain. VMware recommends to configure minimum 3 or more fault domains in the VSAN cluster, and also you should assing the same number of hosts per fault domain. It's not necessary however assign all hosts to fault domains. Note: If a host is moved to another cluster, VSAN hosts retain their fault domain assignements. Tools:
vSphere Installation and Setup Guide vSphere Storage Guide Multipathing Configuration for Software iSCSI Using Port Binding vSphere Client / vSphere Web Client
VCP6-DCV OBJECTIVE 3.4 - PERFORM ADVANCED VMFS AND NFS CONFIGURATIONS AND UPGRADES This post covers VCP6-DCV Objective 3.4 - Perform Advanced VMFS and NFS Configurations and Upgrades. Important storage chapter where you'll learn the inside out about VMFS, datastores, management or enable/disable vStorage API for array integration. For whole exam coverage I created a dedicated VCP6-DCV page which follows the exam's blueprint. If you just want to look on some how-to, news, videos about vSphere 6 – check out my vSphere 6 page. If you find out that I missed something in this post, don’t hesitate to comment.
83
VMware vSphere Knowledge
Identify VMFS and NFS Datastore properties Identify VMFS5 capabilities Create/Rename/Delete/Unmount a VMFS Datastore Mount/Unmount an NFS Datastore Extend/Expand VMFS Datastores Place a VMFS Datastore in Maintenance Mode Identify available Raw Device Mapping (RDM) solutions Select the Preferred Path for a VMFS Datastore Enable/Disable vStorage API for Array Integration (VAAI) Disable a path to a VMFS Datastore Determine use case for multiple VMFS/NFS Datastores
IDENTIFY VMFS
AND
NFS DATASTORE
PROPERTIES
What's Datastore? - it's kind of a logical container which stores VMDKs of your VMs. VMFS is a clustered file system which allows multiple hosts access files on shared datastore. VMFS uses locking mechanism (ATS or ATS + SCSI) which prevents multiple hosts from concurrently writing to the metadata and ensure that there is no data corruption. Check Page 149 for vSphere Storage guide for more on the ATS or ATS+SCSI locking mechanism. NFS - Network file system, can be mounted by ESXi host (which uses NFS client). NFS datastores supports vMotion or SvMotion, HA, DRS, FT or host profiles (note that NFS 4.1 do not supports FT). NFS v3 and NFS v4.1 are supported with vSphere 6.0. VMDKs are provisionned as "Thin" by default on the NFS datastore.
IDENTIFY VMFS5
CAPABILITIES
Larger than 2TB storage devices for each VMFS5 extent. Support of virtual machines with large capacity virtual disks, or disks greater than 2TB. Increased resource limits such as file descriptors. Standard 1MB file system block size with support of 2TB virtual disks. Greater than 2TB disk size for RDMs Support of small files of 1KB.Ability to open any file located on a VMFS5 datastore in a shared mode by a maximum of 64 hosts. Can reclaim physical storage space on thin provisioned storage devices.
Upgrades from previous version of VMFS:
VMFS datastores can be upgraded without disrupting hosts or virtual machines. If creating new VMFS datastore there is choice to create VMFS 3 or VMFS 5 version of datastore New VMFS datastores are created with the GPT format. VMFS datastore which has been upgraded will continue to use the MBR format until it is expanded beyond 2TB. If that's the case then the MGS format is converted to GPT. Maximum VMFS datastores per host - 256 VMFS datastores Host needs to run ESXi 5.0 or higher No way back (VMFS 5 to VMFS 3) the upgrade process cannot downgrade back to VMFS v3.
84
CREATE/R ENAME/D ELETE/UNMOUNT
A
VMFS DATASTORE
Create Datastore - vSphere Web Client > Hosts and Clusters > Select Host > Actions > Storage > New Datastore
And you have a nice assistant which you follow...
The datastore can be created also via vSphere C# client. To rename datastore > Home > Storage > Right click datastore > Rename
85
As you can see you can also unmount or delete datastore via the same right click.
Make sure that:
There are NO VMs on that datastore you want to unmount. If HA configured, make sure that the datastore is not used for HA heartbeats Check that the datastore is not managed by Storage DRS Verify also that Storage IO control (SIOC) is disabled on the datastore
M OUNT /UNMOUNT
AN
NFS DATASTORE
Create NFS mount. Similar way as above Right click datacenter > Storage > Add Storage.
86
You can use NFS 3 or NFS 4.1 (note the limitations of NFS 4.1 for FT or SIOC). Enter the Name, Folder, and Server (IP or FQDN) To Mount/unmout NFS datastore...
And then choose the host(s) to which you want this datastore to mount...
E XTEND /E XPAND VMFS DATASTORES It's possible to expand existing datastore by using extent OR by growing an expandable datastore to fill the available capacity.
87
and then you just select the device..
You can also Add a new extent. Which means that datastore can span over up to 32 extents and appear as a single volume.... But in reality, not many VMware admins likes to use extents....
P LACE
A
VMFS DATASTORE
IN
M AINTENANCE M ODE
Maintenance mode for datastore is available if the datastore takes part in Storage DRS cluster. (SDRS). Regular datastore cannot be placed in maintenance mode. So if you want to activate SDRS you must first create SDRS cluster by Right click Datacenter > Storage > New Datastore Cluster. then only you can put the datastore in maintenance mode...
88
IDENTIFY
AVAILABLE
R AW DEVICE M APPING (RDM)
SOLUTIONS
vSphere storage guide p. 203. RDM allows a VM directly access a LUN. Think of an RDM as a symbolic link from a VMFS volume to a raw LUN.
An RDM is a mapping file in a separate VMFS volume that acts as a proxy for a raw physical storage device. The RDM allows a virtual machine to directly access and use the storage device. The RDM contains metadata for managing and redirecting disk access to the physical device. When to use RDM?
When SAN snapshot or other layered applications run in the virtual machine. The RDM better enables scalable backup offloading systems by using features inherent to the SAN. In any MSCS clustering scenario that spans physical hosts — virtual-to-virtual clusters as well as physical-tovirtual clusters. In this case, cluster data and quorum disks should be configured as RDMs rather than as virtual disks on a shared VMFS.
If RDM is used in physical compatibility mode - no snapshoting of VMs... Virtual machine snapshots are available for RDMs with virtual compatibility mode. Physical Compatibility Mode - VMkernel passes all SCSI commands to the device, with one exception: the REPORT LUNs command is virtualized so that the VMkernel can isolate the LUN to the owning virtual machine. If not, all physical characteristics of the underlying hardware are exposed. It does allows the guest operating system to access the hardware directly. VM with physical compatibility RDM has limits like that you cannot clone such a VM or turn it into a template. Also sVMotion or cold migration is not possible. Virtual Compatibility Mode - VMkernel sends only READ and WRITE to the mapped device. The mapped device appears to the guest operating system exactly the same as a virtual disk file in a VMFS volume. The real hardware characteristics are hidden. If you are using a raw disk in virtual mode, you can realize the benefits of VMFS such as advanced file locking for data protection and snapshots for streamlining development processes. Virtual mode is also more portable across storage hardware than physical mode, presenting the same behavior as a virtual disk file. (VMDK). You can use snapshots, clones, templates When an RDM disk in virtual compatibility mode is cloned or a template is created out of it, the contents of the LUN are copied into a .vmdk virtual disk file. Other limitations:
89
You cannot map to a disk partition. RDMs require the mapped device to be a whole LUN. VFRC - Flash Read Cache does not support RDMs in physical compatibility (virtual compatibility is compatible). If you use vMotion to migrate virtual machines with RDMs, make sure to maintain consistent LUN IDs for RDMs across all participating ESXi hosts
S ELECT
THE
P REFERRED P ATH
FOR A
VMFS DATASTORE
For each storage device, the ESXi host sets the path selection policy based on the claim rules. The different path policies we treated in our earlier chapter here - Configure vSphere Storage Multi-pathing and Failover. Now if you want just to select preferred path, you can do so. Ifyou want the host to use a particular preferred path, specify it manually. Fixed is the default policy for most active-active storage devices Fixed – (VMW_PSP_FIXED) the host uses designated preferred path if configured. If not it uses first working path discovered. Preffered path needs to be configured manually.
E NABLE/DISABLE V S TORAGE API
FOR
A RRAY INTEGRATION (VAAI)
You need to have hardware that supports the offloading storage operations like:
Cloning VMs Storage vMotion migrations Deploying VMs from templates VMFS locking and metadata operations Provisioning thick disks Enabling FT protected VMs
HOW TO DISABLE ? OR ENABLE ? Enable = 1 Disable = 0 vSphere Web Client > Manage tab > Settings > System, click Advanced System Settings > Change the value for any of the options to 0 (disabled):
VMFS3.HardwareAcceleratedLocking DataMover.HardwareAcceleratedMove DataMover.HardwareAcceleratedInit 90
you can check the status of the hardware via CLI (via esxcli storage core device vaai status get)
or on the NAS devices with (esxcli storage nfs list). Via vSphere web client you can also see if a datastore has hardware acceleration support...
91
DISABLE
A PATH TO A
VMFS DATASTORE
It's possible to temporarily disable storage path, for example for maintenance reasons. Check Storage Paths in the vSphere Storage Guide on p 192.
One can disable the path from through the web client from the datastore view OR storage device OR adapter view.
DETERMINE
USE CASE FOR MULTIPLE
VMFS/NFS DATASTORES
Usually the choice for multiple VMFS/NFS datastores are based on performance, capacity and data protection. Separate spindles – having different RAID groups to help provide better performance. Than you can have multiple VMs, executing applications which are I/O intensive. If you make a choice with single big datastore, than you might have performance issues... Separate RAID groups. – for certain applications, such as SQL server you may want to configure a different RAID configuration of the disks that the logs sit on and that the actual databases sit on. Redundancy – You might want to replicate VMs to another host/cluster. You may want the replicated VMs to be stored on different disks than the production VMs. In case you have failure on production disk system, you most likely still be running the secondary disk system just fine. Load balancing - you can balance performance/capacity across multiple datastores. Tiered Storage – Arrays comes often with Tier 1, Tier 2, Tier 3 and so you can place your VMs according to performance levels... Tools
vSphere Installation and Setup Guide
vSphere Storage Guide VMware vSphere® Storage APIs – Array Integration (VAAI) 92
VCP6-DCV OBJECTIVE 3.5 - SETUP AND CONFIGURE STORAGE I/O CONTROL This post will cover VCP6-DCV Objective 3.5 - Setup and Configure Storage I/O Control. Storage I/O is one of the features that are overlooked. But Storage I/O can "heal" part of your storage performance problems by setting a priority at the VM level (VMDK). You know the "noisy neighbor story".... When you enable Storage I/O Control on a datastore, ESXi host starts to monitor the device latency that hosts observe when communicating with that datastore. When device latency exceeds a threshold, the datastore is considered to be congested and each VM that accesses that datastore is allocated I/O resources in proportion to their shares. (by default all VMs are set to Normal (1000) You set shares per VMDK. You can adjust the number for each based on need. Default is 1000. I started to cover this VCP6-DCV exam blueprint since few weeks and It seems that for VCP6 there is more material to study and more topics to master than for previous version of VCP as the technology has evolving with each release of vSphere. But this never mind, we like technology, we like virtualization and we like VMware. Let's kick some tires.. -:) For whole exam coverage I created a dedicated VCP6-DCV page. VMware vSphere Knowledge
Enable/Disable Storage I/O Control Configure/Manage Storage I/O Control Monitor Storage I/O Control
E NABLE/DISABLE S TORAGE I/O CONTROL Before we jump in I'd like to explain how storage I/O control helps to prioritize certain VMs over the others. I think it's best to check out this image from VMware which shows basically that after activating the SIOC on shared datastore and setting up the shares (at the VMDK level) on the VMs properties, the SIOC is able to prioritize those VMs over the others....
Quote from VMware:
93
Storage I/O Control operates as a “datastore-wide disk scheduler.” Once Storage I/O Control has been enabled for a specific datastore, it will monitor that datastore, summing up the disk shares for each of the VMDK files on it. Storage I/O Control will then calculate the I/O slot entitlement per ESXi host based on the percentage of shares virtual machines running on that host have relative to the total shares for all hosts accessing that datastore.
Few limitations and requirements:
NFS v4.1 isn't supported (it is for NFS v3). Storage I/O Control does not support datastores with multiple extents. SAN with auto-tiering has to be certified for SIOC. Datastores that are Storage I/O Control-enabled must be managed by a single vCenter Server system. Must be disabled before removing a datastore. Raw Device Mapping (RDM) is not supported. (it is on iSCSI NFS and FC).
Storage I/O Requirements at the Online vSphere 6 documentation center. (here) Activate at the datastore level via vSphere client or vSphere Web client. Configure/Manage Storage I/O Control Configuring Storage I/O Control is a two-step process 1. Enable Storage I/O Control for the datastore In the vSphere Client > select a datastore > Configuration tab > Properties > Storage I/O Control, select the Enabled check box.
The advanced settings - Threshold - default value there. Check if the value is 30ms. 2. Set the number of storage I/O shares and upper limit of I/O operations per second (IOPS) allowed for each virtual machine. Those settings at the VMDK level so you could possibly prioritize disk where you important production DB sits! Set the threshold. More the VM is important, greater the number...... You can use the drop down or the custom and enter your value...
94
In case you're getting error on activating SIOC this can be due 2 reasons:
Not having proper licensing - Enterprise Plus is required. Storage I/O Control (SIOC) requires Enterprise Plus licensing. Without this license, the option to enable SIOC is grayed out Check that the host is installed with ESXi 4.1 or higher.
M ONITOR S TORAGE I/O CONTROL There is a Performance TAB to monitor Storage I/O. How Storage I/O Control handles the I/O workloads of the virtual machines accessing a datastore based on their shares. Datastore performance charts allow monitoring:
Average latency and aggregated IOPS on the datastore. Latency among hosts n Queue depth among hosts. Read/write IOPS among hosts. Read/write latency among virtual machine disks n Read/write IOPS among virtual machine disks.
95
W HERE ?
vSphere Web client > Datastore > Monitor tab > Performance tab > View drop-down menu > select Performance.
Tools
Administering VMware Virtual SAN vSphere Storage Guide vSphere Resource Management Guide vSphere Client / vSphere Web Client
Links: VCP6-DCV page.
VCP6-DCV OBJECTIVE 4.1 - PERFORM ESXI HOST AND VIRTUAL M ACHINE UPGRADES We will In no particular order start to cover VCP6-DCV sections to help out folks learning towards VCP6-DCV VMware certification exam. Due to VMware recertification policy the VCP exam has now an expiration date. You can renew by passing delta exam while still holding current VCP or pass VCAP. Today's topic of VCP6-DCV Objective 4.1 - Perform ESXi Host and Virtual Machine Upgrades.
96
For whole exam coverage I created a dedicated VCP6-DCV page. Or if you're not preparing to pass a VCP6-DCV, you might just want to look on some how-to, news, videos about vSphere 6 - check out my vSphere 6 page. We'll cover the topic today present on the VMware VCP6-DCV blueprint:
Identify upgrade requirements for ESXi hosts Upgrade a vSphere Distributed Switch Upgrade VMware Tools Upgrade Virtual Machine hardware Upgrade an ESXi Host using vCenter Update Manager Stage multiple ESXi Host upgrades Determine whether an in-place upgrade is appropriate in a given upgrade scenario
IDENTIFY
UPGRADE REQUIREMENTS FOR
ESX I
HOSTS
Even if this post we will now talk about the host requirement, the vCenter server shall be upgraded first.... Then you should definitely check the vSphere 6 Upgrade PDF from VMware which has all the details. ESXi 6 support booting via UEFI or BIOS, but if you plan to use autodeploy, then you might privilege BiOS as UEFI isn't supported. Changing from BIOS to UEFI after install isn't supported.
CPU with 2 cores at least VMware Supported Hardware via VMware HCL – http://www.vmware.com/go/hcl NX/XD bit enabled in the BIOS To support 64-bit virtual machines, support for hardware virtualization (Intel VT-x or AMD RVI) must be enabled on x64 CPUs. Note: for very old CPUs to check if they support x64 see this post - VMware Guest 64 Check – Free Utility To check if CPU can run 64 Bit Workflows Minimum of 4 GB of physical RAM, (if planning VSAN then at least 6Gb of RAM is required) with 8 GB of physical RAM recommended. For Serial ATA (SATA), a disk connected through supported SAS controllers or supported on-boardSATA controllers. SCSI disk or a local, non-network, RAID LUN with unpartitioned space for the virtual machines. At least 1 GbE Nic Minimum 1 GB boot device. But even if 1GB USB or SD device suffices for a minimal installation, you should use a 4GB or larger device where the extra space will be used for an expanded coredump partition on the USB/SD device.
UPGRADE
A V S PHERE
DISTRIBUTED S WITCH
The upgrade from 5.x to 6.0 is not reversible. There are two requirements: 1. You have upgraded your vCenter to vCenter 6.0 2. You have upgraded your hosts to ESXi 6.0 (check different methods of upgrading ESXi - via CLI or ISO, VUM, Online VMware repository) W HERE ? vSphere Web client > Networking > Right-click the distributed switch and select > Upgrade > Upgrade Distributed Switch It's non disruptive operation, so no downtime.
97
Check the the vSphere Networking Guide (page 28) for more.
UPGRADE VM WARE TOOLS VMware tools shall always have the latest version, but they depends on which vSphere Hardware Version (VHV) you run your VMs. You can very well have the virtual machine compatibility set for version 5.0 (vmx-09) because of some reasons and not the vmx-11 (vSphere 6). But the VMware tools will run being updated to the latest version for this VM compatibility.
vSphere Virtual Machine Administration Guide on page 22
UPGRADE V IRTUAL M ACHINE
HARDWARE
98
Where? Edit VM's Settings via vSphere web client.
Note that once you upgrade the Virtual machine hardware, there is no easy way back. There is three ways to downgrade virtual machine hardware version (supported by VMware). Upgrade an ESXi Host using vCenter Update Manager Note that only hosts running ESXi 5.0, ESXi 5.1, or ESXi 5.5 are directly upgradable to the ESXi 6.0. If you're still on 4.1 then you must first upgrade to 5.0. vCenter server 6 and vSphere Update Manager 6 (VUM) must be used for the upgrade. Details - vSphere Upgrade Guide (p. 135). 1. If you didn’t downloaded the ESXi 6.0 installation ISO, you’ll need to do so. Download Link. 2. You’ll need to install/configure VMware Update Manager – follow this guide. 3. Connect via vSphere client > select your host (or cluster) and go to the Update Manager TAB > Admin View > ESXi Images > Import ESXi Image
99
4. Follow with the assistant and create a new baseline (we have named it ESXi 6.0) > Change to Compliance View and Attach this new baseline > Scan > Remediate > Watch and wait till the server apply the upgrade and reboots the server. Stage multiple ESXi Host upgrades The same principle, but you selecting the host candidates for the upgrade at the cluster level (not at the host level). In case you’re applying the upgrade to a whole cluster you have other options, like deactivate DPN. But basically what’s happening is that host after host is patched and rebooted where the VMs residing on those hosts are "vMotioned" elsewhere before the patches are applied. Hosts that are part of VSAN cluster might need more time to evacuate VMs out as the local storage holding the VMDKs must shift some of those VMDKs elsewhere in order to be able to put the host into maintenance mode and launch the upgrade. 1 host at a time.
DETERMINE
WHETHER AN IN- PLACE UPGRADE IS APPROPRIATE IN A GIVEN UPGRADE SCENARIO
Upgrade using vSphere Update Manger (VUM) Interactive Upgrade from an ESXi image on a CD/DVD or USB flash drive Scripted Upgrade Using Auto Deploy and reboot it with a new image profile. Upgrade the ESXi via CLI (using SSH and Putty utility) by usigng esxcli software vib update -d - check this post.
Tools and Resources:
vSphere Upgrade Guide vSphere Virtual Machine Administration Guide vSphere Client / vSphere Web Client
VCP6-DCV OBJECTIVE 4.2 - PERFORM VCENTER SERVER UPGRADE In no particular we started to cover VCP6-DCV sections to help out folks learning towards VCP6-DCV VMware certification exam. Due to VMware re-certification policy the VCP exam has now an expiration date. You can renew by passing delta exam while still holding current VCP or pass VCAP. Today's topic VCP6-DCV Objective 4.2 - Perform vCenter Server Upgrade.
100
For whole exam coverage I created a dedicated VCP6-DCV page. Or if you're not preparing to pass a VCP6-DCV, you might just want to look on some how-to, news, videos about vSphere 6 - check out my vSphere 6 page. vSphere Knowledge covered in today's objective:
Identify steps required to upgrade a vSphere implementation Identify upgrade requirements for vCenter Upgrade vCenter Server Appliance (VCA) Identify the methods of upgrading vCenter Identify/troubleshoot vCenter upgrade errors
IDENTIFY
STEPS REQUIRED TO UPGRADE A V S PHERE IMPLEMENTATION
VMware recommends few steps before going straight to the upgrade. You should take few precautions, like backing up vCenter DB (or the whole VM). Also, depending of the vCenter installation (its size, if there are several sites, etc..) it's necessary to start the upgrade process by:
Read the release notes (what is and what is not supported) Verify that your system meets vSphere hardware and software requirements. Check the Update sequence for vSphere 6.0 and its compatible VMware products (2109760) Best Practices KB - Upgrading to vCenter Server 6.0 best practices (2109772) Verify compatibility of your Backup/replication/monitoring products (if ok Upgrade them Before you run the vCenter upgrade). THIS is my take on it. Because If you find yourself in trouble with vCenter upgrade process, you can always revert to the backup of your vCenter VM that you have done just before you started the upgrade process. With the latest release of the backup product... Check the VMware Product Interoperability Matrix in case you're using other VMware solutions (vCD, SRM, ....) Upgrade vCenter Server Upgrade vSphere Update Manager Upgrade ESXi hosts Upgrade VR, SRM... Apply vSphere 6 licensing Upgrade virtual hardware of your VMs and VM tools
IDENTIFY
UPGRADE REQUIREMENTS FOR V C ENTER
vCenter Server requires a 64-bit operating system, and the 64-bit system DSN is required for vCenter Serverto connect to the external database.
OS support:
2008 SP2 with latest patches and upgrades 2012R2
Internal or external DB
For environments with up to 20 hosts and 200 virtual machines, you can use the bundled PostgreSQL database. External DB support Oracle, Microsoft SQL. Check Interoperability Matrix!
101
For Windows - synchronize clocks on all machines running the vCenter Server 5.x services (if distributed). the vSphere Upgrade Guide (p. 30). If your vCenter Server service is running in a user account other than the Local System account, check that the account account in which the vCenter Server service is running is:
Member of the Administrators group Log on as a service Act as part of the operating system (if the user is a domain user) Verify that the LOCAL SERVICE account has read permission on the folder in which vCenter Server is installed and on the HKLM registry. Check that the connection between the virtual machine or physical server and the domain controller is working
L OGON AS A SERVICE [TIP ] - WHERE TO CHECK
The steps:
Click Start, point to Control Panel, point to Administrative Tools, and then double-click Local Security Policy. In the console tree, double-click Local Policies, and then click User Rights Assignment. In the details pane, double-click Log on as a service.
102
Click Add User or Group, and then add the appropriate account to the list of accounts that possess the Log on as a service right.
vCenter Requirements - Storage
vCenter Requirements - Hardware
Video upgrade 5.5 to 6.0: https://youtu.be/IRsa8a_YApk
UPGRADE V C ENTER S ERVER A PPLIANCE (VCA)
vCenter Server Appliance 5.1U3 and vCenter Server Appliance 5.5 can be upgraded to vCenter Server Appliance 6. (Not 5.1U2). VMware vCenter Server Appliance can be deployed only on hosts that are running ESXi version 5.0 or later. If an external vCenter SSO is used, check out the upgrade process here. The vCenter Server Appliance PostgreSQL database supports up to 1000 hosts and 10,000 virtual machines. An Oracle 11g database or an Oracle 12c database are the only external databases supported by the vCenter Server Appliance.
If you're plan using VUM: vSphere Update Manager also requires a supported database. Use separate databases for vCenter Server and vSphere Update Manager. Upgrade from VCSA 5.5 to VCSA 6.0 is not in place upgrade but rather side-by-side upgrade. We setup a new VCSA 6.0 appliance which will pull all configuration of the current environment from the old VCSA 5.5 appliance (including historical/performance data).
103
CHECK THIS BEFORE STARTING THE UPGRADE :
Do a backup or create a snapshot of your existing VCSA. You should check that the vCenter Server SSL certificate for the curent environment is valid and without missconfiguration. There is a VMware KB 2057223. Verify that the clocks of all machines on the vSphere network are synced. Synchronizing Clocks on the vSphere Network. Verify that the ESXi host on which you deploy the vCenter Server Appliance is not in lockdown or maintenance mode. In case you are on external database (SQL for example), make sure that you back it up. The upgrade outline can be found at the vSphere 6 documentation page
Here is what I’ve done to upgrade to the latest vSphere 6.0 vCenter (VCSA). After downloading the VCSA 6.0 iso image from VMware (The latest one is the VMware-VCSA-all-6.0.0-2562643.iso version), there is just very few steps to do: 1. Mount the ISO and go to the vcsa folder to install VMware Client Integration plugin.
2. Once done, double-click the vcsa-setup.html file located at the root of the DVD…
3. This bring the famous window offering you to do a clean install or an upgrade. You might have seen it in my detailed post about here.
104
You’ll get a nag telling you basically that you’ll have to be on VCSA 5.1 U3 or VCSA 5.5 in order to upgrade to VCSA 6.0… That’s the only options. If you’re on other version, you must first upgrade to those two supported ones…
VMware has a new KB article on the simple upgrade too. vCenter Applicance
Appliance Version vCenter Server IP or FQDN vCenter Administrator Username vCenter Administrator Password vCenter HTTPS Port Appliance Root password (when using https://vc-address:5480
Source ESXi Host
ESXi host IP or FQDN ESXi host username ESXi host password
Check ALL the steps for upgrade VCA in my Detailed Step-by-Step post here - How to Upgrade from VCSA 5.5 to 6.0 – Lab Time. Note that I run into a problem with default certificate (solved) during the upgrade. In case you're doing CLEAN install you might want to check scripted install guide of vCenter server appliance here. 105
IDENTIFY
THE METHODS OF UPGRADING V C ENTER
Embedded Deployment Model – The Platform Service Controller (PSC) and the vCenter Server are installed on the same machine. External Deployment Model – PSC is installed on a separate machine from the vCenter Server.
vCenter 5.5 and earlier deployed using Simple Install option will be upgraded to vCenter Server with embedded Platform Services Controller. If vCenter Single Sign-On was on a different machine than vCenter Server, the upgrade will be an external deployment model. If vCenter Single Sign-On was on the same node as vCenter Server, the upgrade will product an embedded deployment model. Upgrade external SSO servers to Platform Service Controllers, then upgrade vCenter Servers. Check also this:
List of recommended topologies for VMware vSphere 6.0.x (2108548) vCenter Server Example Upgrade Paths in the vSphere 6.0 Documentation Center.
Upgrade including an AutoDeploy Server (4) - the upgrade process upgrades it when upgrading the associated vCenter Server instance. Auto Deploy server included with an earlier version of the product cannot be used in conjunction with vCenter Server 6.0. If the Auto Deploy server is running on a remote system, it is upgraded and migrated to the same system as vCenter Server during the upgrade process. Settings are migrated to the new location. ESXi hosts must be reconfigured to point to the new Auto Deploy location.
106
Upgrading with Remote Web Client Server (5) - it is upgraded along with the vCenter Server instance to which it is registered and migrated to the same location as the vCenter Server instance.
IDENTIFY / TROUBLESHOOT V CENTER
UPGRADE ERRORS
Windows Based - Logs collection Via:
Installation wizard - browse the generated .zip file on the desktop Manually - navigate to
%PROGRAMDATA%VMwareCISlogs directory, usually C:ProgramDataVMwareCISlogs OR to Temp directory %TEMP% directory, usually C:UsersusernameAppDataLocalTemp Which files? vminst.log, pkgmgr.log, pkgmgr-comp-msi.log, and vim-vcs-msi.log For vCenter server appliance
via DCUI (Alt+F1)
pi shell to access the Bash shell
vc-support.sh This generates a .tgz archive in /var/tmp Export it with
scp /var/tmp/vc-etco-vm-vlan11-dhcp-63-151.eng.vmware.com-2014-02-28--21.11.tgz
[email protected]:/tmp And Determine which firstboot script failed.
cat /var/log/firstboot/firstbootStatus.json VMware Resources:
vSphere Installation and Setup Guide vSphere Upgrade Guide VMware vCenter Server™ 6.0 Deployment Guide Command-Line Installation and Upgrade of VMware vCenter Server 6.0 for Windows Command-Line Installation and Upgrade of VMware vCenter Server Appliance 6.0 vSphere Client / vSphere Web Client
107
VCP6-DCV OBJECTIVE 5.1 - CONFIGURE ADVANCED/M ULTILEVEL RESOURCE POOLS Today's VCP6-DCV topic will touch resource pools. Resource pools aren't folders, remember? Hey, resource pools are cool when used sparingly, not with 3 levels of inception... VCP6-DCV exam blueprint has this chapter about resource pools and it's important to know the insight out - VCP6-DCV Objective 5.1 - Configure Advanced/Multilevel Resource Pools. The whole exam details, and all topics from the blueprint can be found on the VCP6-DCV page. So in In today's topic we will learn about resource pools, but also there is a chapter about vFlash architecture. As you know vFRC caching has been here since vSphere 5.5 and it allows read-only caching mechanism to accelerate applications and VMs.
vSphere Knowledge
Describe the Resource Pool hierarchy Define the Expandable Reservation parameter Describe vFlash architecture Create/Remove a Resource Pool Configure Resource Pool attributes Add/Remove virtual machines from a Resource Pool Create/Delete vFlash Resource Pool Assign vFlash resources to VMDKs Determine Resource Pool requirements for a given vSphere implementation Evaluate appropriate shares, reservations and limits for a Resource Pool based on virtual machine workloads
DESCRIBE
THE
R ESOURCE P OOL
HIERARCHY
vSphere resource management p. 51. vSphere Resource pools can be grouped into hierarchies and used to hierarchically partition available CPU and memory resources. Resource pools always start at the root level. Each standalone host and DRS cluster has (invisible) root resource pool. You have to enable DRS first in order to create a resource pool. Note: DRS is available in vSphere Enterprise and Enterprise Plus editions. Resource Pools should be used when you would need to limit or to guarantee resources to VMs. By having resource pool you don't have to gurantee the resources to VMs individually, but only at the pool level.
Child resource pool - It's possible to create child resource pools under the root resource pool or under any other usercreated resource pool. Each child resource pool owns some of the parent's resources. Inside of each child resource pool it's possible create another resource pool. (Russian dolly like). Resource pool can contain:
108
Child resource pools VMs Both
Siblings - Resource pools and VMs at the same level are called siblings. Creating multiple RP allows you to aggregate computing capacity from the underlying hosts within the DRS cluster. You then can set resources for each resource pool instead on individual VMs. For each resource pool you specify reservation, limit, shares and you can also specify if the reservation shall be expandable.
DEFINE
THE
E XPANDABLE R ESERVATION
PARAMETER
Expandable Reservation parameter is a value allowing the resource pool resources became available to child resource pools and virtual machines.
If a VM’s workload increases and its resource pool cannot allocate more resources because there aren’t any available, the resource pool will asks its parent resource pool to borrow resources. Resource pools that have VMs and workloads varies, then you should possibly enable expandable reservations. When the check box is selected (default), expandable reservations are considered during admission control. If you power on a virtual machine in this resource pool, and the combined reservations of the virtual machines are larger than the reservation of the resource pool, the resource pool can use resources from its parent or ancestors.
DESCRIBE V F LASH
ARCHITECTURE
New version of VMware vSphere has introduced VMware vFlash Read cache, which enables you to use local SSD devices pooled together forming a pool of storage tier. The vFlash is integrated with vMotion, HA and DRS. The solution, the vFlash caching software, is tightly integrated into the hypevizor (placed into the data path), as an API, which is also available for third party caching modules. vFlash is a service within vSphere. 109
Flash Pooling as a resource pool:
vFlash will appear as a new type of resource pool No consumption when VM is powered Off vMotion and DRS can be used The allocation of resources is based per virtual object (VM, Host…)
The Flash Resource management uses:
Reservations, limits Uses per VMDK or per VM allocation (the config is at the VM level). Enforces admission control vFlash is a broker and manager for the entities which consumes the resources
V F LASH
R ELEASE 1.0
SUPPORTS
W RITE THROUGH CACHE (R EAD
ONLY )
The first release supports write through mode, which is read only. The write back mode will be available in future releases. It’s important to understand the the publicly available APIs gives opportunity to other storage companies to integrate their flash caching solution.
W HAT’ S
NEEDED FOR V FRC?
A configured of hosts with each one with at least one SSD or PCIe SSD… vSphere 5.5 (vCenter 5.5 and ESXi 5.5)
W HERE
TO START WITH V FRC?
At the cluster level. You have the choice. You can right-click the cluster >All vCenter actions > Add virtual flash resource capacity.
110
On the next screen you select available SSD from each ESXi host and click OK.
CREATE/R EMOVE
A
R ESOURCE P OOL
To be able to create Resource pool you must enable DRS. You can use both vSphere C# client or vSphere Web Client. (Web client) Select Hosts and clusters > Manage > vSphere DRS > Edit > Check the Turn ON.
111
Easiest way to create resource pool is perhaps the Right click at the cluster > New resource pool...
To delete, simple too. Right click the Resource Pool > Delete
CONFIGURE R ESOURCE P OOL
ATTRIBUTES
Navigate to the Host and Clusters view (View > Inventory > Hosts and Clusters) Right-click on the resource pool you want to edit and select Edit Settings… Change the name if desired Change the CPU Shares, Reservation, Expandable Reservation and Limit if desired Change the Memory Shares, Reservation, Expandable Reservation and Limit if desired
CPU R ESOURCES Shares - Specify shares for this resource pool with respect to the parent’s total resources. The amounts of shares you allocate to a resource pool are relative to the shares of any sibling (virtual machine or resource pool) and relative to its parent’s total resources. Sibling resource pools share resources according to their relative share values bounded by the reservation and limit. Different types of shares - Low (1), Normal (2), or High (4) which specify share values in a ratio. Or you can select Custom to give each RP a specific number of shares, which expresses a proportional weight.
112
Reservation - Specify a guaranteed CPU or memory allocation for this resource pool. Defaults to 0. A nonzero reservation is subtracted from the unreserved resources of the parent (host or resource pool). The resources are considered reserved, regardless of whether virtual machines are associated with the resource pool. Limit - Upper limit for this resource pool’s CPU allocation. Select Unlimited to specify no upper limit. Memory Resources Shares - Memory shares for this resource pool with respect to the parent’s total. Sibling resource pools share resources according to their relative share values bounded by the reservation and limit. Select Low (1), Normal (2), or High (4), which specify share values in a ratio. Select Custom to give each virtual machine a specific number of shares, which expresses a proportional weight. Reservation - Guaranteed memory allocation for this resource pool. Limit - Upper limit for this resource pool’s memory allocation. If you give RP limit 32Gb RAM it will never receive more RAM even if the host/cluster is able to allocate more. Select Unlimited to specify no upper limit.
A DD /R EMOVE
VIRTUAL MACHINES FROM A
R ESOURCE P OOL
No difficulties here. It's possible to use both clients. Drag and drop... -:)
113
Or when creating new VM, during the wizard creation you're asked whether you want to place the VM into specific resource pool... If the resource pool does not have enough resources to guarantee the virtual machine reservation(s) then the move into the resource pool will fail (for powered-on virtual machine). Create/Delete vFlash Resource Pool To delete RP, similar as creation. Do a right-click on the RP > delete. Drag-and-drop the virtual machine into another resource pool. You can also drag it into the root of the DRS cluster which will move it into the root resource pool.
A SSIGN V F LASH
RESOURCES TO
VMDK S
Once you've added a capacity to the cluster by providing some flash resources from each of the hosts present in the cluster, you can now add those ressources to individual VMs (or respectively the VMDKs).You can check the Flash Read cache resource availability through the Summary Tab.
114
You have to go and do it at the VM level (vFRC operates per VMDK). Select an individual VM and click edit settings > Virtual hardware tab >next to the Virtual Flash Read Cache > click Advanced.
Now can select the amount of GB (Mb) that will be reserved for that particular virtual hard drive. Depending of your workloads, because vFRC has variable block size capability (4kb – 1M). So the best selection will depends on your application, which IO size or your application which runs in your VM. (you can used for example vISCSI stats to find out) Then you carve up those information to match the block size of the vFlash to give the best possible performance. Not every node in the vFlash cluster needs to have SSD installed, but if that’s the case, the particular host won’t be able to provide any vFlash resources.
DETERMINE R ESOURCE P OOL
REQUIREMENTS FOR A GIVEN V S PHERE IMPLEMENTATION
It depends is a good answer... Before determining the requirements you'll need to determine the workloads that will be running in the environment and also priorities within the whole infrastructure. RP are here to help to segment the resources by organization, by workload or other business requirements. Once you have defined the workloads, you can start dividing up the resources pools the way it is able to meet the requirements of the workloads running on the DRS cluster. You should check whether the RP need to reach out to the parent RP to provide more resources -> configure expandable reservations. Check if you need reservations or limits. Do not use per-VM reservations as it's like if you would use per file NTFS permissions... [Administrative Overhead]. If you're using reservation then use it at the resource pool level.
115
E VALUATE
APPROPRIATE SHARES , RESERVATIONS AND LIMITS FOR A
RESOURCE P OOL
BASED ON
VIRTUAL MACHINE WORKLOADS
Know your workload first, then only you'll be able to define shares, reservations and (or) limits. We have talked about CPU shares, reservations, limits and Memory shares, reservations, limits in the chapter above. All the resources available within the cluster can be managed and distributed by Resource pools depending on how they're configured, but this determines the requirements. Note that limits is a resource limit and so it's not the same as if you were used shares which depends on other resources and their availability. Tools and links:
vSphere Resource Management Guide vSphere Virtual Machine Administration Guide What’s New in VMware vSphere® Flash Read Cache® vSphere Client / vSphere Web Client
VCP6-DCV OBJECTIVE 6.1 - CONFIGURE AND ADMINISTER A VSPHERE BACKUPS /RESTORE /REPLICATION SOLUTION VMware vSphere comes with free vSphere Data protection (VDP) product. There is no more VDP and VDP advanced as the VDP inherited all advanced features of VDP Advanced. So VDP is Advanced by Default. This post will cover VCP6-DCV Objective 6.1 - Configure and Administer a vSphere Backups/Restore/Replication Solution. It seems that for VCP6 there is more material to study and more topics to master. For whole exam coverage I created a dedicated VCP6-DCV page. Or if you're not preparing to pass a VCP6-DCV, you might just want to look on some how-to, news, videos about vSphere 6 - check out my vSphere 6 page. VMware vSphere Knowledge:
Identify snapshot requirements Identify VMware Data Protection requirements Explain VMware Data Protection sizing Guidelines Identify VMware Data Protection version offerings Describe vSphere Replication architecture Create/Delete/Consolidate virtual machine snapshots Install and Configure VMware Data Protection Create a backup job with VMware Data Protection Install/Configure/Upgrade vSphere Replication Configure VMware Certificate Authority (VMCA) integration with vSphere Replication Configure Replication for Single/Multiple VMs Identify vSphere Replication compression methods Recover a VM using vSphere Replication Perform a failback operation using vSphere Replication Determine appropriate backup solution for a given vSphere implementation
IDENTIFY
SNAPSHOT REQUIREMENTS
As you know vSphere FT VMs can now be protected (backed up) via backup solutions using snapshots. No manual snapshots for those VMs however as the snapshots are managed through an API calls only. VDP utilizes the Changed Block Tracking (CBT) greatly reducing the backup time of a VMs and so you can process much more VMs during your backup window than without using CBT. Note that CBT is also leveraged during restores
116
where the if restored to the original location, VDP can determine the missing blocks in the destination and only restore those. Not all the blocks. VDP leverages deduplication technology based on Avamar's code. Full VM recovery, File level recovery - both supported in VDP. vSphere data protection (VDP) and vSphere replication (VR) both uses snapshots on regular basis to protect VMs (or to replicate them). In the case of VR the RPO is as low as 15 min.
IDENTIFY VM WARE DATA P ROTECTION
REQUIREMENTS
Image level backups - vSphere Data Protection creates image‐level backups, which are integrated with the vStorage API for Data Protection, a feature set within vSphere to offload the backup processing overhead from the virtual machine to the VDP Appliance. The VDP Appliance communicates with the vCenter Server to make a snapshot of a virtual machine’s .vmdk files. Deduplication takes place within the appliance by using a patented variable‐length deduplication technology. Guest-level backup - VDP supports guest‐level backups for Microsoft SQL Servers, Exchange Servers, and Share Point Servers. With guest‐level backups, client agents (VMware VDP for SQL Server Client, VMware VDP for Exchange Server Client, or VMware VDP for SharePoint Server Client) are installed on the SQL Server, Exchange Server, or SharePoint Server in the same manner that backup agents are typically installed on physical servers VDP can not only protect VMs but also physical systems! - Microsoft Exchange, SQL Server, SharePoint when backed up by VDP, the agents which needs to get installed on those servers in order to protect them efficiently, are leveraged for granular restores. It does not have to be VMs to allow application level recovery.
E XPLAIN VMWARE DATA P ROTECTION
SIZING
G UIDELINES
vSphere web client is necessary for deployment and administration of the VDP, which can be deployed on practically any storage (VMFS, NFS or VSAN). vSphere Data Protection 6.0 Administration Guide p.20 CAPACITY R EQUIREMENTS :
Up to 20 VDP appliances per vCenter server Each appliance can protect up to 400 VMs 8TB of deduplicated backups
S IZING DEPENDS ON FOLLOWING FACTORS :
Types of data being backed up (files, DB, OS files) Data change rate Size of protected VMs and their numbers Retention period (daily, weekly, monthly or yearly) Deployment availability of VDP as 0.5TB, 1Tb, 2Tb, 4Tb, 6Tb, 8Tb (if deployed small size, can be increased later).
vSphere data protection 6.0 administration guide p. 21
117
S OFTWARE R EQUIREMENTS :
Minium requirements is vCenter 5.1 to install VDP 6, but 5.5 or higher is recommended. VDP 6 supports vCSA and Windows based vCenters vSphere Web client where browsers needs Flash player 11.3 or above version installed. NOTE: VDP do not support of backup of vCenter server appliance (VCSA) itself. VMs to be protected must be on virtual hardware version 7 or higher (CBT) and VMware tools installed. VDP repository usually fills rapidly for the first few weeks. This is because nearly every client that is backed up contains unique data. But then VDP deduplication allows to save space when other similar clients have been backed up, or the same clients have been backed up at least once.
U NSUPPORTED VM S DISKS :
Independent RDM Independent - virtual compatibility mode RDM w. physical compatibility mode
IDENTIFY VM WARE DATA P ROTECTION
VERSION OFFERINGS
I guess this is a bit erroneous topic as the VDP is by default now a VDP Advanced. However I think it's worth to know that in the past there were two versions: VDP and VDP advanced. VDP allows:
Disk level Granularity – allows backup/restore individual VMDK (virtual disks). Restore directly with ESXi (if vCenter is not available) – by going to https:///vdp-configure you can access to Emergency restore tab where you can trigger restores. Detachable/remountable data partitions – for DR scenarios of VDP Replication to the cloud – off site backups Time-of-day scheduling – schedule backup to be triggered exactly when you want Removal of the blackout window
VDP has also:
Application-level replication Ability to expand current datastore Backup to a Data Domain system Ability to restore to a granular level on Microsoft Servers and automatic backup verification. VDP also supports guest-level backups and restores of Microsoft SQL Servers, Exchange Servers, and Share Point Servers, providing for application consistent backups of these servers.
A migration tool is included with VDP 5.1.10 and later releases. This tool handles migration of data and restore points. Backup jobs cannot be migrated.
DESCRIBE V S PHERE R EPLICATION
ARCHITECTURE
118
vSphere replication is separate product included in vSphere. It allows to configure replication of VMs from source site to target site. It uses snapshots (points-in-time) to transfer delta informations to the other side. Types of replication:
Within single site - from one cluster to another From multiple source sites - to shared remote site From source site to target site
vCenter server (Windows) or VCSA can be used. Possibility to deploy additional VR servers to enhance. VMware VSAN is supported as target (destination) datastore. A RCHITECTURE :
The vSphere replication appliance contains the following:
vCenter Plugin for vSphere web client. An embedded database storing replication config and management information. vSphere Replication management server - configures vSphere replication server, enables, manages, monitors replication and also authenticate users and check their permissions for VR operations. vSphere Replication Server - provide the core of VR infra.
Below example of architecture with single vCenter server and single site (possible also multi-site to shared location or two sites in between).
From the network perspective it's necessary to setup vmkernel adapter per ESXi host which is used as a replication source, for isolation of the replication traffic.
CREATE/DELETE/CONSOLIDATE
VIRTUAL MACHINE SNAPSHOTS
To create a VM snapshot. Two ways possible (vSphere client or vSphere web client). Select VM > take snapshot of this virtual machine. Delete snapshot - via snapshot manager > delete
119
Consolidate VM snapshots - if any VM that shows that needs to consolidate, just select and right click that particular VM and choose Consolidate. Right click > Shapshot > Consolidate
INSTALL
AND
C ONFIGURE VM WARE DATA P ROTECTION
VDP is VSA based (Linux). The deployment as an OVF is fast and convenient.
Requirements:
NTP - All vSphere hosts and the vCenter Server must have NTP configured properly. The VDP Appliance gets the correct time through vSphere and must not be configured with NTP. DNS - create DNS forward and reverse record and check that you have vCenter server responding via nslookup.
Deploy the OVF file via vSphere Web client to a VMFS5 datastore (to avoid block size limitations). After the deployment and start up of the VM go to the IP address precised on the console. https://ip_of_vdp:8543/vdp-configure Login: pass: changeme
root
Follow the assistant, you should have the info pre-filled when you click the next button... 120
continue with the wizard. Test your connection to vCenter to avoid issues...
Create storage. Here you can (but don't have to) check the box "store with appliance" in case you have enough space on the shared storage datastore you have chosen.
121
Continue with the assistant until the end. After the setup finished the appliance will reboot...
122
It takes up to 15 min to fully setup after the reboot... -:) You'll have to log off and log in back again through vSphere web client to see this new plugin to appear..
CREATE
A BACKUP JOB WITH
VM WARE DATA P ROTECTION
To create a first backup job, just click through the new icon on the dashboard in vSphere web client.
Then start an assistant...
123
continue..
Choose a VM(s)...
124
Backup schedule...
Specify retention policy.... Note that this can be changed later. (Think of sizing).
Give the job some meaningful name...
And off you go. 125
Just created first backup job. If you go and click the Configuration TAB, then down there you can configure the Backup window configuration... If not the default backup starts at 8PM...
INSTALL/C ONFIGURE/UPGRADE V S PHERE R EPLICATION vSphere Replication is distributed as ISO. Mount the ISO to access the OVF file to be deployed. Requirements:
Source and target site must have vSphere web client and the client integration plugin is installed as well Select the vCenter Server instance on which you are deploying vSphere Replication, click Manage > Settings > Advanced Settings, and verify that the VirtualCenter.FQDN value is set to a fully-qualified domain name or a literal address
Network ports - For a list of all the ports that must be open for vSphere Replication, see http://kb.vmware.com/kb/2087769 Bandwidth - vSphere Replication transfers blocks based on the RPO schedule. If you set an RPO of one hour, vSphere Replication transfers any block that has changed in that hour to meet that RPO. vSphere Replication only transfers the block once in its current state at the moment that vSphere Replication creates the bundle of blocks for transfer. vSphere Replication only registers that the block has changed within the RPO period, not how many times it changed
V S PHERE
R EPLICATION DEPLOYMEN T
vSphere Replication 6.0 administration guide p. 31 Select cluster and then Actions > deploy OVF template > local file > browse... and so on... If you don't want to relay on the DHCP you can use fixed IP.... Select a network from the list of available networks, set the IP protocol and IP allocation, and click Next. vSphere Replication supports both DHCP and static IP addresses. You can also change network settings by using the virtual appliance management interface (VAMI) after installation.
126
And then
Once done. Log off and log back again to see the VR plugin
127
CONFIGURE VM WARE CERTIFICATE A UTHORITY (VMCA) R EPLICATION
INTEGRATION WITH V SPHERE
You can change the SSL certificate, for example if your company's security policy requires that you use trust by validity and thumbprint or a certificate signed by a certification authority. You change the certificate by using the virtual appliance management interface (VAMI) of the vSphere Replication appliance. For information about the SSL certificates that vSphere Replication uses, see “vSphere Replication Certificate Verification,” on page 45 and “Requirements When Using a Public Key Certificate with vSphere Replication,” on page 46.
128
CONFIGURE R EPLICATION
FOR
S INGLE/M ULTIPLE VM S
Before this, make sure that you have the permissions. Step 1: Select VM(s) > Right click > All vSphere Replication Actions > configure Replication Now if you haven't restarted the vCenter service, you see this (1), because after restart you should see this (2). Also, you'll get some error on the permissions if you don't restart, and so you won't be able to configure the replication for your VMs. That "from the field" experience ...
Step 2: Replicate to a vCenter server (or service provider) > select target site > target location...
129
And enable compression...
Step 3: You can change the RPO settings and enable the Point in time instances on this screen...
IDENTIFY V S PHERE R EPLICATION
COMPRESSION METHODS
vSphere Replication 6.0 administration guide p. 16. The compression settings depends on the version of VR and version of ESXi at the destination. But basically if source or destination has earlier than ESXi 6.0 and VR earlier than 6.0 the compression is not used. But what's interesting is the fact that if compression is enabled. Quick quote: However, if the target ESXi host is earlier than 6.0,vSphere Replication prevents vMotion from moving replication source VMs to that host because it does notsupport data compression. This prevents DRS from performing automated vMotion operations to hosts thatdo not support compression. Therefore, if you need to move a replication source VM to an ESXi host earlier than 6.0, before you perform the vMotion operation, you must reconfigure the replication to disable data compression.
R ECOVER
A
VM
USING V S PHERE
R EPLICATION
vSphere Replication 6.0 administration guide p. 77. With Sphere Replication, you can recover virtual machines that were successfully replicated at the target site. You can recover one virtual machine at a time.
130
Web client > vSphere replication > Home tab > Monitor > Incoming replication
From there you have two options: 1. Recover with recent changes - Performs a full synchronization of the virtual machine from the source site to the target site before recovering the virtual machine. Selecting this option avoids data loss, but it is only available if the data of the source virtual machine is accessible. You can only select this option if the virtual machine is powered off. 2. Recover with latest available data - Recovers the virtual machine by using the data from the most recent replication on the target site, without performing synchronization. Selecting this option results in the loss of any data that has changed since the most recent replication. Select this option if the source virtual machine is inaccessible or if its disks are corrupted.
You continue and select folder where you want to recover the VM...
P ERFORM
A FAILBACK OPERATION USING V S PHERE
R EPLICATION
vSphere Replication 6.0 administration guide p. 79. Failback is manual, it means that after performing a successful recovery on the target vCenter Server site, you can perform failback. You log in to the target site and manually configure a new replication in the reverse direction, from the target site to the source site. The disks on the source site are used as replication seeds, so that vSphere Replication only synchronizes the changes made to the disk files on the target site. Before you configure a reverse replication, you must unregister the virtual machine from the inventory on the source site.
DETERMINE
APPROPRIATE BACKUP SOLUTION FOR A GIVEN V S PHERE IMPLEMENTATION
131
Depending on your needs it's necessary to size accordingly your backup solution. You must take into account the daily delta changes within your all environment and see if the product you want to use as a backup solution is suitable. How it scale? What's the limitations? You must also take into account the possible conflicts with other vSphere products you may be using (vSphere replication, SRM, vCD....). If you're planning to use VDP, than you should certainly check vSphere compatibility matrix. Tools:
VMware vSphere® Data Protection™ 6.0 vSphere Data Protection Administration Guide VMware vSphere® Data Protection™ Evaluation Guide What’s New in the VMware vSphere® 6.0 Platform VMware vSphere Replication Administration VDR Data Migration Tool VDP Configure Utility vSphere Client / vSphere Web Client
VCP6-DCV OBJECTIVE 7.1 - TROUBLESHOOT VCENTER SERVER , ESXI HOSTS , AND VIRTUAL M ACHINES In today's Objective we'll discuss VCP6-DCV Objective 7.1 - Troubleshoot vCenter Server, ESXi Hosts, and Virtual Machines. You can check the whole VCP6-DCV Study Guide page for all topics there. You can also check the vSphere 6 page where you’ll find many how-to, videos, and tutorials about vSphere 6. Another troubleshooting chapter today. After we cracked the troubleshooting of vSphere upgrades, in another troubleshooting chapter we hit the storage and network issues, today we'll hit the Toubleshooting of vCenter, ESXi and VMs. When something goes wrong with vCenter, only things that rely on vCenter does suffer. Things like HA, DRS or FT continues to work, but you can't manually vMotion a VM if you don't have an access to vCenter. It can be that one of the vCenter services went down or something like that. Today well' have a look at those different things which can happened.
vSphere Knowledge
Identify general ESXi host troubleshooting Guidelines Identify general vCenter troubleshooting Guidelines Troubleshoot Platform Services Controller (PSC) issues Troubleshoot common installation issues Monitor ESXi system health Locate and analyze vCenter and ESXi logs Export diagnostic information Identify common Command Line Interface (CLI) commands Troubleshoot common virtual machine issues Troubleshoot virtual machine resource contention issues Identify Fault Tolerant network latency issues Troubleshoot VMware Tools installation issues Identify/Troubleshoot virtual machines various states (e.g. orphaned, unknown, etc.) Identify virtual machine constraints Identify the root cause of a storage issue based on troubleshooting information Identify common virtual machine boot disk errors 132
Identify and detect common knowledge base article solutions
IDENTIFY
GENERAL
ESX I
HOST TROUBLESHOOTING
G UIDELINES
When starting troubleshooting, you should first:
Identify symptoms - WTF? .... is going on? Define problem space - software? Hardware? What is causing the problem? What's excluded? Test solutions - Once knwing the symptoms and problem space, you can test solutions, one by one until problem resolved.
check vSphere 6 troubleshooting guide p.7 and onward...
IDENTIFY
GENERAL V C ENTER TROUBLESHOOTING
G UIDELINES
Few good troubleshooting scenarios is in the vSphere 6 troubleshooting guide p.33 You'll find problems (and their resolution) like those one below:
vCenter Server Upgrade Fails When Unable to Stop Tomcat Service Microsoft SQL Database Set to Unsupported Compatibility Mode Causes vCenter Server Installation or Upgrade to Fail Error When You Change vCenter Server Appliance Host Name vCenter Server System Does Not Appear in vSphere Web Client Inventory Unable to Start the Virtual Machine Console Unable to View the Alarm Definitions Tab of a Data Center vCenter Server Cannot Connect to the Database vCenter Server Cannot Connect to Managed Hosts
TROUBLESHOOT P LATFORM S ERVICES CONTROLLER (PSC)
ISSUES
PSC logs location and names:
cis-license - VMware Licensing Service SSO - VMware Secure Token Service VMCA - VMware Certificate Service vmdird - VMware Directory Service
For Platform Services Controller node deployments, additional runtime logs are located at C:\ProgramData\VMware\CIS\runtime\VMwareSTSService\logs including logs for these services:
VMware Secure Token Service VMware Identity Management Service
TROUBLESHOOT
COMMON INSTALLATION ISSUES
Recursive panic might occur when using ESXi Dump Collector - PSOD. Check release notes.
vSphere installation guide p.245
133
V C ENTER SERVER ON
W INDOWS
Collect Installation Logs by Using the Installation Wizard - You can use the Setup Interrupted page of the installation wizard to browse to the generated .zip file of the vCenter Server for Windows installation log files. If the installation fails, the Setup Interrupted page appears with the log collection check boxes selected by default.
The installation files are collected in a .zip file on your desktop, for example, VMware-VCS-logs-time-of-installationattempt.zip You can then unzip the log file located on your desktop and start checking what's wrong. Manual retrieve of logs: C:\ProgramData\VMware\vCenterServer\logs C:\Users\username\AppData\Local\Temp The files in the %TEMP% directory include vminst.log, pkgmgr.log, pkgmgr-comp-msi.log, and vim-vcs-msi.log V C ENTER
A PPLIANCE The full path to the log files is displayed in the vCenter Server Appliance deployment wizard. 1. Log in to the Windows host machine on which you want to download the bundle. 2. Open a Web browser and enter the URL to the support bundle displayed in the DCUI. https://appliance-fully-qualified-domain-name:443/appliance/support-bundle 3. Enter the user name and password of the root user. 4. Click Enter > The support bundle is downloaded as .tgz file on your Windows machine. 5. (Optional) To determine which firstboot script failed, examine the firstbootStatus.json file. If you ran the vc-support.sh script in the vCenter Server Appliance Bash shell, to examine the firstbootStatus.json file, run cat /var/log/firstboot/firstbootStatus.json
Attempt to Install a Platform Services Controller After a Prior Installation Failure Collect Installation Logs by Using the Installation Wizard.
M ONITOR ESX I
SYSTEM HEALTH
Hardware Monitoring on ESXi - The Common Information Model (CIM) is used on ESXi instead of installing the hardware agents in the Service Console. The different CIM providers are available for different hardware installed in the server (HBA, Network cards, Raid Controllers etc). [source...] If connected through vCenter:
134
OR, If connected directly to the ESXi host:
LOCATE
AND ANALYZE V C ENTER AND
ESX I
LOGS
VMware KB - Location of log files for VMware products (1021806) Export diagnostic information Create a Log Bundle (via Web client) Locate/Analyze VMware Log Bundles To collect ESX/ESXi and vCenter Server diagnostic data: 1. Start the vSphere Web Client and log in to the vCenter Server system. 2. Under Inventory Lists, select vCenter Servers. 3. Click the vCenter Server that contains the ESX/ESXi hosts from which you want to export logs. 135
4. Click the Monitor tab and click System Logs. 5. Click Export System Logs.
1. 2. 3. 4. 5.
Select the ESX/ESXi hosts from which you want to export logs. Select the Include vCenter Server and vSphere Web Client logs option. This step is optional. Click Next. Select the system logs that are to be exported. Select Gather performance data to include performance data information in the log files.Note: You can update the duration and interval time between which you want to collect the data. 6. Click Next. 7. Click Generate Log Bundle. The Download Log Bundles dialog appears when the Generating Diagnostic Bundle task completes.
1. Click Download Log Bundle to save it to your local computer.Note: The host or vCenter Server generates .zip bundles containing the log files. The Recent Tasks panel shows the Generate diagnostic bundles task in progress.
TO
EXPORT THE EVENTS LOG :
1. 2. 3. 4.
Select an inventory object. Click the Monitor tab, and click Events. Click the Export icon. In the Export Events window, specify what types of event information you want to export. 136
5. Click Generate CSV Report, and click Save.
Same covered in VCP6-DCV Objective 7.3 – Troubleshoot vSphere Upgrades.
IDENTIFY
COMMON
C OMMAND LINE I NTERFACE (CLI)
COMMANDS
Cli commands. Depending what you want to do, which part of the infrastructure you targetting:
vmkping - simple ping via vmkernel interface (ex. How-to troubleshoot iSCSI connection to your SAN ) vmkfstools - works with VMFS volumes, VMDKs ... (ex Recreate a missing VMDK header file ) esxcli network - ( ex. How to create custom ESXi Firewall rule ) esxcli storage - ( ex. How to tag disk as SSD VMware esxi 5.x and 6.0 ) esxtop - performance monitoring - (ex. How-to check Queue Depth Of Storage Adapter or Storage Device )
TROUBLESHOOT COMMON VIRTUAL MACHINE ISSUES TROUBLESHOOT VIRTUAL MACHINE RESOURCE CONTENTION IDENTIFY F AULT TOLERANT NETWORK LATENCY ISSUES
ISSUES
For FT you'll need 10GbE pipe. That's a fact. vSphere 6 Features - New Config Maximums, Long Distance vMotion and FT for 4vCPUs.
TROUBLESHOOT VM WARE TOOLS
INSTALLATION ISSUES
VMware KB Article 1003908 – Troubleshooting a Failed VMware Tools Installation in a Guest Operating System. How to remove VMware Tools manually if uninstall or upgrade finish with error Manual Download of VMware Tools from VMware Website
IDENTIFY /TROUBLESHOOT ETC .)
VIRTUAL MACHINES VARIOUS STATES
( E. G .
ORPHANED , UNKNOWN,
A virtual machine is deleted outside of vCenter Server - A user can delete a virtual machine through the VMware Management Interface while vCenter Server is down, through the vSphere Client directly connected to an ESX/ESXi host, or by deleting the virtual machine's configuration file through the service console. These virtual machines can be removed from the vCenter Server by right-clicking the virtual machine and selecting delete
Virtual machines appear as invalid or orphaned in vCenter Server (1003742)
137
IDENTIFY
VIRTUAL MACHINE CONSTRAINTS
VMware KB Article 1008360 – Troubleshooting Virtual Machine Performance Issues Troubleshooting a virtual machine that has stopped responding: VMM and Guest CPU usage comparison (1017926) VMware KB Article 2001003 – Troubleshooting ESX/ESXi Virtual Machine Performance Issues
IDENTIFY
THE ROOT CAUSE OF A STORAGE ISSUE BASED ON TROUBLESHOOTING INFORMATION
Often the root cause is storage. We all know that spinning media are slowly replaced by SSDs, but they still have some years to come. Storage contention happens when the demand of hosts for IOs exceeds the the storage and hba(s). The contention can happens at the VM level, HBA level or at the arrray level. ESXTOP: davg – average response time for a command which are sent to the device. kavg – average response time a command is in the vmkernel gavg – response time as it appears to the VM. (davg + kavg). CMD/s – number of IOps sent or received from the device or the VM
IDENTIFY
IDENTIFY
COMMON VIRTUAL MACHINE BOOT DISK ERRORS
kb.vmware.com/kb/1006296 - Cannot boot or start a virtual machine converted by VMware vCenter Converter 4.x/5.x (1006296) Identifying critical Guest OS failures within virtual machines AND DETECT COMMON KNOWLEDGE BASE ARTICLE SOLUTIONS
KB 2000988 – Troubleshooting vSphere Auto Deploy KB 653 – Collecting Diagnostic Information for VMware ESX/ESXi KB 1008360 – Troubleshooting Virtual Machine Performance Issues KB 2001003 – Troubleshooting ESX/ESXi Virtual Machine Performance Issues KB 1003908 – Troubleshooting a Failed VMware Tools Installation in a Guest Operating System KB 1003999 – Identifying Critical Guest OS Failures Within Virtual Machines.
Tools used for this Objective
vSphere Installation and Setup Guide vSphere Troubleshooting Guide vSphere Virtual Machine Administration Guide vSphere Server and Host Management Guide vSphere Monitoring and Performance Guide vSphere Security Guide vSphere Client / vSphere Web Client
138
VCP6-DCV OBJECTIVE 7.2 - TROUBLESHOOT VSPHERE STORAGE AND NETWORK ISSUES Today's topic of VCP6-DCV Study Guide is touching troubleshooting. In case something goes wrong and you loose connectivity to your application, you must probably troubleshoot the underlying VM first, the network second, but also a storage. When storage is under a pressure then your whole infrastructure just slows down and you might experience disconnections at the VM/application level. VCP6-DCV Objective 7.2 - Troubleshoot vSphere Storage and Network Issues is today's lesson. You can also check vSphere 6 page where you'll find how-to's, news, videos concerning vSphere 6.x. Last but not least, my Free Tools page where are the post popular tools for VMware and Microsoft. Daily updates of the blog are taking time, but we do it in the goal to provide a guide which is helpful for the community and folks learning towards VCP6-DCV certification exam. If you find one of those posts useful for your preparation, just share.. -:).
vSphere Knowledge
Verify network configuration Verify storage configuration Troubleshoot common storage issues Troubleshoot common network issues Verify a given virtual machine is configured with the correct network resources Troubleshoot virtual switch and port group configuration issues Troubleshoot physical network adapter configuration issues Troubleshoot VMFS metadata consistency Identify Storage I/O constraints Monitor/Troubleshoot Storage Distributed Resource Scheduler (SDRS) issues
V ERIFY
NETWORK CONFIGURATION
Start from one end. Either from the host level > physical switch > uplinks > switches > port groups > VMs
Check the vNIC status - connected/disconnected Check the networking config inside Guest OS - yes it might also be one of the issues. Bad network config of the networking inside of a VM. Verify physical switch config Check the vSwitch or vDS config ESXi host network (uplinks)
Guest OS config Check for disabled/inactive adapters or other unused hardware (if Guest OS has been P2V) In Windows VM do this: Click on Start > Run > devmgmt.msc > click + next to network adapters > check if it's not disabled or not present You can also check the network config like IP address, Netmask, default gateway and DNS servers. Make sure that those informations are correct.
If a VM was P2V - check if there are no "ghosted adapters". To check that:
On your VM go to Start > RUN > CMD > Enter > Type “
set devmgr_show_nonpresent_devices=1 139
While still in the command prompt window type:
devmgmt.msc and then open Device Manager and click on the Menu go to View > Show Hidden Devices (like on the pic).
Then you should see which devices are marked like ghosted devices.They are grayed out. Those devices you can safely remove from the device manager.
Check IP stack - It happened to me several times that the IP stack of a VM was corrupted. The VM has had intermittent networking connectivity, everything seems to be ok but isn't. You can clear the local cache by entering this:
ipconfig /renew For Linux:
dhclient dhclient eth0 V ERIFY
-r
STORAGE CONFIGURATION
Check the documentation of vSphere storage, the basic concepts, iSCSI etc. I've done few posts in configuring iSCSI and vSphere (not particulary related to vSphere 6 but those are step-by-steps:
How to configure FreeNAS 8 for iSCSI and connect to ESX(i) How to configure ESXi 5 for iSCSI connection to Drobo Configuring iSCSI port binding with multiple NICs in one vSwitch for VMware ESXi 5.x and 6.0.x
Also check this VMware KB for Teaming and Failover Policy section in the vSphere Networking guide.
140
TROUBLESHOOT
COMMON STORAGE ISSUES
Storage Issues - Check that the virtual machine has no underlying issues with storage or it is not experiencing resource contention, as this might result in networking issues with the virtual machine. You can do this by logging into ESX/ESXi or Virtual Center/vCenter Server using the VI/vSphere Client and logging into the virtual machine console. Good doc - Troubleshooting Storage guide (p.55 - p.70) which talks about:
Resolving SAN Storage Display Problems - page 56 Resolving SAN Performance Problems on page 57 Virtual Machines with RDMs Need to Ignore SCSI INQUIRY Cache on page 62 Software iSCSI Adapter Is Enabled When Not Needed on page 62 Failure to Mount NFS Datastores on page 63 VMkernel Log Files Contain SCSI Sense Codes on page 63 Troubleshooting Storage Adapters on page 64 Checking Metadata Consistency with VOMA on page 64 Troubleshooting Flash Devices on page 66 Troubleshooting Virtual SAN on page 69 Troubleshooting Virtual Volumes on page 70
TROUBLESHOOT
COMMON NETWORK ISSUES
Again, networking can be tricky to troubleshoot. But choosing one end to start with should help. Another tip is perhaps to check load balancing policies when more than 1 nic is connected to a VM. Verify that the virtual machine is configured with two vNICs to eliminate a NIC or a physical configuration issue. To isolate a possible issue:
If the load balancing policy is set to Default Virtual Port ID at the vSwitch or vDS level: o Leave one vNIC connected with one uplink on the vSwitch or vDS, then try different vNIC and pNIC combinations until you determine which virtual machine is losing connectivity. If the load balancing policy is set to IP Hash: a. Ensure the physical switch ports are configured as port-channel. For more information on verifying the configuration on the physical switch, see Sample configuration of EtherChannel / Link aggregation with ESX/ESXi and Cisco/HP switches (1004048). b. Shut down all but one of the physical ports the NICs are connected to, and toggle this between all the ports by keeping only one port connected at a time. Take note of the port/NIC combination where the virtual machines lose network connectivity. Load balancing and failover policies - configure VM with 2 vNICs to eliminate physical NIC problems. Check esxtop using the n option (for networking) to see which pNIC the virtual machine is using. Try shutting down the ports on the physical switch one at at time to determine where the virtual machine is losing network connectivity. Check the vNIC's connection - check the status of the vNIC, (connected/disconnected) at the VM level AND also the NIC inside of the Guest OS (activated/deactivated).
Check more in this KB: Troubleshooting virtual machine network connection issues (1003893)
V ERIFY
A GIVEN VIRTUAL MACHINE IS CONFIGURED WITH THE CORRECT NETWORK RESOURCES
I've invoked few areas already above. All or most of the possible problems can be found in this KB - KB 1003893
TROUBLESHOOT
VIRTUAL SWITCH AND PORT GROUP CONFIGURATION ISSUES
141
Same name for port groups - Make sure that the Port Group name(s) associated with the virtual machine's network adapter(s) exists in your vSwitch or Virtual Distributed Switch and is spelled correctly. Usually if this isn't done right on per-port group then you have connectivity problems
VLANs - check VLANS on each standard switch
TROUBLESHOOT
PHYSICAL NETWORK ADAPTER CONFIGURATION ISSUES
Physical switch config is usually simple if "trunking" ports are used. Perhaps some of the issues might be if vNICs are not set to automatic (default) but fixed network speed, which do not match the speed of the physical switch... I doubt it... If beacon probing is used, make sure that you have more than 2 pNICs in the team.... VMware KBs:
1005577 - What is beacon probing? 1004048 - Sample configuration of EtherChannel / Link Aggregation Control Protocol (LACP) with ESXi/ESX and Cisco/HP switches (1004048) 1001938 - Host requirements for link aggregation for ESXi and ESX
TROUBLESHOOT VMFS
METADATA CONSISTENCY
There is a VMware KB which explains what to do if:
You have problems accessing certain files on a VMFS datastore. You cannot modify or erase files on a VMFS datastore. Attempting to read files on a VMFS datastore may fail with the error:
invalid argument You can run file system metadata check by using VOMA. Check it out - Using vSphere On-disk Metadata Analyzer (VOMA) to check VMFS metadata consistency (2036767) Quote:
To perform a VOMA check on a VMFS datastore and send the results to a specific log file, the command syntax is: voma -m vmfs -d /vmfs/devices/disks/naa.00000000000000000000000000:1 -s /tmp/analysis.txt where naa.00000000000000000000000000:1 is replaced with the LUN NAA ID and partition to be checked. Note the ":1" at the end. This is the partition number containing the datastore and must be specified. See note below. As an advisory, if you run voma more than once, add the NAA ID and a time stamp to the output log file name. EG: -s /tmp/naa.00000000000000000000000000:1_analysis_.txt Note: VOMA must be run against the partition and not the device.
IDENTIFY S TORAGE I/O
CONSTRAINTS
Again, Good KB article to check - VMware KB 1008205. Per LUN basis - To monitor storage performance on a per-LUN basis:
Start esxtop > Press u to switch to disk view (LUN mode). 142
Press f to modify the fields that are displayed. Press b, c, f, and h to toggle the fields and press Enter. Press s and then 2 to alter the update time to every 2 seconds and press Enter.
Per HBA - To monitor storage performance on a per-HBA basis:
Start esxtop by typing esxtop > Press d to switch to disk view (HBA mode). To view the entire Device name, press SHIFT + L and enter 36 in Change the name field size. Press f to modify the fields that are displayed. Press b, c, d, e, h, and j to toggle the fields and press Enter. Press s and then 2 to alter the update time to every 2 seconds and press Enter.
Then the metrics to check out:
GAVG, DAVG, KAVG - latency stats. You should check this community thread from which I quote the main part because I think that it's a very good work done by the community: Latency values are reported for all IOs, read IOs and all write IOs. All values are averages over the measurement interval. All IOs: KAVG/cmd, DAVG/cmd, GAVG/cmd, QAVG/cmd Read IOs: KAVG/rd, DAVG/rd, GAVG/rd, QAVG/rd Write IOs: KAVG/wr, DAVG/wr, GAVG/wr, QAVG/wr GAVG - This is the round-trip latency that the guest sees for all IO requests sent to the virtual storage device. GAVG should be close to the R metric in the figure. Q: What is the relationship between GAVG, KAVG and DAVG? A: GAVG = KAVG + DAVG KAVG - These counters track the latencies due to the ESX Kernel's command. The KAVG value should be very small in comparison to the DAVG value and should be close to zero. When there is a lot of queuing in ESX, KAVG can be as high, or even higher than DAVG. If this happens, please check the queue statistics, which will be discussed next. DAVG - This is the latency seen at the device driver level. It includes the roundtrip time between the HBA and the storage. DAVG is a good indicator of performance of the backend storage. If IO latencies are suspected to be causing performance problems, DAVG should be examined. Compare IO latencies with corresponding data from the storage array. If they are close, check the array for misconfiguration or faults. If not, compare DAVG with corresponding data from points in between the array and the ESX Server, e.g., FC switches. If this intermediate data also matches DAVG values, it is likely that the storage is under-configured for the application. Adding disk spindles or changing the RAID level may help in such cases. QAVG - The average queue latency. QAVG is part of KAVG.
143
M ONITOR /TROUBLESHOOT S TORAGE DISTRIBUTED R ESOURCE S CHEDULER (SDRS)
ISSUES
Even when Storage DRS is enabled for a datastore cluster, it might be disabled on some virtual disks in the datastore cluster. Check the vSphere, ESXi and vCenter server troubleshooting guide p.47 and p.52. Scenarios like the one below are invoked there: Storage DRS generates an alarm to indicate that it cannot operate on the datastore. Problem - Storage DRS generates an event and an alarm and Storage DRS cannot operate. Cause - The following scenarios can cause vCenter Server to disable Storage DRS for a datastore.
The datastore is shared across multiple data centers - Storage DRS is not supported on datastores that are shared across multiple data centers. This configuration can occur when a host in one data center mounts a datastore in another data center, or when a host using the datastore is moved to a different data center. When a datastore is shared across multiple data centers, Storage DRS I/O load balancing is disabled for the entire datastore cluster. However, Storage DRS space balancing remains active for all datastores in the datastore cluster that are not shared across data centers. The datastore is connected to an unsupported host - Storage DRS is not supported on ESX/ESXi 4.1 and earlier hosts. The datastore is connected to a host that is not running Storage I/O Control. The datastore must be visible in only one data center. Move the hosts to the same data center or unmount the datastore from hosts that reside in other data centers. Ensure that all hosts associated with the datastore cluster are ESXi 5.0 or later. Ensure that all hosts associated with the datastore cluster have Storage I/O Control enabled.
Tools
vSphere Networking Guide vSphere Storage Guide vSphere Troubleshooting Guide vSphere Server and Host Management Guide vSphere Client / vSphere Web Client
VCP6-DCV OBJECTIVE 7.3 - TROUBLESHOOT VSPHERE UPGRADES In today's Objective we'll discuss VCP6-DCV Objective 7.3 - Troubleshoot vSphere Upgrades. You can check the whole VCP6-DCV Study Guide page for all topics there. You can also check the vSphere 6 page where you’ll find many how-to, videos, and tutorials about vSphere 6. VCP6-DCV exam validates you have the skills required to successfully install, deploy, scale and manage VMware vSphere 6. If someone asks you to activate trivial logging you must know how to do it and where.... And this also is part of today's Objective for the VCP6 exam. Note that Trivia logging (Extended verbose) - Displays information, error, warning, verbose, and trivia log entries.... vSphere Knowledge:
Identify vCenter Server and vCenter Server Appliance Upgrade Issues 144
Create a Log Bundle Locate/Analyze VMware Log Bundles Identify Alternative Methods to Upgrade ESXi Hosts in Event of Failure Configure vCenter Logging Options
Tools:
VMware Documentations and KB
IDENTIFY V C ENTER S ERVER
AND V C ENTER
S ERVER A PPLIANCE UPGRADE ISSUES
First thing to do is to check logs:
Check logs for vCenter server or ESXi- Collecting logs for ESXi and vCenter via Web Client - VMware KB Article 2032892.... or VMware KB Article 1011641 for vCenter. Create a log bundle. Collect logs via vSphere Client - VMware KB Article 653
Blog posts from the lab, which gives you step-by-step to follow...
ESXi 5.5 upgrade to 6.0 – via VMware Online Repository Plus few other CLI commands ESXi Offline Bundle Download – To Upgrade ESXi Free (Internet connection is necessary) [Guide] Patch ESXi 5.5 to ESXi 6.0 – Lab Time (via vSphere Upgrade bundle OR via ISO) [Guide] Upgrade ESXi with VMware Update Manager (VUM) – [Guide] – Needs to install VUM first. How to Upgrade from VCSA 5.5 to 6.0 – Lab Time [Guide]
CREATE A LOG B UNDLE ( VIA W EB CLIENT ) LOCATE/A NALYZE VM WARE LOG B UNDLES To collect ESX/ESXi and vCenter Server diagnostic data: 1. 2. 3. 4. 5.
Start the vSphere Web Client and log in to the vCenter Server system. Under Inventory Lists, select vCenter Servers. Click the vCenter Server that contains the ESX/ESXi hosts from which you want to export logs. Click the Monitor tab and click System Logs. Click Export System Logs.
1. 2. 3. 4. 5.
Select the ESX/ESXi hosts from which you want to export logs. Select the Include vCenter Server and vSphere Web Client logs option. This step is optional. Click Next. Select the system logs that are to be exported. Select Gather performance data to include performance data information in the log files.Note: You can update the duration and interval time between which you want to collect the data. 6. Click Next. 7. Click Generate Log Bundle. The Download Log Bundles dialog appears when the Generating Diagnostic Bundle task completes. 145
1. Click Download Log Bundle to save it to your local computer.Note: The host or vCenter Server generates .zip bundles containing the log files. The Recent Tasks panel shows the Generate diagnostic bundles task in progress.
TO
EXPORT THE EVENTS LOG :
1. 2. 3. 4. 5.
Select an inventory object. Click the Monitor tab, and click Events. Click the Export icon. In the Export Events window, specify what types of event information you want to export. Click Generate CSV Report, and click Save.
TO RUN A VM- SUPPORT IN A CONSOLE SESSION Open an SSH session via putty for examle and run the following command:
vm-support As a result..
146
A compressed bundle of logs is produced and stored in a file with a .tgz extension in one of these locations:
/var/tmp/ /var/log/
The current working directory To export the log bundle to a shared vmfs datastore, use this command: vm-support /vmfs/volumes/DATASTORE_NAME
-f
-w
More VMware KBs...
Using vm-support command line tool (VMware KB 1010705, Collecting Diagnostic Information Using the vmsupport Command in VMware ESX/ESXi) How-to obtain vCenter Server Log Bundles (VMware KB 1011641, Collecting Diagnostic Information for VMware vCenter Server) By Using PowerCLI (VMware KB 1027932, Collecting Diagnostic Information for VMware vCenter Server and ESX/ESXi Using the vSphere PowerCLI) How-to obtain vCenter Server and ESXi Log Bundles (VMware KB 653, Collecting Diagnostic Information for Vmware ESX/ESXi Using the vSphere Client)
IDENTIFY A LTERNATIVE M ETHODS
TO
UPGRADE ESX I H OSTS
IN
E VENT
OF
F AILURE
There is quite a few methods to upgrade ESXi.
Via VUM - vsphere update manager. I've done the step-by-step in the lab. Via Scripted upgrade - not my prefered. Check the steps here in the VMware documentation. vSphere Auto Deploy - via autodeploy you can provision a host with new image profile which would contain the ESXi upgrade to 6.0. It would be necessary to use Image builder. You can check VCP6-DCV Autodeoploy Objective here. ESXCLI - well know for free ESXi. And easy to do. Interactive Upgrade - And old fashion method, but easy. By booting the CD. You'll need to burn a CD first with the ISO image. Step-by-step here.
CONFIGURE V CENTER L OGGING O PTIONS Not often used but it's on the blueprint! You might need to change the logging settings when implementing a monitoring solution too... vSphere web client > vCenter Inventory Lists > vCenter servers, click vCenter > Manage TAB > Settings > General > Edit > Logging Settings
147
The options are:
N OTES
None (Disable logging) - Turns off logging Error (Errors only) - Displays only error log entries Warning (Errors and warnings) - Displays warning and error log entries Info (Normal logging) - Displays information, error, and warning log entries Verbose (Verbose) - Displays information, error, warning, and verbose log entries Trivia (Extended verbose) - Displays information, error, warning, verbose, and trivia log entries AND REMARKS ...
Info about SQL – SQL 2012 Enterprise SP1 and SQL 2008 Standard R2 SP1 are supported as upgrade option...
vSphere 6 page on ESX Virtualization - how to, videos, step-by-steps Enabling trivia logging in VMware vCenter Server - VMware KB1001584 Important Information before upgrading to vSphere 6 (KB 2110293) Upgrading to vCenter Server 6.0 best practices (KB 2109772) List of recommended topologies for vSphere 6.0.x (KB 2108548) Update sequence for vSphere 6.0 and its compatible VMware products (KB 2109760) Methods for upgrading to VMware ESXi 6.0 (KB 2109711) vSphere 6.0 is here! – KBs you need to know about (link) Release Notes – here – here is a third-party interpretation (part 1 and part 2) of the release notes that is a long read but has some good points. Whats new – here For those home lab guys – like myself who sometimes cut corners you will need to check this out to learn more about dropped supported hardware. You can find out the status of your backup software here, or at the vendor too of course. I use Veeam and I know that it will break with this upgrade and I will need to wait a bit for it to work!
148
VCP6-DCV OBJECTIVE 7.4 - TROUBLESHOOT AND MONITOR VSPHERE PERFORMANCE In today's Objective we'll discuss VCP6-DCV Objective 7.4 - Troubleshoot and Monitor vSphere Performance. You can check the whole VCP6-DCV Study Guide page for all topics there. You can also check the vSphere 6 page where you’ll find many how-to, videos, and tutorials about vSphere 6. Performance is a key to everything. When your application is slow, you must pinpoint many values to find out what's going on at your virtual infrastructure. If it's the underlying VM which is experiencing problems (wrong sizing of CPU, Memory, Disk...) or is it the underlying storage system, network or physical CPU of the host. Quite complex to find out what's going on. vSphere Knowledge
Describe how Tasks and Events are viewed in vCenter Server Identify critical performance metrics Explain common memory metrics Explain common CPU metrics Explain common network metrics Explain common storage metrics Identify CPU/Memory contention issues Identify Host Power Management Policy Monitor performance through esxtop Troubleshoot Enhanced vMotion Compatibility (EVC) issues Troubleshoot virtual machine performance via vRealize Operations Compare and contrast Overview and Advanced Charts
DESCRIBE
HOW
TASKS
AND
E VENTS
ARE VIEWED IN V C ENTER
S ERVER
TASKS You can view tasks that are associated with a single object or all objects in the vSphere Client inventory. The Tasks & Events tab lists completed tasks and tasks that are currently running. By default, the tasks list for an object also includes tasks performed on its child objects. You can filter the list by removing tasks performed on child objects and by using keywords to search for tasks.
Select Host, VM, Datastore or network TAB > Below, Select object on the left > Monitor TAB > Tasks. You can also select cluster, datacenter or vCenter object to see the tasks...
149
EVENTS The same for events. Example showing the events at the cluster level. Again, you can choose another object like host, datastore, VM....
IDENTIFY
CRITICAL PERFORMANCE METRICS
Performance metrics are organized into logical groups based on the object or object device. Statistics for one or more metrics can be displayed in a chart through vSphere client or web client. Most important and common metrics are CPU, memory, storage and network.
E XPLAIN
COMMON MEMORY METRICS
Memory overhead - this metrics shows how much memory is necessary for the ESXi to be able to run a VM workload. Active guest memory - is amount of memory that VMkernel thinks that it has been used by VM actively. Host memory (consumed) - amount of memory allocated to a VM Host memory (overhead) - is amount consumed for the virtualization overhead to run this particular VM. Avg Memory Usage in KB - similar to Average CPU Usage, this should be reported at both Host and Guest levels. It can give you an indication in terms of who is using the most memory but high usage does not necessarily indicate a bottleneck. If memory usage is high, check the values for Memory Ballooning/Swapping. Balloon (KB) - MCTL - Host cannot meet its memory requirements, so there is a memory pressure on the host. The Balloon driver is installed via VMware Tools onto Windows and Linux guests and its job is to force the operating system, of lightly used guests, to page out unused memory back to ESX so it can grand more memory to other VMs. Swap Used KB - if you see values being reported at the Host for Swap, this indicates that memory demands cannot be satisfied and processes are swapped out to the vSwp file. This is going bad as swapping is the last resort for the hypervisor to manage the memory at some point... Consider vMotioning some VMs out of this host or plan to add more physical RAM.... Consumed - Consumed memory is the amount of Memory Granted on a Host to its guests minus the amount of Memory Shared across them. Memory can be over-allocated, unlike CPU, by sharing common memory pages such as Operating System pages. This metric displays how much Host Physical Memory is actually being used (or consumed) and includes usage values for the Service Console and VMkernel. Active - this metric reports the amount of physical memory recently used by the guests on the Host and is displayed as “Guest Memory Usage” in vCenter at Guest level. From vSphere Monitoring and Performance guide p. 136
150
SWR/s (MB) - Rate at which the ESXi host swaps in memory from disk for the resource pool or virtual machine. SWW/s (MB) - Rate at which the ESXi host swaps resource pool or virtual machine memory to disk. SWCUR (MB) - Current swap usage by this resource pool or virtual machine. SWTGT (MB) - Target where the ESXi host expects the swap usage by the resource pool or virtual machine to be MCTL? - Check if the memory balloon driver is installed or not. N means no, Y means yes. MCTLSZ (MB) - Amount of physical memory reclaimed from the resource pool by way of ballooning. MCTLTGT (MB) - Amount of physical memory the ESXi system attempts to reclaim from the resource pool or virtual machine by way of ballooning. MCTLMAX (MB) - Maximum amount of physical memory the ESXi system can reclaim from the resource pool or virtual machine by way of ballooning. This maximum depends on the guest operating system type.
E XPLAIN
COMMON
CPU
METRICS
vSphere Monitoring and Performance guide p. 131
%USED - Percentage of physical CPU core cycles used by the resource pool, virtual machine, or world. %USED might depend on the frequency with which the CPU core is running. When running with lower CPU core frequency, %USED can be smaller than %RUN. On CPUs which support turbo mode, CPU frequency can also be higher than the nominal (rated) frequency, and %USED can be larger than %RUN. %USED = %RUN + %SYS - %OVRLP %RDY - Percentage of time the resource pool, virtual machine, or world was ready to run, but was not provided CPU resources on which to execute. 100% = %RUN + %RDY + %CSTP + %WAIT %CSTP - Percentage of time a resource pool spends in a ready, co-deschedule state. NOTE You might see this statistic displayed, but it is intended for VMware use only. 100% = %RUN + %RDY + %CSTP + %WAIT
%SYS - Percentage of time spent in the ESXi VMkernel on behalf of the resource pool, virtual machine, or world to process interrupts and to perform other system activities. This time is part of the time used to calculate %USED. %USED = %RUN + %SYS - %OVRLP
%WAIT - Percentage of time the resource pool, virtual machine, or world spent in the blocked or busy wait state. This percentage includes the percentage of time the resource pool, virtual machine, or world was idle. 100% = %RUN + %RDY + %CSTP + %WAIT
E XPLAIN
COMMON NETWORK METRICS
vSphere Monitoring and Performance guide p 141.
MbTX/s -MegaBits transmitted per second. MbRX/s -MegaBits received per second.
Dropped packed metrics:
%DRPTX - Percentage of transmit packets dropped %DRPRX - Percentage of receive packets dropped.
E XPLAIN
COMMON STORAGE METRICS
Latency, latency, latency...
GAVG (Guest Average Latency) total latency as seen from vSphere
151
KAVG (Kernel Average Latency) time an I/O request spent waiting inside the vSphere storage stack. QAVG (Queue Average latency) time spent waiting in a queue inside the vSphere Storage Stack. DAVG (Device Average Latency) latency coming from the physical hardware, HBA and Storage device.
IDENTIFY CPU/M EMORY CONTENTION ISSUES IDENTIFY H OST P OWER M ANAGEMENT P OLICY
High Performance - This power policy maximizes performance, using no power management features. It keeps CPUs in the highest P-state at all times. It uses only the top two C-states (running and halted), not any of the deep states (for example, C3 and C6 on the latest Intel processors). Balanced - This power policy is designed to reduce host power consumption while having little or no impact on performance. The balanced policy uses an algorithm that exploits the processor’s P-states. Balanced is the default power policy for ESXi. Low Power - This power policy is designed to more aggressively reduce host power consumption, through the use of deep C-states, at the risk of reduced performance. Custom - This power policy starts out the same as balanced, but it allows individual parameters to be modified. If the host hardware does not allow the operating system to manage power, only the Not Supported policy is available. (On some systems, only the High Performance policy is available.)
152
M ONITOR
PERFORMANCE THROUGH ESXTOP
Check this community thread ESXTOP. It's excellent!
TROUBLESHOOT E NHANCED V M OTION C OMPATIBILITY (EVC)
ISSUES
From this VMware KB - EVC and CPU Compatibility FAQ you can learn that: EVC is short for Enhanced vMotion Compatibility. EVC allows you to migrate virtual machines between different generations of CPUs. with EVC you can mix older and newer server generations in the same cluster and be able to migrate virtual machines with vMotion between these hosts. This makes adding new hardware into your existing infrastructure easier and helps extend the value of your existing hosts.
ESXi 6.0 supports these EVC modes:
AMD Opteron Generation 1 (Rev. E) AMD Opteron Generation 2 (Rev. F) AMD Opteron Generation 3 (Greyhound) AMD Opteron Generation 3 (no 3Dnow!) (Greyhound) AMD Opteron Generation 4 (Bulldozer) AMD Opteron "Piledriver" Generation Intel "Merom" Generation (Intel Xeon Core 2) Intel "Penryn" Generation (Intel Xeon 45nm Core2) Intel "Nehalem" Generation (Intel Xeon Core i7) Intel "Westmere" Generation (Intel Xeon 32nm Core i7) Intel "Sandy Bridge" Generation Intel "Ivy Bridge" Generation Intel "Haswell" Generation
TROUBLESHOOT
VIRTUAL MACHINE PERFORMANCE VIA V R EALIZE
O PERATIONS
vROPs is a separate vSphere product and needs really deep understanding on what's going on. I think that there should be a separate chapter on the blueprint if required for the exam.... The architecture has changed as well (there is no more UI VM and Analytics VM like int he vCOPS 5.8). The appliance works in cluster, and from within the dashboard you’ll be able to deploy/add an additional appliance (node) to the system to scale out. The solution is highly resilient, by using Gemfire to spread the data across at least 2 nodes. Two slices has the copy of the data (at least). If there is a failure of one of the slices, then another slice takes over.
153
V R EALIZE
O PERATION M ANAGEMENT S UITE 6.0 – N EW
AND IMPROVED FEATURES
Increased Scale of a single deployment Cluster – shared data and UI Resiliency (application RAID!) Smart alerts with problem-definitions Customizable Dashboards and Reports (drag and drop to create new) Advanced capacity modeling via possibility to save capacity project and do a what-if analyse. Public APIs released to partners in order to work on additional extensibilities.
Newly the product will feature a management pack integration (add-ons) which will be delivered by VMware and partners for specific storage devices. There is 40-50 management packs available on the VMware Solution Exchange and those management packs can be installed inside the vRealize Management Operation. (vROPS) From the overview dashboard you can see which problems arise or will arise (in the Risk alerts section). By clicking the link you can drill down to see the problem.
COMPARE
AND CONTRAST
O VERVIEW
AND
A DVANCED CHARTS
OVERVIEW CHARTS vSphere Performance guide p14. Display multiple data sets in one panel to easily evaluate different resource statistics, display thumbnail charts for child objects, and display charts for a parent and a child object. Advanced charts display more information than overview charts, are configurable, and can be printed or exported.
154
Overview chart from my lab. Select Host > Monitor TAB > Performance > Drop down chose between Home or Virtual Machines.
A DVANCED CHARS Use advanced charts, or create your own custom charts, to see more performance data. Advanced charts can be useful when you are aware of a problem but need more statistical data to pinpoint the source of the trouble.
Slect Host > Monitor TAB > Performance > Click Advanced
Advanced charts include the following features:
More information. Hover over a data point in a chart and details about that specific data point are displayed. Customizable charts. Change chart settings. Save custom settings to create your own charts. Export to spreadsheet. Save to image file or spreadsheet
155
Tools o o o o o
vSphere Resource Management Guide vSphere Troubleshooting Guide vSphere Monitoring and Performance Guide vCenter Operations Manager Getting Started Guide (vSphere UI) vSphere Client / vSphere Web Client
VCP6-DCV OBJECTIVE 7.5 - TROUBLESHOOT HA AND DRS CONFIGURATIONS AND FAULT TOLERANCE Today's VCP6 topic is following: VCP6-DCV Objective 7.5 - Troubleshoot HA and DRS Configurations and Fault Tolerance. A large topic, which is difficult to fit into single post. The VCP6-DCV certification exam validates that you have the skills required to successfully install, deploy, scale and manage VMware vSphere 6 environments. Check the VCP6-DCV Study Guide [Unofficial] page on my blog for all topics required to pass the exam. Stay tuned for the PDF version .... Check also other How-to articles, videos, and news concerning vSphere 6 - dedicated vSphere 6 page. vSphere Knowledge
Identify HA/DRS and vMotion requirements Verify vMotion/Storage vMotion configuration Verify HA network configuration Verify HA/DRS cluster configuration Troubleshoot HA capacity issues Troubleshoot HA redundancy issues Interpret the DRS Resource Distribution Graph and Target/Current Host Load Deviation Troubleshoot DRS load imbalance issues Troubleshoot vMotion/Storage vMotion migration issues Interpret vMotion Resource Maps Identify the root cause of a DRS/HA cluster or migration issue based on troubleshooting information Verify Fault Tolerance configuration Identify Fault Tolerance requirements
IDENTIFY HA/DRS
AND VM OTION REQUIREMENTS
vSphere HA is very easy to set up and manage and is the simplest high-availability solution available for protecting virtual workloads. HA R EQUIREMENTS :
Redundant Management Network - Verify that you are using redundant management network connections for vSphere HA. For information about setting up network redundancy, see “Best Practices for Networking.” Proper Licensing - vSphere Essentials Plus and higher licensing. Essentials (only) won't do the job... Minimum 2 hosts in a cluster - HA needs 2 hosts to be able to initiate failover. Static IP config - Host which participate in HA/DRS clusters has to be configured with static IP address. 156
Shared Storage - VMs must run on shared storage Access All hosts to VM neworks and datastores - All Hosts shall be able to reach the VM's networks and datastores. VMware tools on VMs - All VMs has to have VMware tools in stalled in order to be able to activate VM Monitoring Configure Two Shared Datastores at least - to have redundancy for vSphere HA datastore hearbeating. ipv6 and ipv4 are supported - vSphere HA supports both IPv4 and IPv6. See “Other vSphere HA Interoperability Issues,” on page 31 for considerations when using IPv6. Enable APD Timeout - If you want to use VM Component Protection, hosts must have the All Paths Down (APD) Timeout feature enabled. Wants VMCP with HA? - To use VM Component Protection, clusters must contain ESXi 6.0 hosts or later.
DRS R EQUIREMENTS : vCenter server resource management p.63
Shared storage - SAN/NAS, VSAN... any supported shared storage. Configure all managed hosts to use shared VMFS volumes. Place the disks of all virtual machines on VMFS volumes that are accessible by source and destination hosts. Make sure that the VMFS volume is sufficiently large to store all virtual disks for your virtual machines and also make sure that all VMFS volumes on source and destination hosts use volume names, and all virtual machines use those volume names for specifying the virtual disks. CPU Requirements - use EVC to help you out with different hardware in your cluster.
V M OTION
R EQUIREMENTS :
Gigabit ethernet for vMotion is a bare minimum - make sure you comply with that No RDM or MSCS support -Microsoft Cluster service (MSCS) isn't supported.
157
VMs with CDROM Unattached - Cannot vMotion a VM that is backed by a device that isn't accessible to the target host. I.E. A CDROM connected to local storage on a host. You must disconnect these devices first. USB is supported as long as the device is enabled for vMotion For VMs with USB - must enable all USB devices that are connected to the virtual machine from a host for vMotion. If one or more devices are not enabled for vMotion, migration will fail. TCP port 8000 - incoming and outgoing firewall port for ESXi hosts, this is a required port for vMotion.
V ERIFY V M OTION/S TORAGE V M OTION
CONFIGURATION
Check the vmkernel network interfaces for the correct network config. Make sure that the EVC in the cluster is configured (if needed) and tested prior enabling DRS. Make sure that all hosts within cluster can reach the shared storage and no VMs are left on local storage somewhere....
V ERIFY HA
NETWORK CONFIGURATION
Check this section at the vSphere Availability Guide p.29 and p.39
When you change the networking configuration on the ESXi hosts themselves, for example, adding port groups, or removing vSwitches, suspend Host Monitoring. After you have made the networking configuration changes, you must reconfigure vSphere HA on all hosts in the cluster, which causes the network information to be reinspected. Then re-enable Host Monitoring.
On ESXi hosts in the cluster, vSphere HA communications, by default, travel over VMkernel networks. With an ESXi host, if you wish to use a network other than the one vCenter Server uses to communicate with the host for vSphere HA, you must explicitly enable the Management traffic check-box. Der, Die, Das! Isolation Address das.isolationaddress By default, the network isolation address is the default gateway for the host. Only one default gateway is specified, regardless of how many management networks have been defined. You should use the das.isolationaddress[...] advanced option to add isolation addresses for additional networks. This address is pinged only when heartbeats are not received from any other host in the cluster. If not specified, the default gateway of the management network is used. This default gateway has to be a reliable address that is available, so that the host can determine if it is isolated from the network. You can specify multiple isolation addresses (up to 10) for the cluster:
158
das.isolationaddressX, where X = 0-9. Typically you should specify one per management network. Specifying too many addresses makes isolation detection take too long. Check p.37 for all advanced options.
V ERIFY HA/DRS
CLUSTER CONFIGURATION
You can check the cluster summary through vSphere client or vSphere web client.
vSphere client...
TROUBLESHOOT HA
CAPACITY ISSUES
As you know the 3 possible HA admission config policies you must know are:
159
Host Failures Cluster Tolerates - With the Host Failures Cluster Tolerates admission control policy, VMware HA ensures that a specified number of hosts can fail and sufficient resources remain in the cluster to fail over all the virtual machines from those hosts Percentage of Cluster Resources - You can configure VMware HA to perform admission control by reserving a specific percentage of cluster resources for recovery from host failures. With the Percentage of Cluster Resources Reserved admission control policy, VMware HA ensures that a specified percentage of aggregate cluster resources is reserved for failover. Specify a Failover Host - when a host fails, VMware HA attempts to restart its virtual machines on a specified failover host. If this is not possible, for example the failover host itself has failed or it has insufficient resources, then VMware HA attempts to restart those virtual machines on other hosts in the cluster.
The three HA admission configuration policies...
What can go wrong? Hosts disconnected, unconfigured (right click > reconfigure for HA). Also when (if) setting "specify failover host" policy, than you might end up with some VMs non restarted if several hosts fails, as you did not set enough hosts for failover. I usually use "percentage of cluster resources" or "host failures cluster tolerates" policies. If your cluster contains any virtual machines that have much larger reservations than the others, they will distort slot size calculation. To avoid this, you can specify an upper bound for the CPU or memory component of the slot size by using the das.slotcpuinmhz or das.slotmeminmb advanced attributes, respectively. Slot size is comprised of two components, CPU and memory.
vSphere HA calculates the CPU component by obtaining the CPU reservation of each powered-on virtual machine and selecting the largest value. If you have not specified a CPU reservation for a virtual machine, it is assigned a default value of 32MHz. You can change this value by using the das.vmcpuminmhz advanced attribute.) vSphere HA calculates the memory component by obtaining the memory reservation, plus memory overhead, of each powered-on virtual machine and selecting the largest value. There is no default value for the memory reservation.
If large VMs present in the cluster than you might want to use "percentage of cluster resources" admission policy as you won't need to deal with slot sizes.
TROUBLESHOOT HA
REDUNDANCY ISSUES
160
NIC teaming is the answer. Redundancy, redundancy.... Use 2 or more pNICs in a team to provide failover possibility. If possible use separate physical switches to provide redundancy.
INTERPRET THE DRS R ESOURCE DISTRIBUTION G RAPH DEVIATION
AND
TARGET /CURRENT H OST LOAD
Even if VMware is pushing the web client, I feel that the C# client shows more details when flying over with a mouse on a chart to display the memory utilization of a host within cluster, you can actually see an individual VM, how such a VM consumes memory on that particular host... You can access the charts (in vSphere client) from the summary tab when selecting your cluster on the left hand side first. Click the "View resource distribution chart" link, as on the image below....
161
This is not the case of vSphere Web client....
The DRS Resource Distribution chart displays CPU or Memory metrics for each of the hosts in the cluster. YOu can switch from percentage to mebabytes (for memory) resp from percentage to megaherty (for CPU). DRS cluster is load balanced when each of its hosts’ level of consumed resources is equivalent to the others. When they aren’t, the cluster is considered to be imbalanced and VMs must be relocated to restore the balance.
TROUBLESHOOT DRS
LOAD IMBALANCE ISSUES
Imbalanced load issues can happens if:
Host is in maintenance mode VM-host affinity/anti-affinity rules being used VM-VM affinity rules being used
A cluster might become unbalanced because of uneven resource demands from virtual machines and unequal capacities of hosts.
The migration threshold is too high - A higher threshold makes the cluster a more likely candidate for load imbalance. Affinity/Anti-Affinity Rules - VM/VM or VM/Host DRS rules prevent virtual machines from being moved. Disabled DRS - DRS is disabled for some VMs... A device is mounted to one or more virtual machines preventing DRS from moving the virtual machine in order to balance the load. Virtual machines are not compatible with the hosts to which DRS would move them. That is, at least one of the hosts in the cluster is incompatible for the virtual machines that would be migrated. For example, if host A's CPU is not vMotion-compatible with host B's CPU, then host A becomes incompatible for powered-on virtual machines running on host B. It would be more detrimental for the virtual machine's performance to move it than for it to run where it is currently located. This may occur when loads are unstable or the migration cost is high compared to the benefit gained from moving the virtual machine. Unconfigured/disabled vMotion - vMotion is not enabled or set up for the hosts in the cluster.
162
TROUBLESHOOT V M OTION/S TORAGE V M OTION
MIGRATION ISSUES
First, check requirements for vMotion/sVMotion.
VMware tools status - Make sure that VMtools installaiton is not "stuck" in a VM...as during installation of VMware tools it's not possible to do a VMotion of such a VM due to hearbeats. Source destination datastores are available - make sure that this apply... Licensing - sVMotion requires vSphere "standard"licensing... If RDM is used in physical compatibility mode - no sVMotion or snapshoting of VMs... Virtual machine snapshots are available for RDMs with virtual compatibility mode only.Physical Compatibility Mode VMkernel passes all SCSI commands to the device, with one exception: the REPORT LUNs command is virtualized so that the VMkernel can isolate the LUN to the owning virtual machine. If not, all physical characteristics of the underlying hardware are exposed. It does allows the guest operating system to access the hardware directly. VM with physical compatibility RDM has limits like that you cannot clone such a VM or turn it into a template. Also sVMotion or cold migration is not possible.
A quick quote from VMware blog post, which is new (note that sVMotion do not work with such a disks): In vSphere 6.0, you can configure two or more VMs running Windows Server Failover Clustering (or MSCS for preWindows 2012 OSes), using common, shared virtual disks (RDM) among them AND still be able to successfully vMotion any of the clustered nodes without inducing failure in WSFC or the clustered application. What's the bigdeal about that? Well, it is the first time VMware has ever officially supported such configuration without any thirdparty solution, formal exception, or a number of caveats. Simply put, this is now an official, out-of-the-box feature that does not have any exception or special requirements other than the following:
The VMs must be in "Hardware 11" compatibility mode - which means that you are either creating and running the VMs on ESXi 6.0 hosts, or you have converted your old template to Hardware 11 and deployed it on ESXi 6.0 The disks must be connected to virtual SCSI controllers that have been configured for "Physical" SCSI Bus Sharing mode And the disk type *MUST* be of the "Raw Device Mapping" type.
INTERPRET V M OTION R ESOURCE M APS A vCenter map is a visual representation of your vCenter Server topology. Maps show the relationships between the virtual and physical resources available to vCenter Server. Maps are available only when the vSphere Client is connected to a vCenter Server system. The maps can help you determine such things as which clusters or hosts are most densely populated, which networks are most critical, and which storage devices are being utilized. vCenter Server provides the following map views.
Virtual Machine Resources - Displays virtual machine-centric relationships. Host Resources - Displays host-centric relationships. Datastore Resources - Displays datastore-centric relationships. vMotion Resources - Displays hosts available for vMotion migration.
163
You can configure the maximum requested topology entities (helps for large environments) via vSphere client by going to the Client Menu > Edit > Client settings > Maps TAB
IDENTIFY
THE ROOT CAUSE OF A
DRS/HA
CLUSTER OR MIGRATION ISSUE BASED ON
TROUBLESHOOTING INFORMATION
V ERIFY F AULT TOLERANCE
CONFIGURATION
vSphere 6 has introduced New FT with up to 4vCPU support. However if virtual machine has only a single vCPU, however, you can use legacy FT instead, for backward compatibility. But, unless technically necessary, use of legacy FT is not recommended. To use legacy Fault Tolerance, you must configure an advanced option for the virtual machine. After you complete this configuration, the legacy FT VM is different in some ways from other fault tolerant VMs. Difference between Legacy FT (used in previous releases of vSphere) and FT (v6).
164
If you want/need to use legacy FT, check the requirements.
IDENTIFY F AULT TOLERANCE
REQUIREMENTS
Licensing - The number of vCPUs supported by a single fault tolerant VM is limited by the level of licensing that you have purchased for vSphere. Fault Tolerance is supported as follows:
vSphere Standard and Enterprise. Allows up to 2 vCPUs vSphere Enterprise Plus. Allows up to 4 vCPUs
10 GbE Network - hard requirement for FT v6! CPU Requirements - CPUs that are used in host machines for fault tolerant VMs must be compatible with vSphere vMotion or improved with Enhanced vMotion Compatibility. Also, CPUs that support Hardware MMU virtualization (Intel EPT or AMD RVI) are required. The following CPUs are supported.
Intel Sandy Bridge or later. Avoton is not supported. AMD Bulldozer or later.
P OSSIBLE E NFORCING
AT THE HOST LEVEL
Advanced settings:
das.maxftvmsperhost The maximum number of fault tolerant VMs allowed on a host in the cluster. Both Primary VMs and Secondary VMs count toward this limit. The default value is 4. das.maxftvcpusperhost
165
The maximum number of vCPUs aggregated across all fault tolerant VMs on a host. vCPUs from both Primary VMs and Secondary VMs count toward this limit. The default value is 8. Tools
vSphere Resource Management Guide vSphere Monitoring and Performance Guide vSphere Installation and Setup Guide vSphere Troubleshooting Guide vSphere Availability Guide vSphere Client / vSphere Web Client
VCP6-DCV OBJECTIVE 8.1 - DEPLOY ESX I HOSTS USING AUTODEPLOY In today's topic we will take a look at Autodeploy - VCP6-DCV Objective 8.1 - Deploy ESXi Hosts Using Autodeploy, which is quite large topic. Autodeploy allows provision dozens (or hundreds) physical hosts with ESXi images. It's possible to manage large deployments where hosts are booted via network from a central Auto-deploy server. In conjunction with host profiles it's possible to attach hosts to clusters and push a different configurations depending on parameters like hardware vendor. Check the VCP6-DCV Study Guide or other How-to articles, videos, and news concerning vSphere 6 at the dedicated vSphere 6 page.
IDENTIFY ESX I A UTODEPLOY
REQUIREMENTS
There are some requirements but at the same time that there are also limitations, so make sure that any of those limits does not actually puts a brake on your project. Before you can start to use vSphere Auto Deploy, you must prepare your environment. You start with server setup and hardware preparation. You must register the Auto Deploy software with the vCenter Server system that you plan to use for managing the hosts you provision, and also install the VMware PowerCLI on a management station (or Windows based vCenter server).
Hardware requirements for ESXi 6.0 - check here. ESXi hardware must be set to use BIOS (EFI isn't supported) Require Ports opened between vCenter server and ESXi hosts - check here If VLANs used, then check that works properly. Minimum Storage - 2Gb of storage for storing ESXi images, where each of those images requires about 350Mb. So depending of how many profiles you will use by taking that number into consideration. Autodeploy server must use IPv4 - The PXE boot infra does not support IPv6. Install ESXi Dump colletor - this will allow
CONFIGURE A UTODEPLOY You must first enable the service. Go to vSphere Web Client > System Configuration > Services > Select Autodeploy > Actions > Edit Startup Type
166
This will prompt you for the service settings:
And then make sure that you start the service!
On the vCenter Server Appliance, the Auto Deploy service by default is set to Manual (on Windows it's Disabled). If you want the Auto Deploy service to start automatically upon OS startup, select Automatic. CONFIGURE TFTP: In a vSphere Web Client > Inventory list > select the vCenter Server > Manage tab > Settings > Auto Deploy.
167
Then click the Download TFTP Boot Zip to download the TFTP configuration file and unzip the file to the directory in which your TFTP server stores files. Install TFTP server (I usually use the Free TFTP server from Solarwinds). The installer creates a default directory which can be changed. I changed mine to c:tftp to keep it simple. You can configure the option by going to File > Configure menu. While there, make sure that you start the service. (Note: you can also go to Windows services to make the TFTP service start automatically during the boot as by default it has manual start only).
That’s it for TFTP server. There is nothing else to play with and we can move on. DHCP S ERVER O PTIONS Next I’ll show you the options you need to configure on your DHCP server. There are just two options which needs to be configured at the scope level. When you click on the Autodeploy icon in vSphere client, you’ll end up on this page where you can see some strange name of file. But this exact name will be needed for setting up options in our DHCP server! It’s the undionly.kpxe.vmw-hardwired. So next step is to click and download the TFTP boot zip files to the c:tftp directory that we created and set up on our TFTP server. Unzip the file into the same directory You should have a view like this:
168
Once done, we can copy this name of the file (undionly.kpxe.vmw-hardwired) as an option 67 in our DHCP server. In my case I have Windows DHCP server which sits on my domain controller.
Now you should configure each of your ESXi host's BIOS to boot from network.
E XPLAIN P OWER CLI
CMDLETS FOR
A UTODEPLOY
Auto Deploy uses a PXE boot infrastructure in together with vSphere host profiles to provision and customize host(s). No state is stored on the host(s) itself. But rather, the Auto Deploy server manages state information for each host. Autodeploy server has the informations about the location of image profile and host profiles and this information is specified in the rules that map machines to image profiles and host profile. Whe host boots up from the first time it's vCenter server who creates a host objects and stores the information in the vCenter DB. The whole architecture:
169
A UTO DEPLOY CMDLETS There are many more auto deploy cmdlets than the ones I’m using in this post, so here is the full list for reference: Command
Description
Get-DeployCommand
Gives you a list of Auto Deploy cmdlets.
New-DeployRule
Creates a new rule with the specified items and patterns
Set-DeployRule
Updates an existing rule with the specified items and patterns. Rules that belong to a working ruleset can not be updated.
Get-DeployRule
Retrieves rules as specified by an administrator.
Copy-DeployRule
Clones and updates an existing rule.
Add-DeployRule
Adds one or more rules to the working and active ruleset(s). The NoActivate parameter can be specified to add a rule only to the working ruleset.
Remove-DeployRule
Removes one or more rules from the working and active rule set. The rule(s) can be deleted by using the -Delete parameter.
Set-DeployRuleSet
Explicitly sets the list of rules in the working rule set.
Get-DeployRuleSet
Retrieves the current working rule set or active rule set.
Switch-ActiveDeployRuleSet Activates a rule set so that any new requests are evaluated through the rule set.
170
Get-VMHostMatchingValues
Retrieves rules matching a pattern. For example, all rules that apply to hosts can be retrieved.
TestDeployRulesetCompliance
Checks whether items associated with a specified host are in compliance with an active rule set.
RepairDeployRuleSetCompliance
Updates the image profile, host profile and location for each host in the vCenter Server inventory based on the results of Test-DeployRulesetCompliance.
Apply-EsxImageProfile
Associates the specified image profile with the specified host.
Get-VMHostImageProfile
Retrieves the image profile in use by a specified host.
Repair-DeployImageCache
Command can be used if the image cache is accidentally deleted.
Get-VMHostAttributes
Returns attributes for a host that are used when the Auto Deploy server evaluates the rules.
Stateless caching - Autodeploy does not store ESXi configuration or state on the host disk by default. Rather an image profile defines the image that the host is provisioned with, and other host attributes are managed through host profiles. A host that uses Auto Deploy for stateless caching has to have an access to Autodeploy server and vCenter server. That's why the vCenter server has to be UP in order to be able to provission those hosts (SPOF???).
Stateful installs - In this case it is possible to provision a host with Auto Deploy and set up the host to store the image to disk. On subsequent boots, the host boots from disk
DEPLOY /M ANAGE
MULTIPLE
ESX I
HOSTS USING
A UTODEPLOY
1. Install PowerCLI 2. Use the PowerCLI cmdlets to define rule which assigns an image profile and host profile (optional) to the host. 3. Configure reference host and create a host profile where you'll keep what's common for all hosts (storage, networking and other). Write a rule that assigns not only the already tested image profile but also the host profile to the target host. 4. If you need manual information to be entered, you can specify user input in the customization of the host within the vSphere web client. Getting Help with PowerCLI commandlets
Basic help: Get-Help cmdlet_name Detailed help: Get-Help cmdlet_name -Detailed
I have done a blog post series covering host profiles, autodeploy...when learning towards VCAP exam. You can use it as a guide for preparation for the VCP exam as most things hasn't changed...
VCAP Diary – VMware vSphere ESXi Image Builder VMware vSphere AutoDeploy – Run some PowerCLI and you’re the Boss VMware vSphere AutoDeploy – Install and configure VCAP Diary – VMware vSphere Host Profiles
171
VMware vSphere Host Profiles – options and troubleshooting
VMware documentation and Tools
vSphere Installation and Setup Guide vSphere Client / vSphere Web Client Direct Console User Interface (DCUI)
Some more links:
About Reprovisioning Hosts. Test and repair rule compliance
VCP6-DCV OBJECTIVE 8.2 - CUSTOMIZE HOST PROFILE SETTINGS In today's Objective we'll discuss VCP6-DCV Objective 8.2 - Customize Host Profile Settings. Host profiles are feature which is present in the vSphere enterprise plus licensing, and allows the to "uniformize" and/or push configuration changes to all hosts in the cluster. Host profiles are necessary when using VMware vSphere Autodeploy, which takes advantage of host profile after the stateless image is loaded in memory, to apply a configuration through that host profile. What’s needed is also an Autodeploy installed and configured, together with DHCP options enabled for Autodeploy to work with. But we'll look at this Objective another time. Check the VCP6-DCV Study Guide page or other How-to articles, videos, and news concerning vSphere 6 at the dedicated vSphere 6 page. vSphere Knowledge
Create/Edit/Remove a Host Profile from an ESXi host Import/Export a Host Profile Attach/Apply a Host Profile to an ESXi host or cluster Perform compliance scanning and remediation of an ESXi host using Host Profiles
CREATE/E DIT /R EMOVE
A
H OST P ROFILE
FROM AN
ESX I
HOST
Create host profile by extracting a reference host's config. vSphere web client > Host profiles > Click the Plus sign > Select Host > Enter Name for the host profile > Next > Finish
172
TO DELETE HOST PROFILE : Select the host profile to delete > Actions > delete
TO E DIT H OST PROFILE : Select the Host profile > Actions > Edit settings > Next > Edit Host profile > When done, click Finish.
Host Profiles can be also used to validate the configuration of a host by checking compliance of a host or cluster against the Host Profile that is associated with that host or cluster. 173
IMPORT /E XPORT
A
H OST P ROFILE
It's possible to export host profile as a *.vpf file (VMware Profile Format) ... As you can see the administrator's password aren't exported for security reasons.
You will be prompted to re-enter the values for the password after the profile is imported and the password is applied to a host. HOW TO EXPORT ? vSphere Web Client > Host Profiles > Select Profile > Actions > Export Host Profile
A TTACH /A PPLY
A
H OST P ROFILE
TO AN
ESX I
HOST OR CLUSTER
That the second step after creating a host profile from reference host. You need to attach the host or cluster to the Host Profile. Web Client > Select Host profile > Actions > Attach/detach Hosts and Clusters
And then on this screen you can select single host or whole cluster...
174
You can update or change the user input parameters for the Host Profiles policies by customizing the host.
P ERFORM
COMPLIANCE SCANNING AND REMEDIATION OF AN
ESX I
HOST USING
H OST P ROFILES
vSphere host profiles PDF p. 12 You can confirm the compliance of a host or cluster to its attached Host Profile and determine which, if any, configuration parameters on a host are different from those specified in the Host Profile. HOW TO PERFORM COMPLIANCE SCANNING ? After attaching the host/cluster to a profile you can check the compliance....
Select the host profile > click the check the compliance icon (or go to Actions > Check Host Profile compliance) .
To see more detail on compliance failures, select a Host Profile from the Objects tab for which the last compliance check produced one or more failures. In order to see specific detail on which parameters differ between the host that failed compliance and the Host Profile, click on the Monitor tab and select the Compliance view. Then, expand the object hierarchy and select the failing host. The differing parameters are displayed in the Compliance window, below the hierarchy. R EMEDIATE A HOST In the event of a compliance failure, use the Remediate function to apply the Host Profile settings onto the host. This action changes all Host Profile managed parameters to the values contained in the Host Profile attached to the host. Navigate to the Host profile > Select Monitor Tab > Click Compliance > Right click the host > Host profiles > Remediate
175
vSphere Documentation and Tools
vSphere Installation and Setup Guide vSphere Host Profiles Guide vSphere Client / vSphere Web Client
So another VCP6-DCV topic done. Host profiles with autodeploy are advanced enterprise features/topics which some of you might not need every day or will never implement, especially Autodeploy as IMHO it introduces SPOF (single point of failure) - dependent on vCenter server. But it's just my own opinion and it's also possible to mitigate such a risk with protecting vCenter server FT. But that's another story...
176
VCP6-DCV OBJECTIVE 8.3 - CONSOLIDATE PHYSICAL WORKLOADS USING VM WARE CONVERTER VCP6-DCV blueprint covers P2V chapter too. This post will cover VCP6-DCV Objective 8.3 - Consolidate Physical Workloads using VMware Converter. VMware converter was (and still is) very popular free tool for P2V or V2V conversions. This was the first tool I actually started to work with when I first started with datacenter virtualization. Converting physical systems to VMs is kind of fascinating Compared to VCP 5 it seems that for VCP6 there is more material to study and more topics to master. For whole exam coverage I created a dedicated VCP6-DCV page. Or if you’re not preparing to pass a VCP6-DCV, you might just want to look on some how-to, news, videos about vSphere 6 – check out my vSphere 6 page. VMware Knowledge
Identify VMware Converter requirements Convert Physical Workloads using VMware Converter Modify server resources during conversion Interpret and correct errors during conversion
IDENTIFY VM WARE CONVERTER
REQUIREMENTS
VMware vCenter Converter Standalone User's Guide p.17 S YSTEM R EQUIREMENTS :
Windows - Windows XP Professional (32-bit and 64-bit) SP3 and higher, 2003 srv (x32 and x64) and up to 2012 (not 2012R2 - but I think it'll get updated). Linux - RHEL 3.x - 6.x, SUSE 9.x - 11.x, Ubuntu 10.04 LTS - 13.04 .... both x32 and 64bit versions.
177
S UPPORTED F IRMWARE I NTERFACES : The converter standalone supports BIOS and UEFI sources and the firmware intereface is preserved (cannot convert BIOS to UEFI). For UEFI the supported destination types are Workstation 8.0 and later or ESXi 5.0 and later or vCenter 5.0 and later. Supported Sources: P OWERED O N:
Remote Windows (Linux) physical machines Local Windows physical machines Windows VM running on Hyper-V Server Powered On VMware VMs Powered On Hyper-V 2012 VMs Powered On VMs running KVM, XEN
VMWARE V C ENTER VMS :
vCenter server 4.0, 4.1, 5.0, 5.1 and 5.5 ESX 4.0 and 4.1 ESXi 4.1, 5.0, 5.1 and 5.5
VMWARE VIRTUAL M ACHINES :
VMware Workstation 7.x, 8.x, 9.x, and 10.x VMware Fusion 3.x, 4.x, 5.x, and 6.x VMware Player 3.x, 4.x, 5.x, and 6.x
H YPER -V S ERVER VMS
Windows Server 2003 (x86 and x64), SP1 and SP2 Windows Server 2003 (x86 and x64) R2 SP1 and SP2 Windows Server 2008 (x86 and x64) SP2 Windows Server 2008 (x64) R2 and R2 SP1 Windows 7 (except Home editions) Windows Vista SP1 and SP2 (except Home editions) Windows XP Professional SP2, SP3, and x64 SP2
THIRD P ARTY VM S OR SYSTEM IMAGE
Acronis, Norton Ghost, .... S UPPORTED D ESTINATION TYPES :
VMware vCenter VMs - (ESX 4.0 and 4.1), ESXi 4.1, ESXi 4.0, 4.1, 5.0, 5.1, and 5.5, vCenter Server 4.0, 4.1, 5.0, 5.1, and 5.5 VMware Hosted VMs - VMware Workstation 7.x, 8.x, 9.x, and 10.x, VMware Fusion 3.x, 4.x, 5.x, and 6.x, VMware Player 3.x, 4x, 5.x, and 6.x
Unsupported Sources Disk type - RAID, GPT/MBR hybrid disks. Supported destination types - VMware vCenter Converter Standalone User's Guide p.22 178
TCP/IP AND UDP P ORT R EQUIREMENTS FOR CONVERSION VMware vCenter Converter Standalone User's Guide p.25 P2V - Depending on where you connecting.
Converter server to standalone VM or physical system - TCP - 445, 139, 9089;UDP - 137, 138 Converter to vCenter server - TCP 443 Converter Server to ESXi - TCP 902 Powered on Source machine to ESXi - TCP 443, 902 Linux VM uses additionally port 22 (SSH)
V2V - TCP 443, 445, 139; UDP 137, 138 CONVERT P HYSICAL W ORKLOADS
USING
VM WARE CONVERTER
Before launching conversion, make sure to disable Windows firewall (or allow File and Printer Sharing). Turn off simple sharing. The steps to convert a physical system can be resumed like this (but this is only one of the ways that's possible. Other ways client-server are possible as well): 1. Install VMware converter on the Window/Linux server and click Convert Machine > Powered On machine > This local machine 2. Select Destination type > choose VMware infrastructure VM > enter vCenter credentials > Put some meaningful name for your VM
3. Choose Cluster or host > Datastore > Virtual Machine Version > Click Next 4. Click the Advanced Link > chose the disk type of your choice (thick or thin). If you do not copy all disks and maintain layout the volume-based cloning is used. (at the block level).
179
You can also modify other resources which the VM do not need ... like delete some unwanted NICs, Windows services, or adjust the number of vCPUs and Memory... By default, Converter Standalone optimizes the disk partitions alignment. Optimizing the partitionalignment improves the performance of the destination virtual machine. (it's basically says that the process will align the VM to the LUN). So leave the box checked...
M ODIFY
SERVER RESOURCES DURING CONVERSION
Number of concurrent tasks - It's possible to modify the number of concurrent tasks by going to Administration > Maximum concurrent tasks. (1 to 12 concurrent tasks) But the 12 is by default and if your Converter server lacks resources you might want to lower down a bit of number of tasks taking place at the same time. Number of data connections per task - if you converting systems with multiple disks and volumes, it's possible to decrease the conversion time by cloning multiple disks and volumes simultaneously. Each data transfer uses a separate TCP connection. Check Administration > Data connections per Task. It's possible to synchronize changes after the first conversion has finished. It's because the source machine continues to generate data. So the delta changes can be synced and the source VM powered down...
180
INTERPRET
AND CORRECT ERRORS DURING CONVERSION
Check the following KB articles:
TIPS
Troubleshooting when vCenter Converter fails to complete a conversion of a physical or virtual machine. Testing port connectivity with Telnet (1003487) Best practices for using and troubleshooting VMware Converter (1004588) Troubleshooting a virtual machine converted with VMware Converter that fails to boot with the error: STOP 0x0000007B INACCESSIBLE_BOOT_DEVICE (1006295) Required VMware vCenter Converter 4.x/5.x ports (1010056) Collecting diagnostic information for VMware Converter (1010633) TCP and UDP Ports required to access VMware vCenter Server, VMware ESXi and ESX hosts, and other network components (1012382) VMware vCenter Converter is unable to see the disks when converting Windows operating systems (1016992) vCenter Standalone Converter errors when an ESXi 5.x host is selected as a destination: The access to the host resource settings is restricted. Use the management server as a destination (2012310) AND
TRICKS
FROM
ESX V IRTUALIZATION
AND
V LADAN... -:)
How-to disable SSL in VMware vCenter Converter Standalone to speed up P2V conversions How-to Reduce VMDK size: VMware Converter How to use VMware Converter to Synchronize changes when P2V (or V2V) VMware Converter Best Practices
VMware Tools and Guides
vSphere Installation and Setup Guide VMware vCenter Converter Standalone Guide vSphere Client / vSphere Web Client VMware vCenter Converter Standalone Client
VCP6-DCV OBJECTIVE 9.1 - CONFIGURE ADVANCED VSPHERE HA FEATURES VMware VCP6-DCV certification exam is kind of holy grail as it's an exam you can't fake. You have to know your stuff. Many folks also need to re-certify after expiring their VCP 4 or VCP 5. For current VCP5-DCV holders it's also possible to pass the VCP6-DCV delta exam, which has 45 questions only. Today's topc? VCP6-DCV Objective 9.1 Configure Advanced vSphere HA Features. Those study blog posts are covering topics and objectives from the blueprint from VCP 6 page and are here to help out with studying towards the VMware Certification Exam VCP6-DCV (Datacenter Virtualization). This exam validates you have the skills required to successfully install, deploy, scale and manage VMware vSphere 6. vSphere Knowledge
Explain Advanced vSphere HA settings Enable/Disable Advanced vSphere HA settings Explain how vSphere HA interprets heartbeats Interpret and correct errors during conversion Identify virtual machine override priorities Identify Virtual Machine Component Protection (VMCP) settings 181
E XPLAIN A DVANCED V S PHERE HA
SETTINGS
vSphere HA Advanced Options do not need to be changed in most environments.The HA advanced settings are applied at the cluster level. There is a very good VMware knowledge base article at http://kb.vmware.com/kb/2033250, which is based on vSphere 5.x but still relevant for vSphere 6. From vSphere 6.0 documentation center:
das.isolationaddress[...] - Sets the address to ping to determine if a host is isolated from the network. This address is pinged only when heartbeats are not received from any other host in the cluster. If not specified, the default gateway of the management network is used. This default gateway has to be a reliable address that is available, so that the host can determine if it is isolated from the network. You can specify multiple isolation addresses (up to 10) for the cluster: das.isolationaddressX, where X = 0-9. Typically you should specify one per management network. Specifying too many addresses makes isolation detection take too long. das.usedefaultisolationaddress - By default, vSphere HA uses the default gateway of the console network as an isolation address. This option specifies whether or not this default is used (true|false). das.isolationshutdowntimeout - The period of time the system waits for a virtual machine to shut down before powering it off. This only applies if the host's isolation response is Shut down VM. Default value is 300 seconds. das.slotmeminmb - Defines the maximum bound on the memory slot size. If this option is used, the slot size is the smaller of this value or the maximum memory reservation plus memory overhead of any powered-on virtual machine in the cluster. das.slotcpuinmhz - Defines the maximum bound on the CPU slot size. If this option is used, the slot size is the smaller of this value or the maximum CPU reservation of any powered-on virtual machine in the cluster. das.vmmemoryminmb - Defines the default memory resource value assigned to a virtual machine if its memory reservation is not specified or zero. This is used for the Host Failures Cluster Tolerates admission control policy. If no value is specified, the default is 0 MB. das.vmcpuminmhz - Defines the default CPU resource value assigned to a virtual machine if its CPU reservation is not specified or zero. This is used for the Host Failures Cluster Tolerates admission control policy. If no value is specified, the default is 32MHz. das.iostatsinterval - Changes the default I/O stats interval for VM Monitoring sensitivity. The default is 120 (seconds). Can be set to any value greater than, or equal to 0. Setting to 0 disables the check. Note: Values of less than 50 are not recommended since smaller values can result in vSphere HA unexpectedly resetting a virtual machine. das.ignoreinsufficienthbdatastore - Disables configuration issues created if the host does not have sufficient heartbeat datastores for vSphere HA. Default value is false. das.heartbeatdsperhost - Changes the number of heartbeat datastores required. Valid values can range from 2-5 and the default is 2. fdm.isolationpolicydelaysec - The number of seconds system waits before executing the isolation policy once it is determined that a host is isolated. The minimum value is 30. If set to a value less than 30, the delay will be 30 seconds. das.respectvmvmantiaffinityrules - Determines if vSphere HA enforces VM-VM anti-affinity rules. Default value is "false", whereby the rules are not enforced. Can also be set to "true" and rules are enforced (even if vSphere DRS is not enabled). In this case, vSphere HA does not fail over a virtual machine if doing so violates a rule, but it issues an event reporting there are insufficient resources to perform the failover. das.maxresets - The maximum number of reset attempts made by VMCP. If a reset operation on a virtual machine affected by an APD situation fails, VMCP retries the reset this many times before giving up das.maxterminates - The maximum number of retries made by VMCP for virtual machine termination. das.terminateretryintervalsec - If VMCP fails to terminate a virtual machine, this is the number of seconds the system waits before it retries a terminate attempt
182
das.config.fdm.reportfailoverfailevent - When set to 1, enables generation of a detailed per-VM event when an attempt by vSphere HA to restart a virtual machine is unsuccessful. Default value is 0. In versions earlier than vSphere 6.0, this event is generated by default. vpxd.das.completemetadataupdateintervalsec - The period of time (seconds) after a VM-Host affinity rule is set during which vSphere HA can restart a VM in a DRS-disabled cluster, overriding the rule. Default value is 300 seconds. das.config.fdm.memreservationmb - By default vSphere HA agents run with a configured memory limit of 250 MB. A host might not allow this reservation if it runs out of reservable capacity. You can use this advanced option to lower the memory limit to avoid this issue. Only integers greater than 100, which is the minimum value, can be specified. Conversely, to prevent problems during master agent elections in a large cluster (containing 6,000 to 8,000 VMs) you should raise this limit to 325 MB.
Note : Once one of the options is changed, for all hosts in the cluster you must run the Reconfigure HA task. Also, when a new host is added to the cluster or an existing host is rebooted, this task should be performed on those hosts in order to update this memory setting. E NABLE/DISABLE A DVANCED V S PHERE HA
SETTINGS
If you change the value of any of the following advanced options, you must disable and then re-enable vSphere HA before your changes take effect. You can use both clients (Windows C# client or vSphere Web client). You enable/disable always at the cluster level
Using the vSphere Web Client 1. 2. 3. 4. 5. 6. 7. 8.
Log in to VMware vSphere Web Client. Click Home > vCenter > Clusters. Under Object click on the cluster you want to modify. Click Manage. Click vSphere HA. Click Edit. Click Advanced Options. Click Add and enter in Option and Value fields as appropriate (see below).
1. Deselect Turn ON vSphere HA. 2. Click OK. 183
3. Wait for HA to unconfigure, click Edit and check Turn ON vSphere HA. 4. Click OK and wait for the cluster to reconfigure.
To get back to the defaults: remove fdm.cfg file on each hosts in the cluster OR reset the values to defaults on each host in the cluster.
E XPLAIN
HOW VS PHERE
HA
INTERPRETS HEARTBEATS
When configuring VMware High Availability (HA) cluster, you have the possibility to check as a secondary communication channel a datastore (or several ones), during the configuration wizard. VMware Datastore Hearbeating provides an additional option for determining if host is in failed state or not. In case the Master cannot communicate with a slave (don’t receives the heartbeat), but the heartbeat datastore answers, the server is still working. So if that’s the case, the host is partitioned from the network, or isolated. The Datastore heartbeat function helps greatly to determine the difference between host which failed and host that has just been isolated from others.
THE P URPOSE OF THE . VS PHERE -HA FOLDER This folder resides on shared datastore which is used as a secondary communication channel in HA architecture. This folder has several files inside, and everyone of them has different rôle (I don't think that's the required topic of the exam, but it's interesting to know in case you browse your shared datastore and see the folder inside):
host-xxx-hb files – those files are for the heartbeat datastore. The heartbeat mechanism uses the part of the VMFS volume for regular updates. Each host in cluster has it’s own file like this in the .vSphere-HA folder. protected list file – when you open this file, you’ll see a list of VMs protected by a HA. The master host uses this file for storing the inventory and the state of each VM. host-xxx-poweron files – this files role’s is to track the running VMs for each host of the cluster. The file is read by the master host which will know if a slave host is isolated from the network. Slave hosts uses this poweron file to tell the master host “hey, I’m isolated”. The content of this file reveals that there can be two states: zero or one. Zero = not isolated and One = isolated. If the slave host is isolated, master host informs vCenter. 184
The .vSphere HA folder is created only on datastores that are used for the datastore heartbeating. You shouldn’t delete or modify those files. The space used is minimum, depending on the VMFS version used and number of hosts that uses this datastore for heartbeating. It can be maximum about 3 Gb for on VMFS 3 and 2Mb on VMFS 5 (maximm and typical usage). The overhead isn’t big either.
Limitations of Datastore hearbeating:
No VSAN support
INTERPRET
AND CORRECT ERRORS DURING CONVERSION
This chapter is concerning VMware converter. It's been recently update to version 6.
TIPS
Troubleshooting when vCenter Converter fails to complete a conversion of a physical or virtual machine. Testing port connectivity with Telnet (1003487) Best practices for using and troubleshooting VMware Converter (1004588) Troubleshooting a virtual machine converted with VMware Converter that fails to boot with the error: STOP 0x0000007B INACCESSIBLE_BOOT_DEVICE (1006295) Required VMware vCenter Converter 4.x/5.x ports (1010056) Collecting diagnostic information for VMware Converter (1010633) TCP and UDP Ports required to access VMware vCenter Server, VMware ESXi and ESX hosts, and other network components (1012382) VMware vCenter Converter is unable to see the disks when converting Windows operating systems (1016992) vCenter Standalone Converter errors when an ESXi 5.x host is selected as a destination: The access to the host resource settings is restricted. Use the management server as a destination (2012310) AND
TRICKS
FROM
ESX V IRTUALIZATION
AND
V LADAN… -:)
How-to disable SSL in VMware vCenter Converter Standalone to speed up P2V conversions How-to Reduce VMDK size: VMware Converter How to use VMware Converter to Synchronize changes when P2V (or V2V) VMware Converter Best Practices 185
IDENTIFY
VIRTUAL MACHINE OVERRIDE PRIORITIES
You can customize settings for each VM in the cluster for VM restart priority, VMCP (see bellow), Host isolation response or VM monitoring. W HERE ? In the vSphere Web Client, browse to the vSphere HA cluster > Manage tab > Settings > Under Settings, select VM Overrides and click Add > Click the + button to select virtual machines to which to apply the overrides > OK.
If applied on the per-VM level, the settings now have more priority than the cluster settings and so they are different on every other VMs. At the same time you can apply DRS rules there (you can see on the image above I have some VMs which are not balanced automatically by DRS when Fully automated DRS is configured.
IDENTIFY V IRTUAL M ACHINE COMPONENT P ROTECTION (VMCP)
SETTINGS
HA was further enhanced with a function related to shared storage and it’s called VM Component Protection (VMCP). When VMCP is enabled, vSphere can detect datastore accessibility failures, APD (All paths down) or PDL (Permannent device lost), and then recover affected virtual machines by restarting them on other host in the cluster which is not affected by this datastore failure. VMCP allows the admin to determine the response that vSphere HA will make. It can be simple alarm only or it can be the VM restart on other host. The latter one is perhaps what we’re looking for. Let’s HA handle this for us….
Limitations:
VMCP does not support vSphere Fault Tolerance. If VMCP is enabled for a cluster using Fault Tolerance, the affected FT virtual machines will automatically receive overrides that disable VMCP. No VSAN support (if VMDKs are located on VSAN then they're not protected by VMCP). No VVOLs support (same here) No RDM support (same here)
HOW TO ENABLE ? At the cluster level. vSphere Client Select Hosts and clusters > Manage > vSphere HA > Edit > Protect against Storage Connectivity Loss. You must configure it on two places
186
1. Check the box “Protect against Storage Connectivity Loss” 2. Expand the “Failure conditions and VM response”
The second condition allows to specify what happens. There you have to specify 3 options: By default it does not restart the VM on another host so it’s important to do it. There you’ll see to options which you need to configure: 1. Response for Datastore with Permanent Device Lost (PDL) 2. Response for Datastore with All Path down (APD) – with this one you have two choses. To be more conservative or more aggressive. Basically it means to wait longer (or shorter) time in case the problem is resolved. As I mentioned at the beginning of my post, APD can be resolved (can be temporary outage) but PDL can’t. 3. Response for APD recovery after APD timeout – change it to “reset VMs” as by default its disabled.
187
All paths down (APD) - vSphere will restart the VM after user-configured timeout only if there is enough capacity. Action? Restart on a healthy host. Reset a VM if APD clears after APD timeout. Permannent device lost (PDL) - vSphere suppose that the device won’t show up back again and is “lost” due to hardware failure. Action? Terminate VM immediately and restart on a healthy host. If the Host Monitoring or VM Restart Priority settings are disabled, VMCP cannot perform virtual machine restarts. The VMCP settings has to be changed from their default values as by default the Response for APD recovery after APD is disabled. You can check settings at the cluster level, but also via the VM’s properties at the VM level by selecting the VM through vSphere Web client.
188
Those fine-grain options allows to react on unpredictable APD and PDL signals when using shared storage within your environment and give you significant insurance in case of connectivity problems to your shared storage.
LINKS
AND
TOOLS
vSphere Installation and Setup Guide vSphere Availability Guide What’s New in the VMware vSphere® 6.0 Platform vSphere Administration with the vSphere Client Guide vSphere Client / vSphere Web Client
VCP6-DCV OBJECTIVE 9.2 - CONFIGURE ADVANCED VSPHERE DRS FEATURES VMware VCP6-DCV certification exam might seems tough exam you can't fake. True, you have to know your stuff. But we like technology, we like VMware and so we like this exam. Many folks need to re-certify after expiring their VCP 4 or VCP 5. For new people who learning towards this exam I'm currently working on each one of the Objectives from the VMware VCP6-DCV blueprint. For current VCP5-DCV holders it's also possible to pass the VCP6-DCV delta exam, which has 45 questions only. Those study blog posts are covering topics and objectives from the blueprint from VCP 6 page and are here to help out with studying towards the VMware Certification Exam VCP6-DCV (Datacenter Virtualization). This exam validates you have the skills required to successfully install, deploy, scale and manage VMware vSphere 6. vSphere Knowledge
Identify Distributed Resource Scheduler (DRS) affinity rules Enable/Disable Distributed Resource Scheduler (DRS) affinity rules Identify Distributed Resource Scheduler (DRS) Automation levels Configure Distributed Resource Scheduler (DRS)Automation levels
IDENTIFY DISTRIBUTED R ESOURCE S CHEDULER (DRS)
AFFINITY RULES
The affinity rules controls the initial placement of VMs in DRS enabled clusters. From vSphere 6.0 documentation... Two Types:
VM-Host (Between a group of virtual machines and a group of hosts) - An affinity rule specifies that the members of a selected virtual machine DRS group can or must run on the members of a specific host DRS group. An anti-affinity rule specifies that the members of a selected virtual machine DRS group cannot run on the members of a specific host DRS group. VM-VM (Between individual virtual machines) - A rule specifying affinity causes DRS to try to keep the specified virtual machines together on the same host, for example, for performance reasons. With an antiaffinity rule, DRS tries to keep the specified virtual machines apart, for example, so that when a problem occurs with one host, you do not lose both virtual machines.
Requirements:
Licenisng. You have to be able activate vSphere HA and DRS cluster. Shared Storage - you need obviously shared storage to be able to activate HA, DRS, vMotion (yes vMotion as well). 189
VM-HOST AFFINITY RULE specifies an affinity relationship between a group of virtual machines and a group of hosts. There are 'required' rules (designated by "must") and 'preferential' rules (designated by "should".) A VM-Host affinity rule includes the following components:
One virtual machine DRS group. One host DRS group.
VM-VM AFFINITY RULE Whether VMs should run on the same host or be kept on separate hosts. With an anti-affinity rule, DRS tries to keep the specified virtual machines apart. You could use such a rule if you want to guarantee that certain virtual machines are always on different physical hosts. In that case, if a problem occurs with one host, not all virtual machines would be placed at risk.
E NABLE/DISABLE DISTRIBUTED R ESOURCE S CHEDULER (DRS)
AFFINITY RULES
Where? In the vSphere Web Client > Host and clusters > Manage TAB > VM/Host Rules > Add > Give your rule a name From the Type menu, select Virtual Machines to Hosts. Select the virtual machine DRS group and the host DRS group to which the rule applies.
If you select the Keep virtual machines together (third option in the image above), and so be able to use this rule you must first create VM/host Groups.... (option above close to the step 2 on the left hand side in the picture)
Must run on hosts in group - Virtual machines in VM Group 1 must run on hosts in Host Group A. Should run on hosts in group - Virtual machines in VM Group 1 should, but are not required, to run on hosts in Host Group A. Must not run on hosts in group - Virtual machines in VM Group 1 must never run on host in Host Group A. 190
Should not run on hosts in group - Virtual machines in VM Group 1 should not, but might, run on hosts in Host Group A.
Create Affinity Rule...
Create Anti-Affinity Rule...
IDENTIFY DISTRIBUTED R ESOURCE S CHEDULER (DRS) A UTOMATION
LEVELS
TIP: When DRS is disabled, the cluster’s resource pool hierarchy and affinity rules are not reestablished when DRS is turned back on. So if you disable DRS, the resource pools are removed from the cluster. To avoid losing the resource pools, instead of disabling DRS, you should suspend it by changing the DRS automation level to manual (and disabling any virtual machine overrides). This prevents automatic DRS actions, but preserves the resource pool hierarchy. There you can check the drop down menu and try to check the:
191
FT VMs can benefit from DRS (EVC must be enabled) to be initially placed at best. If FT VMs are on cluster with EVC disabled, then the FT VMs are given the DRS automation levels of "disabled". A FFINITY RULES AND FT VM S VM-VM affinity rule is applying to the primary VM only. Host-VM affinity rule applies to both primary and secondary VM.
CONFIGURE DISTRIBUTED R ESOURCE S CHEDULER (DRS)A UTOMATION
LEVELS
Where? Select Hosts and clusters > Manage > settings > vSphere DRS > Edit Then from the drop down menu choose the automation level you need.
Tools
vSphere Installation and Setup Guide vSphere Administration with the vSphere Client Guide What’s New in the VMware vSphere® 6.0 Platform vSphere Resource Management vSphere Client / vSphere Web Client
VCP6-DCV OBJECTIVE 10.1 - CONFIGURE ADVANCED VSPHERE VIRTUAL M ACHINE SETTINGS My VCP6-DCV Study Guide on my blog is getting crowdy with more and more objectives. Today's topic is Objective 10.1 - Configure Advanced vSphere Virtual Machine Settings. We'll be looking into some advanced options (with some tweaks) which are not only needed to pass the VCP6 exam, but are useful in real life. There is many tips and tricks I have published in the past for vSphere 5.x and vSphere 6. You can check the How-to articles, config/troubleshooting videos on vSphere 5.5/vSphere 6.x on those two Wordpress pages. But today's topic needs some more deep info concerning the VMs configuration parameters including settings like disabling VMs acceleration. 192
vSphere Knowledge
Identify available virtual machine configuration settings Interpret virtual machine configuration files (.vmx) settings Identify virtual machine DirectPath I/O feature Enable/Disable Advanced virtual machine settings
IDENTIFY
AVAILABLE VIRTUAL MACHINE CONFIGURATION SETTINGS
The configuration settings of a VM can be accessed through vSphere client and vSphere web client. We'll focus however on the settings through vSphere web client as this is the main client going forward even if it's still flash based and here and there the performance aren't optimal. We shall see HTML5 based client in the next update of VMware vSphere. So start vSphere web client and edit a single VM by going to Select VM > Edit settings > VM Options
General Options - Virtual machine name and location of the virtual machine configuration file and virtual machine working location. View or change the type and version of the guest operating system.
VMware Remote Console Options - Locking behavior and settings for simultaneous connections.
193
VMware Tools - Power Controls behavior, VMware Tools scripts, automatic upgrades, and time synchronization between the guest and host.
Power Management - Virtual machine Suspend behavior and wake on LAN.
Boot Options - You can set the boot delay and other cool stuff here. Virtual machine boot options. Add a delay before booting, force entry into the BIOS or EFI setup screen, or set reboot options.
194
Advanced Advanced virtual machine option:
Settings - Specify acceleration and logging settings. Debugging and statistic - Specify the level of debugging information that is being collected. Swap file location - Specify the swap file location. Configuration Parameters - View, modify, or add configuration parameters. Latency Sensitivity - Set a value for latency sensitivity.
Fibre Channel NPIV Virtual node and port World Wide Names (WWNs).
195
INTERPRET
VIRTUAL MACHINE CONFIGURATION FILES
(.VMX )
SETTINGS
The VMX settings can be changed through the VMs Options > Advanced configuration > Edit configuration
Usually the VMX file is in the same folder as the VM, but it can happen that the VMx files are stored elsewhere. To check where are the files located you can see it in general options where the path to the location of the virtual
196
machine configuration file shows. The path to the virtual machine working location appears in the VM Working Location text box. 1. The location of the VMX file 2. The location of the working location (VMDK,
VMs files:
IDENTIFY
VIRTUAL MACHINE
DIRECT P ATH I/O
FEATURE
VMdirect Path I/O - what's that? When enabled, the VM can access physical PCI functions with an I/O memory management unit (MMU). vSphere DirectPath I/O allows a guest operating system on a virtual machine to directly access physical PCI and PCIe devices connected to a host. Each virtual machine can be connected to up to six PCI devices. PCI devices connected to a host can be marked as available for passthrough from the Hardware Advanced Settings in the configuration tab for the host. L IMITATIONS (QUITE A FEW...):
No snapshot support - Snapshots are not supported with PCI vSphere Direct Path I/O devices No Hot Add - Hot adding and removing of virtual devices No Suspend and resume No Record and replay No FT - No Fault tolerance No HA - No High availability support either... DRS? - A kind of. DRS is limited to static..... The VM can be inside of DRS cluster, but cannot be vMotionned...
W HERE TO ENABLE ? Edit Settings > On the Hardware tab, click Select > select PCI Device and click Add > Select the passthrough device to connect to the virtual machine from the drop-down list > click Next.
197
D IRECT P ATH I/O VS SR-IOV SR-IOV offers performance benefits and tradeoffs similar to those of DirectPath I/O. DirectPath I/O and SR-IOV have similar functionality but you use them to accomplish different things. SR-IOV is beneficial in workloads with very high packet rates or very low latency requirements. Like DirectPath I/O, SR-IOV is not compatible with certain core virtualization features, such as vMotion. SR-IOV does, however, allow for a single physical device to be shared amongst multiple guests. With DirectPath I/O you can map only one physical function to one virtual machine. SR-IOV lets you share a single physical device, allowing multiple virtual machines to connect directly to the physical function. ENABLE SR-IOV ON A H OST P HYSICAL A DAPTER You must first enable it on the host level. In the vSphere Web Client, Select the host > Manage tab > Networking and select Physical adapters > Select the physical adapter > Edit > Select Enabled from the Status drop-down menu > OK > Restart the host.
once enabled at the host level, then it's accessible to the VM as a physical device... The VM must be turned off before starting to add the device. TO A SSIGN V IRTUAL F UNCTION AS SR-IOV P ASSTHROUGH A DAPTER TO A V IRTUAL M ACHINE 198
VM settings > Add new device > Network > from the Adapter type drop-down menu, select SR-IOV passthrough. Than expand the memory section, select reserve all guest memory (All locked) and click OK. I/O memory management unit (IOMMU) must reach all virtual machine memory so that the passthrough device can access the memory by using direct memory access (DMA).
E NABLE/DISABLE A DVANCED
VIRTUAL MACHINE SETTINGS
Well here we could list how to enable/disable different parameters, but I think it's pretty obvious as I added a screenshot for each of those values. Keep in mind that you're modifying config of individual VMs so to keep track of those changens on per-individual VM might be quite tedious, but it might be worthy the effort when seeking to gain a performance or troubleshoot an issue (activate logging). One of the features that we haven't discussed is the Change swap file location. As you know, when a VM is powered On, the ESXi host creates vmkrnel swap file which allows to back up the VMs RAM content. The default swap file (vmname.vswp) location is at the same location as the other VMs files.
Default - Use the settings of the cluster or host containing the VM VMs Directory - store the swap files in the same directory as the VM Datastore specified by host - you can store the swap files in the datastore specified by the host to be used for swap files. Note that using a datastore that is not visible to both hosts during vMotion might affect the performance of the vMotion operation for the VM(s).
CHANGE A S WAP FILE LOCATION H OW - TO ? vSphere web client Select VM > Edit settings > VM Options > Advanced
Tools and documentation for this topic
vSphere Installation and Setup Guide vSphere Administration with the vSphere Client Guide vSphere Virtual Machine Administration Guide vSphere Client / vSphere Web Client
199
VCP6-DCV OBJECTIVE 10.2 - CREATE AND MANAGE M ULTI-SITE CONTENT LIBRARY VCP6-DCV Study Guide is here to help you study towards VCP6-DCV (or delta) exam. Today's topic is new in vSphere 6. Feature called vSphere Content Library was not present in vSphere 5.5 and made its apparition in vSphere 6 during its release. VCP6-DCV Objective 10.2 - Create and Manage Multi-Site Content Library is today’s lesson. vSphere content library centrally manages virtual machine templates, ISO images, and scripts, and it performs the content delivery of associated data from the published catalog to the subscribed catalog at other sites. You can also check vSphere 6 page where you’ll find how-to’s, news, videos concerning vSphere 6.x. Last but not least, my Free Tools page where are the post popular tools for VMware and Microsoft. Daily updates of the blog are taking time, but we do it in the goal to provide a guide which is helpful for the community and folks learning towards VCP6-DCV certification exam. If you find one of those posts useful for your preparation, just share.. -:). Before we start I'd like to point a screenshot showing the ISO management... (must select the Other types button..)
OK, let's get started. vSphere Knowledge
Configure Content Library to work across sites Configure Content Library authentication Set/Configure Content Library roles Add/Remove Content Libraries
CONFIGURE CONTENT LIBRARY
TO WORK ACROSS SITES
Content Library lets you store and manage content from a central location. Admins can organize content logically into several libraries. Each individual library’s storage can be individually configured and managed. Admins can populate each library using several methods:
Clone existing templates in folders into Content Library (migrate your existing templates into Content Library with ease) Clone a VM as a template into Content Library Import from a web server Synchronize content from a vCloud Director catalog Upload contents from file system
Content library can be shared across multiple vCenter server systems. A VM template, vApp template or another type file is considered as a library item. Each item can contain several files (ex. OVF has several files .ovf, .vmdk, .mf, ...) however vSphere client shows only the .ovf through the content library. 200
What's the different types of content libraries? Local Libraries - Local library stores items in single vCenter environment. When you publish to the local library, other users from external vCenter servers can subcribe to this library. And to protect the access you can configure password authentication. Subscribed Libraries - When you subscribe to published library, then you create a subscribed library, which can be created at the same vCenter server as the original content library or in another vCenter server system. Pull the content - there is two different ways that you can pull the content out of vSphere content library: 1. Either you can download all the content of the published library after you create the subscribed library 2. You can download only metadata for the items in the subscribed library so you save space. Permission Requirements
User needs those permissions on the vCenter Server instance where you want to create the library:
Content library Create local library or Content library Create subscribed library
Note that Global permission must be assigned to the user.... Content libraries are not direct children of a vCenter Server system from an inventory perspective. The direct parent for content libraries is the global root. This means that if you set a permission at a vCenter Server level and propagate it to the children objects, the permission applies to data centers, folders, clusters, hosts, virtual machines, and so on, but does not apply to the content libraries that you see and operate with in this vCenter Server instance. To assign a permission on a content library, an Administrator must grant the permission to the user as a global permission. Global permissions support assigning privileges across solutions from a global root object.
201
See the diagram from VMware vSphere 6.0 Documentation...
CONFIGURE CONTENT LIBRARY
AUTHENTICATION
To enable authentication, select the library > Actions > Edit settings > Check the "Enable user authentication for access to this library".
202
S ET/C ONFIGURE CONTENT LIBRARY
ROLES
Content Library Administrator Content Library Administrator role is a predefined role that gives a user privileges to monitor and manage a library and its contents. A user who has this role can perform the following tasks:
Create, edit, and delete local or subscribed libraries. Synchronize a subscribed library and synchronize items in a subscribed library. View the item types supported by the library. Configure the global settings for the library. Import items to a library. Export library items.
You can clone this role or use this role as is and assign this role to the user that shall manage the content library.
A DD /R EMOVE CONTENT LIBRARIES To Add a Content Library: (to create) vCenter Inventory Lists > Content Libraries > Click the Objects tab > Click the Create a New Library icon Give it some meaningful name..
Click next to follow the assistant and choose one of the options...
Then again continue with the next button and choose a storage... 203
Hit next and finish. To Delete a Content library:
vSphere Web Client > vCenter Inventory Lists > Content Libraries > Select library from the list > Actions > Delete > Confirm
Synchronize Library Items: Web Client > vCenter Inventory Lists > Content Libraries > Select a subscribed library from the list, and click the Related Objects tab. > Synchronize the item you want to use. On the Templates tab, right-click a VM or a vApp template, and select Synchronize Item > On the Other Types tab, right-click an item, and select Synchronize Item.
204
After synchronization completes, the item content and metadata are downloaded to the backing storage of the subscribed library, and in the Related Objects tab the value for the item in the Stored Content Locally column changes to Yes. Tools
vSphere Installation and Setup Guide vSphere Administration with the vSphere Client Guide What’s New in the VMware vSphere® 6.0 Platform vSphere Virtual Machine Administration Guide vSphere Client / vSphere Web Client
VCP6-DCV OBJECTIVE 10.3 - CONFIGURE AND MAINTAIN A VCLOUD AIR CONNECTION Last chapter in the big VCP6-DCV series today where we'll learn about vCloud Air and connection through vCenter: VCP6-DCV Objective 10.3 - Configure and Maintain a vCloud Air Connection is the title of the objective. You will learn details on the requirements to setup vCloud Air connection, configuration of vCenter server connection to vCloud Air. The whole VCP6-DCV Study Guide page. Register for the VCP6-DCV exam here. In addition, you might want to visit our Free Tools page or vSphere 6 page for latest updates and news concerning vSphere 6 or free tools for IT administrators. vSphere Knowledge
Identify vCenter Server and vCloud Air Connection requirements Configure vCenter Server connection to vCloud Air Identify connection types Configure replicated objects in vCloud Air Disaster Recovery service
IDENTIFY V C ENTER S ERVER
AND V C LOUD
A IR CONNECTION
Setting up the vCloud Air DR service is done through VMware web site. 205
REQUIREMENTS
Requirements:
vSphere 5.5 or later (6.0 recommended) My VMware Account Firewall Ports - 10000 to 10010 of ESXi hosts are open for outgoing traffic . The required ports are open automatically when you install a VIB on each supported ESXi host in the environment where the vSphere Replication appliance is deployed
Compatible products - vSphere replication appliance 6.0, ESXi 5.0, 5.1.x, 5.5.x or 6.0, vCenter 6.0, vSphere Web client 6.0 Roles, permissions to the cloud - usually assigned through vCloud Air UI after successfully installing vSphere replication. Check that you have VR up and running in your environment Verify that the Disaster Recovery to Cloud service is enabled in the target cloud organization Configure connection to the cloud organization.
CONFIGURE V CENTER S ERVER
CONNECTION TO V C LOUD
A IR
vSphere replication to the cloud p.12 When you create a connection to the cloud, the vCloud Tunneling Agent in the vSphere Replication appliance creates a tunnel to secure the transfer of replication data to your cloud Organization. When a tunnel is created, the vCloud Tunneling Agent opens a port on the vSphere Replication appliance. ESXi hosts connect to that port to send replication data to a cloud organization. The port is picked randomly from a configurable range. The default port range is 10000-10010 TCP. In vSphere Replication, you must establish a connection to your cloud provider before you configure replications to cloud. The vSphere Replication UI requires you to enter the cloud provider address and the cloud organization name. Click VR icon in the vSphere web client > On the Home TAB click the Manage button.
206
The Manage tab should be preselected > click Target Sites > and then click the Connect to a Cloud Provider icon.
A pop-up windows shows up where you'll be able to enter the connection details. The information that you need is included in the subscription email that you receive from VMware vCloud Air.
On the Connection settings page, type the address of your cloud provider, the organization name, and credentials to authenticate with the cloud. By default, vSphere Replication uses these credentials to establish a user session to the cloud and for system monitoring purposes. To enable system monitoring, these credentials will be stored in the vSphere Replication appliance, unless you select to use another user account for system monitoring. (Optional) If you do not want to store the credentials that you used for authentication, select the Use a different account for system monitoring check box, and type the credentials to be used for system monitoring. These credentials are encrypted and stored in the vSphere Replication database. Click Next > The Connect to a Cloud Provider wizard displays a list of virtual data centers to which you can connect. If a virtual data center is already connected to the vCenter Server, that data center does not appear in the list. From the list of virtual data centers, select a target for the connection and click Next > Finish
207
You'll need the Cloud provider address and Organization name. You can find those information when you Connect to your vCloud Air portal > The Replication tab.
IDENTIFY
CONNECTION TYPES
There is two types of credentials when you create a connection to the target virtual data center (VDC):
Connection credentials - used for authentication within the cloud organization. The priviledges are managed by cloud provider. Few rights are required: ManageRight, ViewRight, View Organization Networks, View Organizations, View organization VDC, View Organization VDC. Credentials to the cloud are needed for each target site, once per user session. When the authenticated user session to a target site expires, users are prompted to input their credentials again System monitoring credentials - used for system runtime, so the source and destination sites can communicate together. Those credentials are stored in the VR appliance on the source site. The user name must have VR role with few priviledges: ManageRight, ViewRight, View organization Networks, View Organizations, View Organization VDCs
CONFIGURE
REPLICATED OBJECTS IN V C LOUD
A IR DISASTER R ECOVERY
SERVICE
The installation and deployment of VR has been detailed in the Objective covering vSphere Data protection - VCP6DCV Objective 6.1 – Configure and Administer a vSphere Backups/Restore/Replication Solution. See the details of the deployment there. vCloud air DR user's guide p. 19 You can configure replicate single VM or multiple VMs at a time. The same way as configuring replication between hosts in your On Premise environment. You will be able to set a recovery point objective (RPO) to determine the maximum data loss that you can tolerate. For example, an RPO of 1 hour seeks to ensure that a virtual machine loses the data for no more than 1 hour during the recovery. vSphere Replication guarantees crash consistency amongst all the disks that belong to a virtual machine. (VSS checkbox) NOTE: By default, when you configure a virtual machine for replication to cloud, its NICs and MAC addresses are copied automatically to the target site as part of the provisioning of the placeholder virtual machine. If the test network is not isolated from the production network and these networks have common routing, a test recovery of a replicated virtual machine might result in duplicate MAC addresses in your virtual data center. You can check p.16 of the vSphere Replication to the Cloud document for details how to disable that. When you configure replication by using vSphere Replication at your source site, the Disaster Recovery service creates placeholder virtual machines in vCloud Air which represent the virtual machines at your source site.
208
The placeholders are VM for which you are testing recovery, and virtual machines recovered to the cloud. A placeholder virtual machine appears in the VM's tab after the initial full synchronization of replication data from the source site successfully completes. Use the Virtual Machines tab to test recovery and recover the virtual machines to the cloud in the event your source site is unavailable. The status of each placeholder determines what actions are available for that virtual machine represented. After you test a recovery or recover a virtual machine to the cloud, the Disaster Recovery service replaces the placeholder with a test or production virtual machine respectively. You can enable multiple point in time recovery snapshots.
If you enable multiple point in time (MPIT) setting, you can use previous replication points for better control on failover. It allows you to:
Set up to 24 previous restore points Choose your restore point Restore up to 24 days previous replication points (dependent on your RPO setting)
209
Tools
vSphere Installation and Setup Guide vSphere Administration with the vSphere Client Guide vSphere Networking Guide VMware vCloud Air – Disaster Recovery User’s Guide vSphere Client / vSphere Web Client
210
