VMWARE Troubleshoot

vMotion fails with the error: Migrate_SetFailure: Failed waiting for data. Error bad0004. Busy

vMotion fails with the error: Migrate_SetFailure: Failed waiting for data. Error bad0004. Busy Symptoms   

Unable to vMotion the virtual machine When you attempt to vMotion the virtual machine, the task begins, but fails at 10% In the vmware.log file of the virtual machine, you see messages similar to: Dec 06 16:41:29.167: vmx| Migrate_SetFailure: Failed waiting for data. Error bad0004. Busy. Dec 06 16:41:29.167: vmx| Migrate: cleaning up migration state. Dec 06 16:41:29.167: vmx| MigrateSetState: Transitioning from state 11 to 0. Dec 06 16:41:29.167: vmx| Msg_Post: Error Dec 06 16:41:29.167: vmx| [vob.vmotion.resolve.swap.all.files.failed] vMotion migration [c0a8027c:1291653666313618] failed to find a valid swapfile. Dec 06 16:41:29.167: vmx| [msg.moduletable.powerOnFailed] Module Migrate power on failed.

Resolution This issue may occur due to the slight difference in the type of swap files in ESX 4.1 and earlier ESX versions. To resolve this issue, you must modify the swap type setting on both the source and target host. To modify the setting:

1. 2. 3. 4. 5. 6. 7.

Launch the vSphere Client and log in to your vCenter Server. Select the source ESX host and then click the Configuration tab. Click Software > Advanced Settings > Migrate. Under the Migrate options, locate the line containing Migrate.VMotionResolveSwapType. It will be set to the default value of 1. Change the value to 0. Click OK. Repeat Steps 2 and 3 for the destination host

VMotion fails after a third-party security tool performs a port scan of the ESX/ESXi hosts Symptoms  

VMotion fails after a third-party security tool (such as IBM Internet Security Systems) performs a port scan of the ESX or ESXi hosts. You see errors similar to:

cpu1:1086) Migrate: 2250 Error with migration listen socket, shutting down: I/O error. A general system error occurred; timed out waiting for migration data 

The vmkernel.log contains messages similar to: May 1 21:20:45 ESXsrvr vmkernel: 11:22:29:39.358 cpu13:1915)World: vm 1923: 901: Starting world migSendHelper-1916 with flags 1 May 1 21:20:45 ESXsrvr vmkernel: 11:22:29:39.358 cpu13:1915)World: vm 1924: 901: Starting world migRecvHelper-1916 with flags 1 May 1 21:20:45 ESXsrvr vmkernel: 11:22:29:39.364 cpu1:1086)MigrateNet: vm 1086: 854: Accepted connection from May 1 21:21:05 ESXsrvr vmkernel: 11:22:29:59.642 cpu12:1916)Migrate: 7309: 1241227232551280: Another pre-copy iteration needed with 30737 modified pages (last = -1) May 1 21:21:07 ESXsrvr vmkernel: 11:22:30:02.092 cpu10:1916)Migrate: 7309: 1241227232551280: Another pre-copy iteration needed with 17783 modified pages (last = 30737) May 1 21:21:09 ESXsrvr vmkernel: 11:22:30:03.938 cpu9:1916)Migrate: 7304: 1241227232551280: Stopping pre-copy: Not enough forward progress (Modified pages 17783 vs. 22217) - stopping pre-copy May 1 23:32:52 ESXsrvr vmkernel: 12:00:41:45.964 cpu1:1086)MigrateNet: vm 1086: 854: Accepted connection from May 1 23:32:52 ESXsrvr vmkernel: 12:00:41:45.964 cpu1:1086)WARNING: MigrateNet: vm 1086: 865: Couldn't set nodelay option on socket May 1 23:32:52 ESXsrvr vmkernel: 12:00:41:45.964 cpu1:1086)ALERT: Migrate: 2250: Error with migration listen socket, shutting down: I/O error May 1 23:32:52 ESXsrvr vmkernel: 12:00:41:45.964 cpu1:1086)Migrate: 2312: Exit requested...



The Hostd.log contains messages similar: [2009-05-01 23:31:41.190 'App' 22911920 error] SSLStreamImpl::BIORead ( A6A2D10) failed: Connection reset by peer [2009-05-01 23:31:41.190 'App' 22911920 error] SSLStreamImpl::DoServerHandshake ( A6A2D10) SSL_accept failed with BIO Error [2009-05-01 23:31:41.190 'Proxysvc' 22911920 warning] SSL Handshake on client connection failed for peer , error=SSL Exception: BIO Error [2009-05-01 23:32:22.994 'App' 21588912 error] SSLStreamImpl::DoServerHandshake ( A6B9AA8) SSL_accept failed with Unexpected EOF [2009-05-01 23:32:22.994 'Proxysvc' 21588912 warning] SSL Handshake on client connection failed for peer , error=SSL Exception: Unexpected EOF [2009-05-01 23:32:52.085 'ha-eventmgr' 130374576 info] Event 271 : Issue detected on ESXsrvr.mydomain.com in ha-datacenter: Migrate: 2250: Error with migration listen socket, shutting down: I/O error (12:00:41:45.964 cpu1:1086)

Resolution This issue is resolved in ESX/ESXi 4.0 Update 2. For more information, see vSphere 4 download page. This issue is resolved in ESX and ESXi 3.5. For more details, see KB 1026126 (ESX) at http://kb.vmware.com/kb/1026126, and KB 1026138 (ESXi) at http://kb.vmware.com/kb/1026138.

This issue might occur if a network port-scanner-process attempts to engage VMotion migration port (8000) on the ESX or ESXi host. On ESX/ESXi 3.5.x, you must disable and then re-enable VMotion on the ESX/ESXi host. The workaround provided to resolve the issue was: To disable and re-enable VMotion:

1. 2. 3. 4. 5. 6. 7.

Select the ESX/ESXi host in the VI Client. Select Configuration > Advanced Settings > Migrate > Migrate.enabled. Change the value of Migrate.enabled setting from 1 to 0. Click OK. Select Configuration > Advanced Settings > Migrate > Migrate.enabled. Change the Migrate.enabled setting from 0 to 1. Click OK.

To prevent VMotion from failing, you must exclude port 8000 in your port scanning software. Note: A VMotion network should never be accessible by untrusted sources. You must isolate the management network as described in the VMware Infrastructure 3 Security Hardening Guide

Failed vMotion, host busy, White Spaces and Directory Names. So recently I came across and issue where a "space" at the end of a VM's directory causes it to fail to vMotion. The error is typically cryptic stating that the host is too busy. If you try to browse the directory using the vsphere client it looks empty. You also can't rename the directory using the vsphere. For us the problem came about when we were creating VM's. We typically created quite long machine names in vsphere. "machine name - OS - description. Ask you can see we tend to add Tue description of the vm to the name. Now what vSphere does is take the first 32 characters and use that as the directory name. If the 32nd character is a "space"..... well you get the picture. I solved the issue by using the vMA (this can also be done using the ESX console). I use the vMA a fair amount and have all of our nfs shares mounted in /mnt. I would recommend this as it just good to have that kind of access to the virtual machine files.

Anyway the fix: 1. Shutdown and remove the VM from inventory.( Don't delete from disk!!!). 2. Using the vMA browse to the location of the folder. Rename through folder using the mv command. mv "folder with space " "folder with space". 3. Renregister Tue vm with sphere or the ESX server. If anybody else has had this problem and solved it drop me a line.

vMotion fails with general system error Posted by Monique on July 11, 2011

Not long ago I wanted to vMotion a virtual machine, it failed with a general system error at 80%. First thing that crossed my mind was, it must have a CD/DVD device attached. Wich is usually the reason for failing vMotion. So I editted the virtual machine settings from “client device” to “host device” and back. Checked to make sure there were no VMware tools upgrades that got stuck and tried it again. Still, I got the same error. I vMotionned some other machine off the same host, just to make sure vMotion was still viable. That worked, so vMotion on the cluster was still functional. It had to be something else related to this particular virtual machine. I checked all it’s settings, but didn’t notice anything unusual. I then checked the vmx log files of the virtual machine and the hostd log files, to see if I could find anything out of the ordinary in the logs. Among other things I found these entrances in the logs, wich struck me as odd: Vmon_MigrateWriteGotData: Limit Exceeded MigrateWrite failed Could be an instance of a bug 49917 When I Googled for “migration write data limit exceeded” I found this VMware KB, VMware KB1011971. It states that this issue may occur on ESX(i) clusters earlier than 4.0 update 1, when the Video RAM of a virtual machine is greater than 30 MB. Since this machine was running on one of our (soon to be extinct) ESX 3.5 clusters, I gathered this might be it. I checked the virtual machine settings again. This time for the Video RAM settings and it turned out the machine had 32 MB of Video RAM. After checking with the client, I brought the VM down, editted it’s Video RAM and brought it back online. This time it vMotionned without any problem. Once again, one of life’s little annoyances resolved

------------------------------

Creating a snapshot for a virtual machine fails with the error: File is larger than maximum file size supported Symptoms When creating a snapshot for the virtual machine, you experience these symptoms:



In the vSphere Client, you see the error: File is larger than the maximum size supported by datastore



In the hostd log file, you see the error: Snapshot guest failed: The file is too big for the filesystem.



In the vmware.log file of the virtual machine, you see an error similar to: vmx| FILE: File_VMFSSupportsFileSize: Requested file size (554051831808) larger than maximum supported filesystem file size (274877906944) vmx| DiskLibCreateCustom: if your disk is on VMFS, you may consider increasing the block size. vmx| DISKLIB-LIB : Failed to create link: The destination file system does not support large files (12) vmx| SNAPSHOT: BranchDisk: Failed to create child disk '/vmfs/volumes/uuid/vmname/vmname-000001.vmdk' : The destination file system does not support large files (12) vmx| SNAPSHOT: SnapshotBranch failed: The destination file system does not support large files (5). vmx| [msg.checkpoint.save.fail2.std3] Error encountered while saving snapshot. vmx| The destination file system does not support large files.

Cause This failure occurs when the snapshot file at its maximum size would be unable to fit into a datastore. Starting with version 4.0, ESX and ESXi will compare the maximum size of a snapshot redolog file with the maximum size of files on the datastore. The redolog file may not work correctly once it reaches the maximum size of the datastore. If the file could grow beyond the maximum size, ESX cancels the Create Snapshot operation and displays this error instead. Note: This check does not occur on ESX/ESXi 3.5 and earlier. On these versions, a snapshot will be created even if there is insufficient space to store a full-size redolog.

Resolution

Maximum file size Compare the base disk size of the virtual machine against the block size of the datastore which contains the working directory of the virtual machine. By default, the working directory contains the virtual machine's .vmx configuration file. The maximum file size differs among versions of ESX/ESXi, and among version of VMFS. If you experience this error even after confirming that the snapshot files can fit on the datastore, proceed to the Calculating the overhead required by snapshot files section in this article.

Note: A virtual machine on NFS or VMFS has a maximum virtual disk size of 2048 GB - 512 Bytes, the same as the maximum in each of these tables.

ESX/ESXi 4.0 with VMFS3

On ESX/ESXi 4.0, the maximum file size corresponds to the block size of the VMFS3 datastore:

Block Size Maximum File Size 1 MB

256 GB - 512 Bytes

2 MB

512 GB - 512 Bytes

4 MB

1024 GB - 512 Bytes

8 MB

2048 GB - 512 Bytes

ESX/ESXi 4.1 and ESXi 5.0 with VMFS3, or VMFS3 upgraded to VMFS5

On ESX/ESXi 4.1 and ESXi 5.0 using a VMFS3 datastore, or an ESXi 5.0 host using a VMFS5 datastore upgraded from VMFS3, the maximum file size corresponds to the block size of the VMFS datastore:

Block Size Maximum File Size 1 MB

256 GB

2 MB

512 GB

4 MB

1024 GB

8 MB

2048 GB - 512 Bytes

ESXi 5.0 with VMFS5

On ESXi 5.0 and newly formatted VMFS5, a standard 1 MB block size is available:

Block Size Maximum File Size 1 MB

2048 GB - 512 Byte

Moving files to accommodate space requirements To resolve this issue you can either change the location of the virtual machine configuration files or change the workingDir to a datastore with enough block size. The workingDir is the location where the snapshots are created, By default, workingDir contains the virtual machine's .vmx configuration file. To change the workingDir directory to a datastore with enough block size, see Creating snapshots in a different location than default virtual machine directory (1002929). To move the virtual machine's disks and/or configuration files you can use Storage vMotion or Cold migration with relocation of files. For more information, see:   

vSphere 5.0: Migrating Virtual Machines section of the vCenter Server and Host Management guide. vSphere 4.1: Migrating Virtual Machines section of the vSphere Datacenter Administration guide. vSphere 4.0: Migrating Virtual Machines section of the vSphere Basic System Administration guide.

If the virtual machine already has snapshots, some procedures may not work or may try to create a snapshot. The following table lists the requirement for the various procedures:

Procedure

Requirements

Storage vMotion

Virtual machine must not have snapshots on ESX/ESXi 4.1 hosts and earlier, may

have snapshots on ESXi 5.0 and later. Cold migration with Virtual machine may have snapshots. The source and destination hosts must be relocation of files running ESX/ESXi 3.5 or later. Change workingDir

Virtual machine may have snapshots. When new snapshots are created, new redologs will be placed in the workingDir directory.

Hot clone

Virtual machine may have snapshots, but the snapshot hierarchy must be fewer than 31 snapshots deep. Hot clone of a virtual machine creates a snapshot on the source at the beginning of the process, and deletes the snapshot when complete.

Cold clone

Virtual machine may have snapshots. Cloning the virtual machine creates a new virtual machine with the same content as the original virtual machine, but without snapshots.

vMotion to ESX/ESXi Virtual machine may have snapshots. Virtual machine must use hardware version 4. ESX/ESXi 3.5 does not perform the check described here and allows creation of 3.5 snapshots on virtual machines.

Calculating the overhead required by snapshot files The failure depends on the size of the virtual disk. All virtual machines having disks with a maximum supported size by VMFS may experience this error. Overhead for the snapshot is roughly about 2GB for a disk size of 256GB. If snapshots are to be used, consider the overhead while deciding the size of the disks.

Maximum VMDK size Maximum Overhead Maximum size less overhead 256 GB - 512 B

~ 2 GB

254 GB

512 GB - 512 B

~ 4 GB

508 GB

1024 GB - 512 B

~ 8 GB

1016 GB

2048 GB - 512 B

~ 16 GB

2032 GB

Note: VMware recommends that you to create virtual disks of size less than the maximum minus the overhead, to enable use of features like Snapshot, Clone, and Independent-nonpersistent disks. Additional Information When performing a storage vMotion, you may encounter the error: Moving a virtual machine that has snapshots is not supported when the virtual machine has disks placed outside of its home datastore.

For more information on the maximum file size per VMFS block size, see the Configuration Maximums document for your version of ESX/ESXi.

Tags cannot-take-snapshot create-snapshot create-snapshot-fails create-snapshots take-snapshot createdatastore

---------------------------

Committing snapshots when there are no snapshot entries in the snapshot manager Details The snapshot manager shows no snapshots but there are delta files present. One or more sets of 00000X.vmdk and -00000X-delta.vmdk files are in the directory with the virtual disk. The .vmx file points to one of the -00000X.vmdk files, usually the highest numbered file, indicating that the snapshot file is in use.

Note: This article applies only to ESX/ESXi 3.x and 4.x. For information about ESXi 5.0, see Consolidating snapshots in vSphere 5 (2003638).

Solution Most of the snapshot related issues have improved with changes to the Delete All snapshot process in the patch releases for ESX/ESXi 3.5 and 4.0. For more information, see:   

VMware ESX 3.5, Patch ESX350-201006401-SG: Updates VMkernel, VMware Tools, hostd, VMX, VMnix (1022899) VMware ESXi 3.5, Patch ESXe350-201006401-I-SG: Updates firmware (1020052) VMware ESX 4.0, Patch ESX400-201006201-UG: Updates the VMware ESX 4.0 Core and CIM components (1017721)



VMware ESXi 4.0, Patch ESXi400-201006201-UG: Updates Firmware (1017739)

Resolution Confirm that the virtual machine is not pointing to the base disk. Open the virtual machine configuration file ( .vmx) or edit the settings of the virtual machine and see if any of the virtual disks are using a 00000X.vmdk file. If no disks are using -00000X.vmdk, this virtual machine is not using any of these files. Although unlikely, it is possible that another virtual machine is storing its snapshots in this directory. Check the other virtual machines. If none of them refer to these files, they can be safely erased.

Typically, the file is in use by the virtual machine. When a snapshot is deleted, any additional files in the hierarchy that are not identified by the snapshot manager are included in the commit process. Creating a new snapshot and deleting it clears the entire hierarchy. This means that all snapshot files on the virtual machine are committed, then deleted.

A bit of free space is required to create the new snapshots. If the virtual machine needs to remain running, more space must be allowed for, as the new snapshot grows (accepting new changes to the virtual disks) as the older snapshots commit.

To commit all snapshots by using the Virtual Infrastructure Client:

1. Take a Snapshot. For more information, see the Take a Snapshot section of the VMware vSphere Online Library .

2. Delete all Snapshots. For more information, see the Delete all Snapshots section of the VMware vSphere Online Library . On an ESXi host, to commit all snapshots using the command line:

1. Log in to the ESXi host as root via the console or an SSH session. For more information about SSH, see see Tech Support Mode for Emergency Support (1003677) or Using Tech Support Mode in ESXi 4.1 (1017910). Note: The following commands can also be executed remotely using the vSphere Command Line for both ESX and ESXi hosts. For more information, see vSphere Command Line Interface documentation.

2. Run this command to get a list of virtual machines and the VMID for each virtual machine: vim-cmd vmsvc/getallvms

Output similar to the following is displayed: Vmid Name File Guest OS Version Annotation 1 vm1 [datastore1] vm1/vm1.vmx windows7Server64Guest vmx-08 3 testvm [iscsi1] testvm/testvm.vmx winNetDatacenterGuest vmx-08 Make a note of the VMID for the specific virtual machine.

3. To verify if the snapshot exists, run this command and check the Snapshot Name, Snapshot Created On, and Snapshot State: vim-cmd vmsvc/snapshot.get [VMID] You see an output similar to: Get Snapshot: |-ROOT --Snapshot Name : Test --Snapshot Desciption : --Snapshot Created On : 7/27/2011 13:49:55 --Snapshot State : powered on

4. Run this command to create a new snapshot:

vim-cmd vmsvc/snapshot.create [VmId] [snapshotName] [snapshotDescription] [includeMemory] [quiesced] For example, to create a snapshot on VM testvm: vim-cmd vmsvc/snapshot.create 3 snapshot1 snapshot 0 0 Note: You can use any name you like. The name appears in the snapshot manager.

5. Run this command to remove all snapshots:

vim-cmd vmsvc/snapshot.removeall [VMID]

On an ESX host, to commit all snapshots by using the command line:

1. Log in to the ESX host as root via the console or an SSH session. For more information about SSH, see Unable to connect to an ESX host using Secure Shell (SSH) (1003807). Note: The following commands can also be executed remotely using the vSphere Command Line for both ESX and ESXi hosts. For more information, see vSphere Command Line Interface documentation.

2. Type vmware-cmd -l and press Enter. The output appears similar to: /vmfs/volumes/UUID/VMNAME/VMNAME.vmx

3. Type vmware-cmd /vmfs/volumes/UUID/VMNAME/VMNAME.vmx hassnapshot and press Enter to confirm that there is a snapshot. If the output displays a value of 1, a snapshot is present. If the output displays a value of 0, there is no snapshot present.

4. Type vmware-cmd /vmfs/volumes/UUID/VMNAME/VMNAME.vmx createsnapshot and press Enter to create a new snapshot. For example, the command vmware-cmd /vmfs/volumes/UUID/VMNAME/VMNAME.vmx createsnapshot "test" "" 0 0 makes a snapshot without memory, quiescing, or a description called test . Note: You can use any name you like. The name appears in the snapshot manager. For more

information about the syntax of the vmware-cmd command, see vSphere Command Line Interface documentation.

5. Type vmware-cmd /vmfs/volumes/UUID/VMNAME/VMNAME.vmx removesnapshots and press Enter to remove the snapshot.

Additional Information  





The remove snapshot process can take a long time to complete if the snapshots are large. A file . vmsd may interfere with the creation of the snapshot in step 4 if the memory snapshot was left behind. For more information, see Understanding virtual machine snapshots in VMware ESX (1015180). If you entered the removesnapshots command via an SSH tool (such as PuTTY or secureCRT) the SSH session must be left open. Closing the SSH program aborts the process. If leaving an SSH session open for an extended time is unacceptable, run the command from the physical console. The commit process has no progress that you can follow. As long as the date on the files continues to update, the process is working. Also, if the virtual machine is off, you can use the file * command to see if any files are in use. If any of the files return the error message can't read `filename' (Device or resource busy)they are locked by the VMkernel. The commit process is actively occurring to those files.

When the commit has completed successfully, there are no -00000X.vmdk or -00000X-delta.vmdk files left unless they were not part of the snapshot tree. These files can be deleted. To confirm the commit succeeded, view the .vmx file and verify that virtual disks are now pointing to a base disk (-flat.vmdk ). If the .vmx contains a disk that is still pointing to a snapshot file, the commit process failed. If the attempt was made with the virtual machine running, plan an outage and try again with the virtual machine off. If that does not work, the virtual disk must be cloned using vmkfstools -i. The source file name is the current active -00000X.vmdk as identified in the .vmx file. When the clone is complete, point the virtual machine to use the newly cloned disk. The original base disk and snapshot tree can be deleted. For more information on consolidating a snapshot from the command line using cloning, see Consolidating Snapshots (1007849). As an alternative to using the command line, you can use VMware Infrastructure (VI) Client to perform the clone operation on the virtual machine. The cloned virtual machine retains the content of the associated snapshot disks at the time of cloning. When the virtual machine has been cloned successfully, test the operation of the resulting cloned virtual machine and decommission the previous virtual machine. For more information on cloning a virtual machine using VI Client, see Cloning Virtual Machines. Note: Cloning using VI Client requires VirtualCenter and all applicable licenses. Cloning a virtual machine using VI Client does not allow you to select individual virtual disks connected to the virtual machine. Relative to using the vmkfstools command to perform the same operation on a single virtual disk, consolidating snapshots using VI Client clones all disks and may require more disk space.

Workaround If you are still not able to consolidate the snapshots and if a restore from backup is not possible, you can use VMware vCenter Converter Standalone to convert the virtual machine into a new virtual machine:

1. 2. 3. 4. 5. 6.

Review the Release Notes. Check the System Requirements as outlined in the Converter Standalone User's Guide. Within the virtual machine, stop all I/O intensive processes. Install VMware vCenter Converter Standalone into the virtual machine. Start VMware vCenter Converter Standalone. In the source dialog select Physical Machine and follow the instructions on the screen. For a detailed description review the Converter Standalone User's Guide. 7. Once the conversation finished successfully, you should test the virtual machine created:

8. a. Check the virtual machine configuration by right clicking on the virtual machine and click Edit Settings.

b. Make sure that the network cards are not connected. c. Power on the virtual machine and verify that everything is proper. 9. If everything is proper, power off the virtual machine with the snapshots. 10. In the new virtual machine, set the network cards to Connected at power on. Note: This works because the Converter Standalone does not know the virtual disk structure. It sees only that what the operating system sees like a physical machine. You should remember that the virtual machine conversion takes a long time because the conversion is done over the network.

See also  

Consolidating snapshots (1007849) Determining if there are leftover delta files or snapshots that VMware vSphere or Infrastructure Client cannot detect (1005049)

For translated versions of this article, see: 

Español: Cómo hacer Commit a snapshots cuando estos no aparecen en el snapshot manager (1000636)

ESXi 4.1 virtual machines in an invalid state fail to power on with the error: FoundryVMDirectlyOpenSocketToVMX: Failed to create socket pair Symptoms When using VMware ESXi 4.1, you may experience one or more of these symptoms:     

Unable to connect to the MKS Unable to power on virtual machines Powering on virtual machines reports some virtual machines in an invalid state Power state of the virtual machine is reported incorrectly in the vCenter Server/ESX inventory If you restart hostd, virtual machines that were previously not in an invalid state may appear as invalid



When a virtual machine is in an invalid state, you see these errors: vmx| VmdbPipeStreamsOvlError: write failed, draining reads vmx| VmdbPipeStreamsOvlError: Couldn't initiate write vmx| Redirecting stdin/stdout/stderr to /dev/null. vmx| SOCKET 1 (91) send error 12: Cannot allocate memory vmx| Vix: [177936 mainDispatch.c:2472]: VMAutomation: Connection Error (1) on connection 0. vmx| SOCKET 1 (91) send error 12: Cannot allocate memory -> not able to create socket,no mem available. mks| SOCKET 2 (92) recv error 104: Connection reset by peer mks| SOCKET 2 (92) destroying VNC backend on socket error: 1





The messages log contains entries similar to: o

sfcb-CIMXML-Processor[9857708]: SendMsg sending to 7 9857708-9 Bad file descriptor sfcb-CIMXML-Processor[9857708]: spSendMsg sending to 7 9857708-9 Bad file descriptor sfcb-CIMXML-Processor[9857708]: --- spSendReq/spSendMsg failed to send on 7 (-1) root: sfcbd-watchdog:Restarting SFCB! Log a bug!!! root: sfcbd-watchdog:stopping sfcbd root: sfcbd Stopping sfcbd root: sfcbd-watchdog:starting sfcbd root: sfcbd Starting sfcbd sfcb-sfcb[9849840]: --- Using /etc/sfcb/sfcb.cfg

o

FoundryVMDirectlyOpenSocketToVMX: Failed to create socket pair.

The hostd logs may report this error when you try to power on the virtual machine: Cannot connect to /var/run/vmware/root_0/1299674323658606_59831734/testAutomation-fd: File not found

 

vMotion fails intermittently at a random percentage Connecting to the remote console via the vSphere Client fails with the error: Unable to connect to the MKS: There is no VMware process running for config file

  

Unable to retrieve any files from the datastore via Datastore Browser The vm-support diagnostic information gathering utility is unresponsive The Health Status tab may not load correctly

Cause This issue occurs due to exhaustion of VMkernel socket resources on ESXi hosts. It occurs more frequently on hosts where OEM CIM providers have been installed, which may be caused by additional CIM Providers loaded under sfcbd. To ensure the stability of the OEM CIM providers, ensure that they are updated to the latest release.

Note: This issue only occurs on ESXi. It does not occur on ESX. On ESX, sfcbd runs as a process in the service console rather than from a world under the VMkernel.

Resolution

Resolution This issue is resolved by VMware ESXi 4.1 Patch ESXi410-201107401-BG. For more information, see VMware ESXi 4.1 Patch ESXi410-201107401-BG: Updates Firmware (2000609).

Note: A system which is found to be in a bad state must be rebooted prior to applying the patch.

Workaround To work around this issue, stop the sfcbd hardware monitoring agent on the ESXi host. When sfcbd is disabled, hardware status information for the ESXi host will be unavailable.

To stop sfcbd: 1. Log in to the VMware ESXi host as the root user. For more information, see Using Tech Support Mode in ESXi 4.1 (1017910). 2. Run the command: /etc/init.d/sfcbd-watchdog stop

3. To make this change persistent on reboot, run these commands: chkconfig sfcbd-watchdog off chkconfig sfcbd off

If hardware monitoring is an environmental requirement, you can extend the amount of time before the issue re-occurs by changing the configuration of sfcbd: 1. From the ESXi shell, edit the /etc/sfcb/sfcb.cfg file using a text editor. 2. Search for the entry provProcs: 16, and change the value from 16 to 12. 3. Restart sfcbd for the changes to take effect using the command: /etc/init.d/sfcbd-watchdog restart

Note: Depending on the system workload, this change may only temporarily resolve the issue.

If the symptoms persist after applying this patch or workaround, collect logs from the environment and contact VMware Support. For more information, see:   

Filing a Support Request (1021619) How to Submit a Support Request Collecting diagnostic information for VMware ESXi Server (1010705)

Unable to create snapshot: Operation failed because file already exists or Cannot complete the operation because the file or folder [DatastoreName] VMname/VMname.vmx already exists Symptoms While creating a virtual machine snapshot from the VI or vSphere Client, or from the command line, you may receive any of these errors depending on your product version: 

ESX/ESXi 3.5 or VirtualCenter 2.5 VI Client: Operation failed because file already exists



ESX/ESXi 3.5 Service Console VMControl error -3: Invalid arguments Example: # vmware-cmd VMname.vmx createsnapshot name desc 0 0 VMControl error -3: Invalid arguments



ESX/ESXi/vCenter Server 4.0 vSphere Client Cannot complete the operation because the file or folder [DatastoreName] VMname/VMname.vmx already exists



ESX/ESXi/vCenter Server 4.1 vSphere Client Cannot complete the operation because the file or folder already exists



ESX/ESXi 4.0 Service Console # vmware-cmd /vmfs/volumes/47ecbe04-acfca740-564c001a4be960e0/VMname/VMname.vmx createsnapshot name desc 0 0 Task reported error: (vim.fault.FileAlreadyExists) {

dynamicType = , dynamicProperty = (vmodl.DynamicProperty) [], msg = 'Cannot complete the operation because the file or folder [DatastoreName] VMname/VMname.vmx already exists', faultCause = , faultMessage = (vmodl.LocalizableMessage) [], file = '[DatastoreName] VMname/VMname.vmx' } createsnapshot(name, desc, 0, 0) = 0 

ESX/ESXi 4.1 Service Console # vmware-cmd /vmfs/volumes/47ecbe04-acfca740-564c001a4be960e0/VMname/VMname.vmx createsnapshot name desc 0 0 Task reported error: (vim.fault.FileAlreadyExists) { dynamicType = , dynamicProperty = (vmodl.DynamicProperty) [], msg = 'Cannot complete the operation because the file or folder already exists', faultCause = , faultMessage = (vmodl.LocalizableMessage) [], file = '[DatastoreName] VMname/VMname.vmx' } createsnapshot(name, desc, 0, 0) = 0



ESX/ESXi 3.5 hostd.log file contains these entries: 'BaseLibs' 51723184 info] SNAPSHOT: SnapshotBranchDisk failed: The file already exists (39). 'BaseLibs' 51723184 info] SNAPSHOT: SnapshotBranch failed: The file already exists (5). 'BaseLibs' 51723184 info] DISKLIB-LINK : File '/vmfs/volumes/4c0f81ad-b85976a8-21d1001cc493ce16/VMname/VMname-000002-delta.vmdk' already exists. 'BaseLibs' 51723184 info] DISKLIB-LIB : Failed to create link: The file already exists (39) 'BaseLibs' 51723184 info] SNAPSHOT: SnapshotBranchDisk failed: The file already exists (39). 'BaseLibs' 51723184 info] SNAPSHOT: SnapshotBranch failed: The file already exists (5).



ESX/ESXi 4.x hostd.log file contains these entries: info 'Libs'] SNAPSHOT: SnapshotBranch failed: The file

already exists (5). info 'Vmsvc'] Failed to do Snapshot op: Error: (12) The file already exists info 'DiskLib'] DISKLIB-LINK : File '/vmfs/volumes/47ecbe04-acfca740-564c001a4be960e0/VMname/VMname-000004-delta.vmdk' already exists. info 'DiskLib'] DISKLIB-LIB : Failed to create link: The file already exists (39) info 'DiskLib'] DISKLIB-LIB : Failed to create link: The file already exists (39) info 'Libs'] SNAPSHOT: SnapshotBranch failed: The file already exists (5). info 'Vmsvc'] Failed to do Snapshot op: Error: (12) The file already exists Resolution This issue occurs when VMware ESX or ESXi tried to create a new vDisknamedelta.vmdk redolog file, but the generated filename already exists. This can occur if the delta.vmdk redolog file does not have an associated descriptor file. Every virtual disk is composed of two files, a "flat" or "delta" file containing the actual data and a descriptor file containing geometry and topology information. For example, vDiskname.vmdk and vDiskname-flat.vmdk or vDiskname-delta.vmdk. For more information, see Understanding virtual machine snapshots in VMware ESX (1015180). When creating a snapshot, the host uses the lowest file number that is not in use by the chain of snapshots and does not exist on the directory. The host calculates this by checking through the descriptor files. When the host tries to create the snapshot, it first creates the new descriptor file, then creates the associated -delta.vmdk file. If the host detects that a -delta.vmdk file with that name already exists the snapshot operation fails and the virtual machine reverts to its previous state. To resolve this issue, locate and manually remove the -delta.vmdk file that does not have an associated descriptor file. 1. 2. 3. 4. 5.

Connect to the ESX or ESXi host using the vSphere Client. Locate the affected virtual machine in the Inventory. Determine which datastore(s) the virtual machine is stored on. Open the Datastore Browser. Navigate to the working directory containing the Virtual Machine configuration files. For more information, see Creating snapshots in a different location than default virtual machine directory (1002929).

6. Confirm that remnant redolog delta.vmdk files from previous snapshots exist. For example: o o o o o

VM is currently pointed to vmname-000001.vmdk vmname-000001.vmdk 's parent is the VM base disk, vmname.vmdk A file named vmname-000002-delta.vmdk exists in the VM directory. The .vmx file does not reference the file named vmname-000002delta.vmdk. The .vmsd file does not reference the file named vmname-000002delta.vmdk.

7. Rename or move the remnant file aside. For example, move the file vmname-000002delta.vmdk into a backup directory. 8. Retry the Create Snapshot operation, it should complete successfully. Note: If the memory state is captured for a snapshot, a .vmsn file is created before the new .vmdk files. If the snapshot operation fails unexpectedly, the process halts and the .vmsn file is left behind. The .vmsn files can be safely removed if the virtual machine has no snapshots. If there are any snapshots, look in the .vmsd file to identify the which .vmsn files are referenced by the VM and remove the rest. For more information, see Creating a virtual machine snapshot fails with the error: File already exists (1002190).

Understanding virtual machine snapshots in VMware ESX Symptoms This article may be helpful when you encounter these issues:     

Virtual machines are not responding or cannot start due to broken parent and child virtual disk dependencies. Virtual machines are not responding or do not start due to redo logs residing on datastores that do not have free space. Snapshot creation takes too long when specifying the memory snapshot option. Snapshot delete or remove operations result in the vSphere or VMware Infrastructure (VI) Client timing out. Backups fail while quiescing during a snapshot operation.

Purpose This article provides information about virtual machine snapshots.

Resolution

What is a snapshot? A snapshot preserves the state and data of a virtual machine at a specific point in time.  

The state includes the virtual machine’s power state (for example, powered-on, powered-off, suspended). The data includes all of the files that make up the virtual machine. This includes disks, memory, and other devices, such as virtual network interface cards.

A virtual machine provides several operations for creating and managing snapshots and snapshot chains. These operations let you create snapshots, revert to any snapshot in the chain, and remove snapshots. You can create extensive snapshot trees. In VI3 and vSphere 4.x, the virtual machine snapshot delete operation combines the consolidation of the data and the deletion of the file. This caused issues when the snapshot files are removed from the Snapshot Manager, but the consolidation failed. This left the VM still running on snapshots, and the user may not notice until the datastore is full. In vSphere 4.x, an alarm can be created to indicate if a VM was running in snapshot mode. For more information, see Configuring VMware vCenter Server to send alarms when virtual machines are running from snapshots (1018029). In vSphere 5.0, enhancements have been made to the snapshot removal. In vSphere 5.0, you are informed via the UI if the consolidation part of a RemoveSnapshot or RemoveAllSnapshots operation has failed. A new option, Consolidate, is available via the Snapshot menu to restart the consolidation. When creating a snapshot, there are several options you can specify:    

Name: This is used to identify the snapshot. Description: This is used to describe the snapshot. Memory: If the flag is 1 or true, a dump of the internal state of the virtual machine is included in the snapshot. Memory snapshots take longer to create. Quiesce:If the flag is 1 or true, and the virtual machine is powered on when the snapshot is taken, VMware Tools is used to quiesce the file system in the virtual machine. Quiescing a file system is a process of bringing the on-disk data of a physical or virtual computer into a state suitable for backups. This process might include such operations as flushing dirty buffers from the operating systems in-memory cache to disk, or other higher-level applicationspecific tasks.

Note: Quiescing indicates pausing or altering the state of running processes on a computer, particularly those that might modify information stored on disk during a backup, to guarantee a consistent and usable backup.

When a snapshot is created, it is comprised of these files: 

-.vmdk and --delta.vmdk

A collection of .vmdk and -delta.vmdk files for each virtual disk is connected to the virtual machine at the time of the snapshot. These files can be referred to as child disks, redo logs, or delta links. These child disks can later be considered parent disks for future child disks. From the original parent disk, each child constitutes a redo log pointing back from the present state of the virtual disk, one step at a time, to the original. Note: The value may not be consistent across all child disks from the same snapshot. The file names are chosen based on filename availability. 

.vmsd

The .vmsd file is a database of the virtual machine's snapshot information and the primary source of information for the snapshot manager. The file contains line entries which define the relationships between snapshots as well as the child disks for each snapshot. 

Snapshot.vmsn

These files are the memory state at the time of the snapshot.

What products use the snapshot feature? In addition to being able to use snapshot manager to create snapshots, snapshots are used by many VMware and third-party products and features. Some VMware products that use snapshots extensively are:   

VMware Data Recovery VMware Lab Manager VMware vCenter and the VMware Infrastructure Client (Snapshot Manager, Storage vMotion)

Note: This is not an exhaustive list.

How do snapshots work? Our VMware API allows VMware and third-party products to perform operations with virtual machines and their snapshots. This is a list of common operations that can be performed on virtual machines and snapshots using our API: 

CreateSnapshot: Creates a new snapshot of a virtual machine. As a side effect, this updates the current snapshot.

   

RemoveSnapshot: Removes a snapshot and deletes any associated storage. RemoveAllSnapshots: Remove all snapshots associated with a virtual machine. If a virtual machine does not have any snapshots, then this operation simply returns successfully. RevertToSnapshot: Changes the execution state of a virtual machine to the state of this snapshot. (vSphere 5.0 only) Consolidate: Merges the hierarchy of redo logs.

This is a high-level overview of how to create, remove, or revert snapshot requests that are processed within the VMware environment: 1. A request to create, remove, or revert a snapshot for a virtual machine is sent from the client to the server using the VMware API. 2. The request is forwarded to the VMware ESX host that is currently hosting the virtual machine in question. Note: This only occurs if the original request was sent to a different server, such as vCenter, which is managing the ESX host. 3. If the snapshot includes the memory option, the ESX host writes the memory of the virtual machine to disk. Note: The length of time the ESX host takes to write the memory onto the disk is relative to the amount of memory the virtual machine is configured to use. 4. If the snapshot includes the quiesce option, the ESX host requests the guest operating system to quiesce the disks via VMware Tools. Note: Depending on the guest operating system, the quiescing operation can be done by the sync driver, the vmsync module, or Microsoft's Volume Shadow Copy (VSS) service. For more information on quiescing, see Troubleshooting Volume Shadow Copy (VSS) quiesce related issues (1007696) for VSS or A virtual machine can freeze under load when you take quiesced snapshots or use custom quiescing scripts (5962168) for the SYNC driver. 5. The ESX host makes the appropriate changes to the virtual machine's snapshot database (.vmsd file) and the changes are reflected in the snapshot manager of the virtual machine. Note: When removing a snapshot, the snapshot entity in the snapshot manager is removed before the changes are made to the child disks. The snapshot manager does not contain any snapshot entries while the virtual machine continues to run from the child disk. For more information, see Committing snapshots when there are no snapshot entries in the snapshot manager (1002310). 6. The ESX host calls a function similar to the Virtual Disk API functions to make changes to the child disks (-delta.vmdk and .vmdk files) and the disk chain. Note: During a snapshot removal, if the child disks are large in size, the operation may take a long time. This can result in a timeout error message from either VirtualCenter or the VMware

Infrastructure Client. For more information about timeout error messages, see vCenter operation times out with the error: Operation failed since another task is in progress (1004790).

The child disk The child disk, which is created with a snapshot, is a sparse disk. Sparse disks employ the copyon-write (COW) mechanism, in which the virtual disk contains no data in places, until copied there by a write. This optimization saves storage space. The grain is the unit of measure in which the sparse disk uses the copy-on-write mechanism. Each grain is a block of sectors containing virtual disk data. The default size is 128 sectors or 64KB.

Child disks and disk usage It is important to note these points regarding the space utilization of child disks:  

  

If a virtual machine is running off of a snapshot, it is making changes to a child or sparse disk. The more write operations made to this disk, the larger it grows. The space requirements of the child disk are in addition to the parent disk on which it depends. If a virtual machine has a 10 GB disk with a child disk, the space used will be 10 GB + the child disk size. Child disks have been known to grow large enough to fill an entire datastore. The speed at which child disks grow is directly dependent on the amount of I/O being done to the disk. The size of the child disk has a direct impact on the length of time it takes to delete the snapshot associated to the child disk.

These Knowledge Base articles touch on the topic of child disks and disk usage:   

No more space for the redo log error when attempting to start a virtual machine (1002103) Why snapshot removal can stop a virtual machine for long time (1002836) Troubleshooting a datastore or VMFS volume that is full or near capacity (1003412)

The disk chain Generally, when you create a snapshot for the first time, the first child disk is created from the parent disk. Successive snapshots generate new child disks from the last child disk on the chain. The relationship can change if you have multiple branches in the snapshot chain. This diagram is an example of a snapshot chain. Each square represents a block of data or a grain as described above:

Caution: Manually manipulating the individual child disks or any of the snapshot configuration files may compromise the disk chain. VMware does not recommend manually modifying the disk chain as it may result in data loss. For more information, see Consolidating snapshots (1007849). Additional Information 

To determine if a virtual machine is running on snapshots, see Determining if a virtual machine is using snapshots (1004343).



There are specific considerations when hosting a Microsoft Active Directory controller in a virtual environment. For a full list of considerations, see Microsoft Knowledge Base article 888794. Note: The preceding link was valid as of November 9, 2009. If you find the link to be broken, provide feedback on the article and a VMware employee will update the article as necessary.



Time-sensitive applications may be impacted by reverting to a previous snapshot. Reverting the snapshot will revert the virtual machine to the point in time when the snapshot was created. This includes any operations conducted by the time-sensitive service or application in the guest operating system.



Reverting virtual machines to a snapshot causes all settings configured in the guest operating system since that snapshot to be reverted. The configuration which is reverted includes, but is not limited to, previous IP addresses, DNS names, UUIDs, guest OS patch versions, etc.



A snapshot operation should not be performed on a virtual machine which uses third-party iSCSI software initiators and is running in VMware Infrastructure 3. You can perform a snapshot operation in a vSphere environment, but it requires additional steps. For more information, see Using a Microsoft or 3rd party software iSCSI Initiator in a VMware environment (1010547) or Running a Third-Party iSCSI initiator in the Virtual Machine section of: o VMware Infrasturcture 3: SAN System Design and Deployment Guide o VMware vSphere 4: SAN System Design and Deployment Guide



There have been changes to the Delete All snapshot process in the patch releases for ESX/ESXi 3.5 and 4.0. For more information, see: o VMware ESX 3.5, Patch ESX350-201006401-SG: Updates VMkernel, VMware Tools, hostd, VMX, VMnix (1022899) o VMware ESXi 3.5, Patch ESXe350-201006401-I-SG: Updates firmware (1020052) o VMware ESX 4.0, Patch ESX400-201006201-UG: Updates the VMware ESX 4.0 Core and CIM components (1017721) o VMware ESXi 4.0, Patch ESXi400-201006201-UG: Updates Firmware (1017739)



For versions prior to VMware ESX 4.0 Update-2, the task of consolidating all snapshots (Remove All Snapshots task) caused unique changes stored only in the second snapshot delta disk to be copied upwards through the snapshot chain and into the first snapshot, or its "parent.". This effect is recursive for each preceding parent file. Example: You have a base disk of size 8 GB and 2 levels of snapshots, each of 4 GB each. During a Remove All Snapshot Tasks, the first snapshot delta disk file can grow, worse-case scenario, to 8GB, as all new blocks from the second snapshot are written. Any common changes stored in both snapshot levels do not require additional space.



From ESX4.0 Update 2 onwards, the snapshot mechanism has changed. VMware ESX now incorporates improved consolidation procedures which lessen the demand of free space. You are able to consolidate virtual machine delta disks even while minimal free space on your datastore is available.

Domain login fails if snapshot is reverted Symptoms     

The virtual machine is joined to a Windows domain The snapshot was reverted to a point more than 30 days in the past You are unable to log in to a virtual machine running Windows You do a system restore to a point more than 30 days in the past A Windows Logon error message appears: Windows cannot connect to the domain, either because the domain controller is down or otherwise unavailable, or because your computer

account was not found. Please try again later. If this message continues to appear, contact your system administrator for assistance.

Resolution This issue occurs because the machine account password changes automatically every 30 days when a computer is joined to a Windows domain. When the computer's snapshot is reverted to a point in the past, or a system restore is run, the password in the virtual machine can revert to a version that does not match the version on the domain. When the passwords do not match the domain cannot authenticate the machine's system account credentials nor any user logins from that machine. To resolve this issue, follow the steps in the Microsoft Knowledge Base article Issues with domain membership after a system restore. Note: To prevent this issue from happening, disable machine account password updates. For more information, see the Microsoft Knowledge Base article How to disable automatic machine account password changes .

----------------------

Resolving the CID mismatch error: The parent virtual disk has been modified since the child was created Symptoms  

You are unable to power on a virtual machine or consolidate snapshots if a parent file has been modified after the child has been created. This will include base disks and snapshot delta files. Depending on the modification or caused mismatch, any of the following error messages can be observed: Failed to open '' with flags 0xe (The parent virtual disk has been modified since the child was created). Failed to open (The parent virtual disk has been modified since the child was created). Failed to open '': The parent virtual disk has been modified since the child was created (18). DISKLIB-LINK : Attach: Content ID mismatch (7b7644b2 != 4f5a6761). DISKLIB-LINK : Attach: the capacity of each link is different (83886080 != 46399652).



You see this error in the vSphere Client: Cannot open the disk '/vmfs/volumes/4a365b5d-eceda1-19-439b000cfc0086f3/examplevm/examplevm-000001.vmdk' or one of the snapshot disks it depends on. Reason: The parent virtual disk has been modified since the child was created.



The virtual machine log contains information similar to: Aug 12 16:36:12.755: vmx| DISKLIB-LINK : Attach: Content ID mismatch (d0fdb25b != ef4854ee).

Aug 12 16:36:12.765: vmx| DISKLIB-CHAIN : "/vmfs/volumes/4a365b5deceda119-439b-000cfc0086f3/examplevm/examplevm.vmdk" : failed to open (The parent virtual disk has been modified since the child was created). Aug 12 16:36:12.766: vmx| DISKLIB-VMFS : "/vmfs/volumes/4a365b5deceda119-439b-000cfc0086f3/examplevm/examplevm-000002-delta.vmdk" : closed. Aug 12 16:36:12.769: vmx| DISKLIB-VMFS : "/vmfs/volumes/4a365b5deceda119-439b-000cfc0086f3/examplevm/examplevm-000001-delta.vmdk" : closed. Aug 12 16:36:12.770: vmx| DISKLIB-VMFS : "/vmfs/volumes/4a365b5deceda119-439b-000cfc0086f3/examplevm/examplevm-flat.vmdk" : closed. Aug 12 16:36:12.771: vmx| DISKLIB-LIB : Failed to open '/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm000002.vmdk' with flags 0xa (The parent virtual disk has been modified since the child was created). Aug 12 16:36:12.772: vmx| DISK: Cannot open disk "/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm000002.vmdk": The parent virtual disk has been modified since the child was created (18). Aug 12 16:36:12.773: vmx| Msg_Post: Error Aug 12 16:36:12.774: vmx| [msg.disk.noBackEnd] Cannot open the disk '/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm000002.vmdk' or one of the snapshot disks it depends on. Aug 12 16:36:12.774: vmx| [msg.disk.configureDiskError] Reason: The parent virtual disk has been modified since the child was created.--------------------------------------Aug 12 16:36:12.787: vmx| Module DiskEarly power on failed.  o o o

The virtual machine shuts down abruptly during snapshot removal, deletion, or commit. The virtual machine shuts down abruptly during or immediately following a Storage vMotion or Migration. You see the following error when attempting to power on a virtual machine: A general system error occurred: Internal error.



Performing a snapshot removal or powering on the virtual machine generates the error: Content ID mismatch

Purpose This document provides information and troubleshooting steps to diagnose Content ID mismatch conditions between two or more virtual disk files for a virtual machine.

THE CONTENT OF THIS ARTICLE IS PROVIDED "AS-IS," AND TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, VMWARE DISCLAIMS ALL OTHER REPRESENTATIONS AND WARRANTIES, EXPRESS OR IMPLIED, REGARDING THIS CONTENT, INCLUDING THEIR FITNESS FOR A PARTICULAR PURPOSE, THEIR MERCHANTABILITY, OR THEIR NONINFRINGEMENT. VMWARE SHALL NOT BE LIABLE FOR ANY DAMAGES ARISING OUT OF OR IN CONNECTION WITH THE USE OF THIS CONTENT, INCLUDING DIRECT, INDIRECT, CONSEQUENTIAL DAMAGES,

LOSS OF BUSINESS PROFITS OR SPECIAL DAMAGES, EVEN IF VMWARE HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Resolution Overview The CID value of a virtual machine disk descriptor file aids in the goal of ensuring content in a parent virtual disk file, such as a flat or base disk, is retained in a consistent state.

The child delta disks which derive from that base disk's snapshot contain all further writes and changes; these changes depend on the source disk to remain intact.

A virtual machine disk descriptor file details the basic geometry, format, or otherwise identification and handling for a virtual disk or virtual disk delta file. A CID resides in each virtual machine's disk descriptor file for integrity or state tracking, per above.

Example descriptor file for a base disk examplevm.vmdk:

Example descriptor file for a delta disk examplevm000001.vmdk:

# Disk DescriptorFile version=1 CID=7b7644b2 parentCID=ffffffff createType="vmfs"

# Disk DescriptorFile version=1 CID=69a1c662 parentCID=7b7644b2 createType="vmfsSparse" parentFileNameHint="examplevm.vmdk"

# Extent description RW 20971520 VMFS "examplevmflat.vmdk" # The Disk Data Base #DDB

# Extent description RW 20971520 VMFSSPARSE "examplevm-000001delta.vmdk" # The Disk Data Base #DDB

ddb.toolsVersion = "0" ddb.adapterType = "lsilogic" ddb.toolsVersion = "7302" ddb.geometry.sectors = "63" ddb.geometry.heads = "255" ddb.geometry.cylinders = "1305" ddb.uuid = "60 00 C2 9f ae de ba e995 4e a7 a6 4e 95 c1 c1" ddb.virtualHWVersion = "4"

Note: examplevm-000001.vmdk refers to, and in another sense, depends on, examplevm.vmdk.

When the virtual machine references a virtual disk, it cites either the base disk's descriptor file, or a snapshot delta's descriptor file. In this example, the virtual machine configuration file, or examplevm.vmx, refers to the delta disk descriptor file:

scsi0:0.present = "true" scsi0:0.fileName = "examplevm-000001.vmdk" scsi0:0.deviceType = "scsi-hardDisk"

Any time a virtual machine is powered on, the referenced base or delta disk descriptor file's CID value is changed (see CID highlighted in blue, above): examplevm-000001.vmdk before power-on:

examplevm-000001.vmdk after power-on:

CID=69a1c662

CID=6aff3ba2 parentCID=7b7644b2

parentCID=7b7644b2

All of the above details a virtual machine in good running condition. A mismatch can be found here, which prevents tasks to succeed for this virtual machine: examplevm.vmdk:

examplevm-000001.vmdk:

CID=12a9ffab parentCID=ffffffff

CID=69a1c662 parentCID=7b7644b2

In effect, a CID mismatch ensures that deviance from the original disk state results in all dependent child delta content being invalidated. This protects stored data from further potential corruption.

Cause Content ID mismatch conditions are triggered by interruptions to major virtual machine migrations such as Storage vMotion or Migration, VMware software error, or user action. These Content IDs are only used for virtual machine disks with snapshots. For additional information on snapshots, see Understanding virtual machine snapshots in VMware ESX (1015180).

Some scenarios to avoid in particular include:   

Interrupting a virtual machine migration or Storage vMotion. For more information, see After cold migration on ESX Server, virtual disk with snapshot has the wrong CID (1005228). Adding snapshotted virtual machine disks to other virtual machines and powering them on. Expanding, enlarging, or modifying virtual machine disks with existing snapshots. For more information, see A virtual machine cannot boot after extending a base virtual disk that is part of a snapshot hierarchy (1646892).

Troubleshooting When there is a CID mismatch, the virtual machine name is provided in the error message, but you must identify:   

What virtual machine disk, or disks are affected. What specific disk descriptor files are affected. The cause of the mismatch, or what changes occurred.

Identifying the virtual machine disk and descriptor files affected There are several methods to log into an ESX host to review content of utilized datastores, depending on the version of ESX utilized. For more information, see Editing configuration files in VMware ESX (1017022).

Notes: 





The datastore browser provided in the VMware vSphere Client or VMware Infrastructure Client do not distinguish between virtual machine descriptor (1) and flat or delta files (2). They are collapsed into singular entities to make datastore browsing simpler. As you need to distinguish between the two files, use the additional access methods provided in the referenced article. Due to the nature of the problem experienced, the quickest method for resolving the issue is with the Command-Line Interface available on the individual ESX host. Utilize this method if you have sufficient background knowledge on command-line usage.Alternatively, you can use the VMware vSphere Command Line Interface (CLI) or VMware vSphere Management Assistant appliance (vMA) to obtain the virtual machine disk descriptor files for review. If you are unable proceed, see Filing a Support Request (1021619) to seek technical assistance from VMware.

After locating the virtual machine's files and directory:

1. The virtual machine's vmware.log file contains information which identifies the specific disk chain affected. Review the logs and note the location and files affected. For example: Aug 12 16:36:12.755: vmx| DISKLIB-LINK : Attach: Content ID mismatch (d0fdb25b != ef4854ee). Aug 12 16:36:12.765: vmx| DISKLIB-CHAIN : "/vmfs/volumes/4a365b5deceda119-439b-000cfc0086f3/examplevm/examplevm.vmdk" : failed to open (The parent virtual disk has been modified since the child was created). Aug 12 16:36:12.766: vmx| DISKLIB-VMFS : "/vmfs/volumes/4a365b5deceda119-439b-000cfc0086f3/examplevm/examplevm-000002-delta.vmdk" : closed.

Aug 12 16:36:12.769: vmx| DISKLIB-VMFS : "/vmfs/volumes/4a365b5deceda119-439b-000cfc0086f3/examplevm/examplevm-000001-delta.vmdk" : closed. Aug 12 16:36:12.770: vmx| DISKLIB-VMFS : "/vmfs/volumes/4a365b5deceda119-439b-000cfc0086f3/examplevm/examplevm-flat.vmdk" : closed. Aug 12 16:36:12.771: vmx| DISKLIB-LIB : Failed to open '/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm000002.vmdk' with flags 0xa (The parent virtual disk has been modified since the child was created). Aug 12 16:36:12.772: vmx| DISK: Cannot open disk "/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm000002.vmdk": The parent virtual disk has been modified since the child was created (18). Aug 12 16:36:12.773: vmx| Msg_Post: Error Aug 12 16:36:12.774: vmx| [msg.disk.noBackEnd] Cannot open the disk '/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm000002.vmdk' or one of the snapshot disks it depends on. Aug 12 16:36:12.774: vmx| [msg.disk.configureDiskError] Reason: The parent virtual disk has been modified since the child was created.--------------------------------------Aug 12 16:36:12.787: vmx| Module DiskEarly power on failed. Note: This indicates that the file examplevm-000002.vmdk references its parent (which in itself references another parent file), one of which has been modified some time after examplevm000002.vmdk was created. You must take corrective measures on any of these files: examplevm.vmdk, examplevm-000001.vmdk, and examplevm-000002.vmdk.

2. With the problem point (or points) determined, make backup copies of the disk descriptor files that require corrections or editing. In the above example, backups ofexamplevm.vmdk, examplevm-000001.vmdk, andexamplevm-000002.vmdkare required.

3. Review the contents of each affected descriptor file and compare the mismatching values. For example: examplevm.vmdk:

examplevm-000001.vmdk:

examplevm-000002.vmdk:

CID=12a9ffab parentCID=ffffffff

CID=69a1c662 parentCID=7b7644b2

CID=59fab513 parentCID=69a1c662

4. Disk examplevm-000002.vmdk links to examplevm-000001.vmdk without error. However, the base disk examplevm.vmdk has been modified, causing the error.

Correcting the Content ID mismatch At this point the problem point has been identified, the virtual machine's files have backups, and corrections must be applied. To correct the Content ID mismatch: Warning:

 

The CID mechanism is in place to prevent data corruption. Depending on the changes incurred upon the parent file or files, the guest operating system may be unable to boot successfully even after making corrections. A backup recovery should be made available for such circumstances. The steps outlined here are potentially hazardous for your environment if they are not followed exactly. If you are not comfortable performing these steps, contact VMware Technical Support and work with them to resolve the issue.

1. Confirm that the disks reference each other: Example descriptor file for a base disk examplevm.vmdk:

Example descriptor file for delta disk examplevm-000001.vmdk:

# Disk DescriptorFile version=1 CID=12a9ffab parentCID=ffffffff createType="vmfs" # Extent description RW 20971520 VMFS "examplevm-flat.vmdk" # The Disk Data Base #DDB ddb.toolsVersion = "0" ddb.adapterType = "lsilogic" ddb.geometry.sectors = "63" ddb.geometry.heads = "255" ddb.geometry.cylinders = "1305" ddb.uuid = "60 00 C2 9f ae de ba e9-95 4e a7 a6 4e 95 c1 c1" ddb.virtualHWVersion = "4"

# Disk DescriptorFile version=1 CID=69a1c662 parentCID=7b7644b2 createType="vmfsSparse" parentFileNameHint= "examplevm.vmdk" # Extent description RW 20971520 VMFSSPARSE "examplevm-000001delta.vmdk" # The Disk Data Base #DDB ddb.toolsVersion = "7302"

Example descriptor file for child delta disk examplevm000002.vmdk: # Disk DescriptorFile version=1 CID=59fab513 parentCID=69a1c662 createType="vmfsSparse" parentFileNameHint= "examplevm-000001.vmdk" # Extent description RW 20971520 VMFSSPARSE "examplevm-000002delta.vmdk" # The Disk Data Base #DDB ddb.toolsVersion = "7302"

2. Note: The linking or chain references are highlighted in blue. This example shows that examplevm-000002.vmdkss a child of examplevm-000001.vmdk, which in turn is a child of examplevm.vmdk. These three disk files make up a singular virtual disk from the virtual machine and guest operating system perspective.

3. Using a text editor (see Preferred Editors in Editing configuration files in VMware ESX (1017022)), correct the mismatch at either of the two problem points. You may either correct examplevm.vmdk, per this example, or the examplevm-000001.vmdk disk file. In either circumstance, the parentCID and CID relationship must be valid. For instance, examplevm.vmdk can have its CID changed to 7b7644b2, making it match examplevm-000001.vmdk's expected parentCID value. Alternatively, examplevm000001.vmdk's parentCID value can be changed to 12a9ffab, to match examplevm.vmdk's base disk descriptor file's CID. You may also consider creating a CID value on your own.

Note: The CID consists of eight (8) hexadecimal lower-case digits (00000000-ffffffff) with no delimiter characters.

Verifying the CID corrections The corrections made to the virtual machine files are usually not immediately acknowledged in the remainder of the product. As such, subsequent power-on attempts may not succeed yet.

To verify the CID corrections:

1. Log in to the VMware vSphere Client or VMware Infrastructure Client. 2. Select the virtual machine in the Inventory and click the Summary tab. 3. Under Resources, right-click on the datastore that contains the virtual machine's configuration file and choose Browse.

4. The Datastore Browser opens. Locate the virtual machine's directory and files. You can minimize it for now, as it will be used in step 7.

5. Right-click on the virtual machine and choose Remove from inventory. 6. When prompted, confirm your selection by clicking Yes. The virtual machine disappears from the inventory on the left.

7. Restore the Datastore Browser from step 4, right-click on the virtual machine's configuration file (for example, examplevm.vmx), and choose Add to Inventory. 8. Follow the on-screen prompts to bring the virtual machine back into the inventory. Warning: Do not power on the virtual machine unless you have a valid copy or backup of all of its files. When the virtual machine has been powered on, further irreversible changes are applied to the disk structure.

9. When ready, power on the virtual machine and verify the guest operating system's status. A file system integrity check may be performed to seek out and repair any complications that arise from the disk chain changes that caused the initial CID mismatch.

Alternative procedure At this point, the the virtual machine should start successfully if the changes incurred were minimal. The guest operating system can exhibit varying symptoms, depending on what occurred in the snapshot.

However, if the virtual machine is not in an acceptable state, you may be required to restore from a backup copy. If one is not available, you can consider starting the virtual machine on older disks.

For example, the virtual disk referenced for the provided example is examplevm-000002.vmdk. If necessary, the virtual machine's configuration file can be modified to boot from examplevm000001.vmdk, permanently invalidating examplevm-000002.vmdk. This, in effect, may allow the guest operating system to proceed as intended, at the cost of all information contained in the examplevm-000002.vmdk delta file.

You are able to log into a VMware ESX or VMware Command Line Interface to edit the virtual machine's configuration file, and you are able to copy the file off of the datastore, for editing on another system using a preferred text editor.

For additional information on this topic, see Editing configuration files in VMware ESX (1017022).

If you are unable proceed, see Filing a Support Request (1021619) to seek technical assistance from VMware.

-------------------------------

Cannot power on a virtual machine because the virtual disk cannot be opened Symptoms  

The virtual machine cannot be powered on You see these errors: o o



Cannot open the disk '/vmfs/volumes/4a365b5d-eceda119-439b000cfc0086f3/examplevm/examplevm-000002.vmdk' or one of the snapshot disks it depends on. Cannot open disk "/vmfs/volumes/4a365b5d-eceda119-439b000cfc0086f3/examplevm/examplevm-000002.vmdk": The parent virtual disk has been modified since the child was created (18).

In the vmware.log file of the virtual machine, you see errors similar to:

Jul 22 09:22:32.141: vmx| DISKLIB-LINK : "myvm.vmdk" : failed to open (The system cannot find the file specified). Jul 22 09:22:32.141: vmx| DISKLIB-CHAIN : "myvm.vmdk" : failed to open (The system cannot find the file specified). Jul 22 09:22:32.141: vmx| DISKLIB-LIB : Failed to open 'myvm.vmdk' with flags 0xa (The system cannot find the file specified). Jul 22 09:22:32.142: vmx| Msg_Post: Error Jul 22 09:22:32.142: vmx| [msg.disk.fileNotFound] VMware ESX Server cannot find the virtual disk "myvm.vmdk". Please verify the path is valid and try again. Jul 22 09:22:32.142: vmx| [msg.disk.noBackEnd] Cannot open the disk 'myvm.vmdk' or one of the snapshot disks it depends on. Jul 22 09:22:32.142: vmx| [msg.disk.configureDiskError] Reason: The system cannot find the file specified.

 

The virtual disk's descriptor file is missing The virtual machine fails when creating or removing snapshots

Purpose This article provides links to relevant documentation for resolving conditions where a virtual machine disk descriptor file is missing or invalid.

Resolution This issue occurs when the virtual machine is unable to open a referenced virtual machine disk file. To resolve this issue: Note: The steps below provide instructions or a link to a document, for validating the step and taking corrective action as necessary. The steps are ordered in the most appropriate sequence to isolate the issue and to identify the proper resolution. Please do not skip a step.

1. Verify that the virtual machine disk files are present. For more information, see VMDK and -flat.VMDK Files and Corrupt VMDK File in Verifying ESX virtual machine file integrity (1003743). 2. If all of the required disk files exist, but you see one of these errors: o o

Cannot open the disk '/vmfs/volumes/4a365b5d-eceda119-439b000cfc0086f3/examplevm/examplevm-000002.vmdk' or one of the snapshot disks it depends on. The parent virtual disk has been modified since the child was created.

See Resolving the CID mismatch error: The parent virtual disk has been modified since the child was created (1007969). 3. If descriptor files are missing, you must recreate them. For more information, see: o o o o o

Recreating a missing virtual disk (VMDK) descriptor file (1002511) Recreating a missing virtual disk (VMDK) descriptor file for delta disks (1026353) Recreating a missing virtual disk (VMDK) descriptor file for disks split into 2GB files (1026266) Recreating a missing virtual disk (VMDK) descriptor file for imported ESX 2.x virtual machines (1026254) Recreating pass-through Raw Device Mapping (RDM) files for a virtual machine (1026256)

If the issue continues to exist after trying the steps in this article:

 

Collect the VMware Support information. For more information, see Collecting diagnostic information for VMware Fusion (1003894). File a support request with VMware Support and quote this Knowledge Base article ID (1004232) in the problem description. For more information, see How to Submit a Support Request.

Knowledge Base The VMware Knowledge Base provides support solutions, error messages and troubleshooting guides

Search the VMware Knowledge Base (KB)

View by Article ID

Best practices for virtual machine snapshots in the VMware environment Purpose This article provides best practice information for snapshots. It also provides links to resources that help you understand snapshots and troubleshoot snapshot issues.

Resolution Best practices   

Snapshots are not backups. As the snapshot file is only a change log of the original virtual disk, do not rely upon it as a direct backup process. The virtual machine is running on the most current snapshot, not the original vmdk disk files. The maximum supported amount in a chain is 32. However, VMware recommends that you use only 2-3 snapshots in a chain. Use no single snapshot for more than 24-72 hours.  o This prevents snapshots from growing so large as to cause issues when deleting/committing them to the original virtual machine disks. Take the snapshot, make the changes to the virtual machine, and delete/commit the snapshot as soon as you have verified the proper working state of the virtual machine.

o

Be especially diligent with snapshot use on high-transaction virtual machines such as email and database servers. These snapshots can very quickly grow in size, filling datastore space. Commit snapshots on these virtual machines as soon as you have verified the proper working state of the process you are testing.|



If using a third party product that takes advantage of snapshots (such as virtual machine backup software), regularly monitor systems configured for backups to ensure that no snapshots remain active for extensive periods of time.  o Snapshots should only be present for the duration of the backup process. o Snapshots taken by third party software (called via API) may not show up in the vCenter Snapshot Manager. Routinely check for snapshots via the command-line.



An excessive number of snapshots in a chain or snapshots large in size may cause decreased virtual machine and host performance. Configure automated vCenter Server alarms to trigger when a virtual machine is running from snapshots. For more information, see Configuring VMware vCenter Server to send alarms when virtual machines are running from snapshots (1018029). Confirm that there no snapshots are present (via command line) before a Storage vMotion. If snapshots are present, delete them prior to the Storage vMotion. For more information, see Migrating an ESX 3.x virtual machine with snapshots in powered-off or suspended state to another datastore might cause data loss and make the virtual machine unusable (1020709). Confirm that there are no snapshots present (via command line) before increasing the size of any Virtual Machine virtual disk or virtual RDM. If snapshots are present, delete them prior to increasing the size of the disk/s. Increasing the size of a disk with snapshots present can lead to corruption of the snapshots and potential data loss. For more information, see Increasing the Size of a Virtual Disk.

 



Understanding Snapshots For more information about snapshots, see:   

Understanding virtual machine snapshots in VMware ESX (1015180) Working with snapshots (1009402) Using Snapshots in the Basic System Administration Guide for your version of ESX

Troubleshooting For assistance troubleshooting snapshot issues, see:         

Creating a snapshot for a virtual machine fails with the error: File is larger than maximum file size supported (1012384) Delete All snapshot operation results in a Consolidate Helper snapshot when a datastore has insufficient disk space (1003302) Taking a snapshot with virtual machine memory stuns the virtual machine while the memory is written to disk (1013163) Committing snapshots fails with the error: Too many levels of redo logs (1004545) Why snapshot removal can stop a virtual machine for a long time (1002836) Large snapshot delete operations time out in VirtualCenter (1004932) Commands to monitor snapshot deletion (1007566) Consolidating snapshots via the command line (1007849) Unable to take a quiesced VMware snapshot of a virtual machine (1009073)

Powering off an unresponsive virtual machine on an ESX host Symptoms       

You cannot power off an ESX host virtual machine. A virtual machine is unresponsive and cannot be killed or stopped. You cannot access or unlock files on a virtual machine. For more information, see Virtual machine does not power on because of missing or locked files (10051). After shutting down a virtual machine, vCenter Server shows the virtual machine as up and running. There is no indication that a virtual machine is shut down. You cannot edit properties in the virtual machine. You see one or more of these errors: o Soap error 999. The operation is not allowed in current state. o The attempted operation cannot be performed in the current state (Powered Off). o The request refers to an object that no longer exists or has never existed.

Purpose This article provides the correct sequence for shutting down or (when necessary) killing an unresponsive virtual machine. Note: This article pertains to ESX and does not apply to ESXi. For ESXi hosts, see Powering off a virtual machine on an ESXi host (1014165).

Resolution Warning: Follow the sections and steps in this article in order. Do not skip a section or step, as each step may have an impact on the virtual machine.

Powering off the virtual machine To determine if you must use the command line, attempt to power off the virtual machine:

1. Connect VMware Infrastructure/vSphere Client to the vCenter Server. Right-click the virtual machine and click Power off.

2. Connect vSphere Client directly to the ESX host. Right-click the virtual machine and click Power off. If this does not work, you must use the command line method.

Determining the virtual machine's state 1. Determine the host on which the virtual machine is running. This information is available in the virtual machine's Summary tab when viewed in the vSphere Client page.

2. Log in as root to the ESX host using an SSH client.

3. Run this command to verify that the virtual machine is running on this host: # vmware-cmd -l The output of this command returns the full path to each virtual machine running on the ESX host. Verify that the virtual machine is listed, and record the full path for use in this process. For example: # /vmfs/volumes///.vmx

4. Run this command to determine the state in which the ESX host believes the virtual machine to be operating: # vmware-cmd getstate If the output from this command is getstate() = on, the vCenter Server may not be communicating with the host properly. This issue must be addressed in order to complete the shutdown process. If the output from this command is getstate() = off, the ESX host may be unaware it is still running the virtual machine. This article provides additional assistance in addressing this issue.

Powering off the virtual machine using the vmware-cmd command Caution: If you want to collect the virtual machine logs to assist in troubleshooting, do not perform the steps in this section. This procedure uses the ESX command line tool and attempts to gracefully power off the virtual machine. It works if the virtual machine's process is running properly and is accessible. If unsuccessful, the virtual machine's process may not be running properly and may require further troubleshooting.

1. From the Service Console of the ESX host, run the command: vmware-cmd stop Note: is the complete path to the configuration file, as determined in the previous section. To verify that it is stopped, run the command: # vmware-cmd getstate

2. From the Service Console of the ESX host, run the command: # vmware-cmd stop hard Note: is the complete path to the configuration file, as determined in the previous section. To verify that it is stopped, run the command: # vmware-cmd getstate

3. If the virtual machine is still inaccessible, proceed to the next section. Powering off the virtual machine while collecting diagnostic information using the vm-support script

Use this procedure when you want to investigate the cause of the issue. This command attempts to power off the virtual machine while collecting diagnostic information. Perform these steps in order, as they are listed in order of potential impact to the system if performed incorrectly. Perform these steps first:

1. Determine the WorldID with the command: # vm-support -x

2. Kill the virtual machine by using this command in the home directory of the virtual machine: # vm-support -X This can take upwards of 30 minutes to terminate the virtual machine. Note: This command uses several different methods to stop the virtual machine. When attempting each method, the command waits for a pre-determined amount of time. The timeout value can be configured to be 0 by adding -d0 to switch to the vm-support command. If the preceding steps fail, perform these steps for an ESX 3.x host:

1. List all running virtual machines to find the VMID of the affected virtual machine with the command: # cat /proc/vmware/vm/*/names

2. Determine the master world ID with the command: # cat /proc/vmware/vm/####/cpu/status | less

3. Scroll to the right with the arrow keys until you see the group field. It appears similar to: Group vm.####

4. Run this command to shut the virtual machine down with the group ID: # /usr/lib/vmware/bin/vmkload_app -k 9 #### If the preceding steps fail, perform these steps for an ESX 4.x host:

1. List all running virtual machines to find the vmxCartelID of the affected virtual machine with the command: # /usr/lib/vmware/bin/vmdumper -l

2. Scroll through the list until you see your virtual machine's name. The output appears similar to: vmid=5151 pid=-1 cfgFile="/vmfs/volumes/4a16a48a-d807aa7e-e674001e4ffc52e9/mdineeen_test/vm_test.vmx" uuid="56 4d a6 db 0a e2 e5 3ea9 2b 31 4b 69 29 15 19" displayName="vm_test" vmxCartelID=####

3. Run this command to shut the virtual machine down with the vmxCartelID: # /usr/lib/vmware/bin/vmkload_app -k 9 ####

Using the ESX command line to kill the virtual machine If the virtual machine does not power off using the steps in this article, it has likely lost control of its process. You need to manually kill the process at the command line. Caution: This procedure is potentially hazardous to the ESX host. If you do not identify the appropriate process id (PID), and kill the wrong process, it may have unanticipated results. If you are not comfortable with these procedures, contact VMware Technical Support and open a Service Request. Refer to this article when you create the SR.

1. To determine if the virtual machine process is running on the ESX host, run the command: # ps auxwww |grep -i .vmx The output of this command appears similar to this if the .vmx process is running: root 3093 0.0 0.3 2016 860 ? S< Jul30 0:17 /usr/lib/vmware/bin/vmkload_app /usr/lib/vmware/bin/vmware-vmx ssched.group=host/user -# name=VMware ESX Server;version=3.5.0;licensename=VMware ESX Server;licenseversion=2.0 build-158874; -@ pipe=/tmp/vmhsdaemon-0/vmx569228e44baf49d1; /vmfs/volumes/49392e30-162037d0-17c6-001f29e9abec//.vmx The process ID (PID) for this process is in bold. In this example, the PID is 3093. Take note of this number for use in these steps. Caution: Ensure that you identify the line specific only to the virtual machine you are attempting to repair. If you continue this process for another virtual machine the one in question, you can cause downtime for the other virtual machine. If the .vmx process is listed, it is possible that the virtual machine has lost control of the process and that it must be stopped manually.

2. To kill the process, run the command: # kill

3. Wait 30 seconds and check for the process again. 4. If it is not terminated, run the command: # kill -9

5. Wait 30 seconds and check for the process again. 6. If it is not terminated, the ESX host may need to be rebooted to clear the process. This is a last resort option, and should only be attempted if the preceding steps in this article are unsuccessful.

Tags not-responding stops-responding vm-not-responding vm-unresponsive

Virtual machines appear as invalid or orphaned in vCenter Server Symptoms     

Virtual machines are showing as invalid or orphaned in vCenter Server. Virtual machines are showing as invalid or orphaned after a VMware High Availability (VMware HA) host failure occurs Virtual machines are showing as invalid or orphaned after an ESX host comes out of maintenance mode Virtual Machines are showing as invalid or orphaned after a failed DRS migration. You see one or more of these errors when trying to start a virtual machine: o Could not power VM, no swap file, failed to power on VM. o VMControl error -11: No such virtual machine. o A general system error occurred. The system returned on error. Communication with the virtual machine may have been interrupted .

Purpose This article explains what orphaned virtual machines are, how they occur, and how you can fix them. The article outlines the most common errors that relate to orphaned virtual machines and how these issues can be resolved. Resolution Notes:  

Before you begin, refer to Restarting the Management agents on an ESX or ESXi Server (1003490) for important informations on restarting the mgmt-vmware service. These solutions may require the use of the vmware-cmd command. For more information on the vmware-cmd command, see the VMware Scripting API.

In vCenter Server, you may find that you have a virtual machine that has an orphan designation or has become invalid. An orphan virtual machine is one that exists in the vCenter Server database but is no longer present on the ESX host. A virtual machine also shows as orphaned if it exists on a different ESX host than the ESX host expected by vCenter Server. A virtual machine can become orphaned: 

After a vMotion or VMware DRS Migration: Connect to the source and destination ESX/ESXi hosts using SSH. For more information, see Opening a command or shell prompt (1003892) .

Check with the vmware-cmd -l command if the orphaned virtual machine is registered on the same ESX host as reported by vCenter Server, this is likely the source machine. If the virtual machine is not registered on that host, use the vmware-cmd -l command to check if it is registered on the destination ESX host. Note: In ESXi, use the vim-cmd vmsvc/getallvms command instead of the vmware-cmd -l command. If the virtual machine is registered on the destination ESX/ESXi host: 1. Run these commands to restart the vpxa and ESX host management services:  service mgmt-vmware restart. For more information, see Virtual Machines Unexpectedly Reboot after Issuing the "mgmt-vmware restart" Command (7301769).  service vmware-vpxa restart. Note: In ESXi, use the /sbin/services.sh command to restart the management services on the host. 2. Restart the vCenter Server Server Service. For more information, see Stopping, starting or restarting the vCenter server service (1003895). Note: For the majority of problems related to orphaned virtual machines on ESX/ESXi, these steps resolve the issue. Note: Ensure there is no time difference between the source and destination ESX/ESXi hosts. If after completing these steps you see these errors when attempting to start your virtual machine: A general system error occurred. The system returned on error. Communication with the virtual machine may have been interrupted. o VMControl error -11: No such virtual machine. 3. Try to start the virtual machine from the command line using: vmware-cmd start 4. Where is the path to the configuration file as determined by „vmwarecmd –l ‟. o

Note: To power on a virtual machine on ESXi host using command line refer to Powering on an ESX/ESXi host's virtual machine (1003738). 5. Try to register the virtual machine through the vmware-cmd -s command. Note: If it does not fail with the error VMControl error -11: No such virtual machine , go to step 7.



You can also try to register a virtual machine by right-clicking on its .vmx file in the datastore browser and choosing Register Guest. 6. View the .vmx file of the virtual machine and verify that the file has valid configuration parameters. Ensure the file does not contain non UTF-8 characters. 7. If possible, compare the .vmx file with the .vmx file of another virtual machine. 8. Create a new virtual machine, choosing to use the existing virtual disks of the original virtual machine. 9. Power on the virtual machine. If the virtual machines on a particular ESX/ESXi host are showing as orphaned after a VMware HA host failure occurs or after the ESX host comes out of maintenance mode: 0. Remove the ESX host from vCenter Server. To remove the ESX host from vCenter Server: a. Select the specified ESX/ESXi host in the vCenter Server inventory. b. Right-click on the ESX/ESXi host and choose Disconnect. c. Right-click and select the Remove option after the ESX/ESXi host has been disconnected. Remove the vCenter Server agent and VMware HA agents by running the commands from the service console of the ESX host: export rpm -e rpm -e rpm -e

LGTO_AAM_VMWARE_REMOVAL=1 LGTOaama LGTOaamvm VMware-vpxa

1. Re-add the ESX/ESXi host to vCenter Server. To re-add the ESX/ESXi host to vCenter Server: a. Select the cluster, datacenter, or farm to which you want to add the ESX/ESXi host. b. Right-click on the cluster, datacenter, or farm and choose Add. c. Enter the ESX/ESXi host's IP address. You must have a username and password with sufficient permissions to add the host. Complete the steps in the Add Host wizard. If you see this error when trying to start the virtual machines: Could not power VM, no swap file, failed to power on VM when trying to start them 1. Run the ps -auxwww | grep -i [vmname] command on each ESX host until you find the server host that is running the virtual machine's process and locking its files. 2. Run the service vmware-vpxa restart command on that ESX host

3. Restart the VMware VirtualCenter Server service. If the issue persists even after performing these steps, then follow the steps below: 1. Power off the virtual machine. 2. Access the ESX/ESXi service console using an SSH client. 3. Open the virtual machine configuration file (.vmx) in a text editor. The default location is /vmfs/volumes///.vmx. 4. Remove the location of the swap file referenced in the configuration file. It should look similar to: sched.swap.derivedName = "" 5. Save the file. 6. Rename or delete the existing swap file from the virtual machine directory. 7. Unregister the virtual machine and register it back for the changes to take effect. For more information, see Registering or adding a virtual machine to the inventory (1006160). 8. Power on the virtual machine. 





If you delete a virtual machine outside of vCenter Server. A user can delete a virtual machine through the VMware Management Interface while vCenter Server is down, through the vSphere Client directly connected to an ESX/ESXi host, or by deleting the virtual machine's configuration file through the service console. Note: If the configuration file was deleted and the virtual disk remains, you can recreate the virtual machine using the VMware Management Interface or the vSphere Client and choosing to attach the existing virtual disk to a newly created .vmx file. If vCenter Server is restarted while a migration is in progress, a virtual machine may show as orphaned. This is a temporary situation. During startup, vCenter Server reconnects to all hosts. If a migration completed while vCenter Server was down, a virtual machine can be reported as an orphan until vCenter Server establishes a connection to the target host for the virtual machine. If you schedule too many virtual machines to be relocated at the same time. An ODBC timeout can cause errors in the database. To resolve an ODBC timeout, delete the orphans after ensuring the original virtual machines can power on. Click Delete VM on the vCenter Server console to delete the virtual machine and its orphan from the vCenter Server database.

vSphere Client stops responding when trying to view custom performance charts Symptoms



Using custom performance charts with a custom time range can cause the vSphere Client session connected to vCenter server to stop responding and consume 100% of one client logical CPU. For information about using custom performance charts refer to Customize Advanced Chart Views section from the vSphere Datacenter Administration Guide.



The vSphere client log may show these errors: SSL Validation failed in WebRequestPatch. System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.21.xx.xx:8443 at System.Net.Sockets.TcpClient..ctor(String hostname, Int32 port) at VirtualInfrastructure.Utils.WebRequestPatch.PreAuthSSL(HttpWebRequest request, Uri uri) SSL Validation failed in WebRequestPatch. System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.21.xx.xx:8443 at System.Net.Sockets.TcpClient..ctor(String hostname, Int32 port) at VirtualInfrastructure.Utils.WebRequestPatch.PreAuthSSL(HttpWebRequest request, Uri uri) Invoke 67 Start WaitForUpdates on PropertyCollector:propertyCollector [xx.vmware.com]. Initial socket creation and SSL authentication failed: System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.21.xx.xx:8443 at System.Net.Sockets.TcpClient..ctor(String hostname, Int32 port) at VpxClient.Plugins.WebBrowserProxy.RedirectBrowser() Invoke 67 Finish WaitForUpdates on PropertyCollector:propertyCollector [xx.vmware.com] - Serial:0.002, Server:018.225

Note: For the location of vSphere client logs, see Common vCenter Server and vSphere Client Windows paths (1028185).

Resolution VMware is aware of this issue and is working on a fix to resolve this issue. However, you can workaround this issue by clearing the custom performance chart from the client, and avoid using a custom start time on a whole round hour (e.g. 1:00, 2:00, 3:00...). To delete the custom chart information from the vCenter client: 

Browse to the this directory: On Windows 7/2008: %USERPROFILE%\AppData\Roaming\VMware On previous versions of Windows: %USERPROFILE%\Application Data\VMware

 

The -charts.xml file contains the custom chart data. This file can be deleted completely if this is the only custom chart or

 

You can edit the file to retain other custom charts:

1. Open the XML file in a text editor. 2. Locate the offending chart by searching for its Name 3. Select all lines starting from and including until you find corresponding to the chart name and Delete them

4. Save the file. The changes take effect in the vSphere client right away. Note: This article assumes that you are familiar with XML concepts and that you have made adequate backups before proceeding.

How To Fix Host Not Responding Error with VMware ESX, vSphere in vCenter Posted on 05.Feb 2010 by Ray Heffer in VMware, Virtualisation

Virtualcenter looses connectivity to an ESX or vSphere host, and all of the virtual machines that are running on the host show as „disconnected‟. You will also see that the host has „not responding‟ in brackets next to it‟s name. This one is very simple to fix, as it is usually caused by the host agent service (mgmt-vmware) failing due to a dead process. First, try and restart the mgmt-vmware service: # service mgmt-vmware restart

If you find this is hanging when trying to restart the host agent, then you‟ll need to kill off the process causing the issue. Open another console session and do the following: # ps -ef | grep hostd

This will output a list of processes using hostd similar to the following: root 23955 1 0 10:42 pts/1 00:00:00 /bin/sh /usr/bin/vmware-watchdog -s hostd -u 60 -q 5 -c /usr/sbin/vmware-hostd-support /usr/sbin/vmware-hostd -u root 23961 23955 4 10:42 ? 00:00:15 /usr/lib/vmware/hostd/vmware-hostd /etc/vmware/hostd/config.xml -u root 24211 23422 0 10:48 pts/1 00:00:00 grep hostd

If you look at the output carefully you‟ll see that the first process is using the vmware-watchdog, this is fine, but the second line is using hostd (config.xml -u). This is the culprit, so lets kill the process. By the way, your virtual machines will continue to run so don‟t worry about that.

# kill -9 23961

You‟ll now find that the hostd service will start and after a few seconds your host and virtual machines will become available again in vCenter

vSphere client connection to vCenter server stops responding at the loading data screen Symptoms  

vSphere client connection to vCenter server hangs with loading data window The vpxd logs on the vCenter server shows: ... Read timeout after apporximately 5000ms. Closing stream ...



The sql server log files in the database server shows: "Error 701. Severity 17, State: 123. "There is insufficient system memory to run this query"



The task manager on the database server shows that the sqlserver.exe memory consumption is a number below 500M while running close to 100 Virtual Machines on all managed ESX hosts

Resolution This issue can occur if the vCenter server is using a Micrsoft SQL server and the SQL server maximum server memory is set to a low value. Increase the SQL Server memory to resolve this issue. To increase the SQL memory:

1. Open Microsoft SQL Server Management Studio or Microsoft SQL Server Management Studio 2. 3. 4. 5. 6.

Express on the vCenter server. Right-click on the SQL server connection under the Object Explorer and then select Properties. Select Memory then increase the Maximum server memory to 2 GB or more and click OK. Click the Windows Start > Run then type services.msc Right-click on the SQL Server service and select Restart. Right-click on the VMware VirtualCenter Server service and select Restart.

Troubleshooting the vCenter Server Agent when it does not start Symptoms  

Unable to connect with the VMware Infrastructure /vSphereClient to the ESX host from VirtualCenter/vCenter Server. Unable to directly connect with the vSphere Client to the ESX host.

Purpose The vCenter Server Agent, also referred to as vpxa or the vmware-vpxa service, is what allows a vCenter Server to connect to a ESX host. Specifically, vpxa is the communication conduit to the hostd, which in turn communicates to the ESX kernel. This article provides troubleshooting steps for when the vpxa does not start.

Resolution Validate that each troubleshooting step below is true for your environment. Each step provides instructions or a link to a document, in order to eliminate possible causes and take corrective action as necessary. The steps are ordered in the most appropriate sequence to isolate the issue and identify the proper resolution.

1. Log in to the ESX host service console and acquire root privileges. 2. Verify that the installation of the vCenter Server agent is not corrupted. From the command-line, run: rpm -V VMware-vpxa There is only output from the command if errors are found. For example: rpm -V VMware-vpxa S.5....T /opt/vmware/vpxa/sbin/vpxa This indicates that the Size, MD5 checksum, and Timestamp for the file /opt/vmware/vpxa/sbin/vpxa are wrong, and therefore the installation of the VMwarevpxa package is corrupt. If you do have a corrupted installation, proceed to the next step to reinstall the VirtualCenter agent. Note: The Red Hat Package Manager does not keep tabs on dynamic configuration files, so you cannot see errors about them.

3. Verify that the vCenter Server agent is the correct version. For more information, see Verifying and reinstalling the correct version of VMware VirtualCenter Server agent (1003714).

4. Verify that there is adequate disk space available on the ESX host service console. For more information, see Investigating disk space on an ESX/ESXi host (1003564).

5. Verify that no processes are over utilizing the resources on the ESX host service console. For more information, see Checking for resource starvation of the ESX service console (1003496).

6. Verify that the vpxa process is not exceeding allocated memory. Change to the the vpxa log directory, run: cd /var/log/vmware/vpxa View the vpxa log file, run: more vpxa.log

Examine the log for errors. For example: [2007-07-28 17:57:25.416 'Memory checker' 5458864 error] Current value 143700 exceeds hard limit 128000. Shutting down process. [2007-07-28 17:57:25.420 'Memory checker' 3076453280 info] Resource checker stopped. These errors indicate there is not enough service console memory allocated for the vCenter Server agent. This may be due to one or more causes: o Management agents are installed in the ESX service console o Backup agents are installed in the ESX service console To resolve this problem, increase the service console memory. For more information, see Increasing the amount of RAM assigned to the ESX Server Service Console (1003501).

Increasing the amount of RAM assigned to the ESX Server Service Console (1003501). Overview

Typically, the service console on an ESX host is not configured by default to utilize the maximum amount of RAM. The default values are as follows:   

ESX 2.x hosts – the amount of RAM is assigned from the install and is based on the number of virtual machines that you plan to run. During the installation, the default RAM selection is 192MB, configurable to a maximum of 800MB ESX 3.x hosts – the default amount of RAM is configured to 272MB, configurable to a maximum of 800MB ESX 4.x hosts – the default amount of RAM is dynamically configured to a value between 300MB and 800MB, depending on the amount of RAM that is installed in the host. For example, if the host has 32GB of memory the service console RAM will be set to 500MB, while a host which has 128GB of RAM will see the service console RAM set to 700MB. The maximum has not changed from 800MB, which would be seen on hosts with 256GB of RAM or higher, if it is being dynamically allocated. Note: VMware ESXi does not have a service console. Therefore, there is no method to change the amount of RAM assigned to it. Tech Support Mode is only a running state of the system and does not require additional resources.

The default values are usually sufficient, however there are several instances where VMware recommends to increase the RAM assigned:    

When third-party system management agents are installed or crashing on the service console When a backup agent is installed on the service console When heavy swap file utilization is noticed on the service console. When the ESX host is part of a VMware High Availability (HA) cluster

Changing the amount of RAM assigned to the ESX 3.x or ESX 4.x Service Console To change the amount of RAM assigned to the service console:

1. Log in to VirtualCenter/vCenter Server from the VMware Infrastructure/vSphere Client with a user that has administrative rights. Note: If you do not have vCenter Server, log in directly to the ESX host as root.

2. 3. 4. 5. 6.

From the Inventory select the ESX host. Click the Configuration tab. Click Memory. Click Properties. On the Memory window enter a value between 256MB and 800MB for the service console parameter. Note: For troubleshooting purposes, VMware recommends that you increase the service console RAM to 800MB.

7. Click OK. Changes do not take effect until the ESX host is rebooted. For more information on rebooting an ESX host, see Rebooting an ESX Server (1003530).

-------------------------------------------------

ESX/ESXi host fails to go into maintenance mode Symptoms    

ESXi host does not enter maintenance mode ESX host does not enter maintenance mode Cannot remove the ESXi or ESX host from the vCenter Server Hosts fail to migrate when attempting to enter maintenance mode

Purpose This article helps you to troubleshoot issues when you are unable to put the ESX host into maintenance mode.

Resolution Validate that each troubleshooting step below is true for your environment. Each step provides instructions or a link to an article to eliminate possible causes and take corrective action as necessary. The steps are ordered in the most appropriate sequence to isolate the issue and identify the proper resolution. Do not skip a step.

1. Verify if there are orphaned virtual machines in the vCenter Server inventory.

All virtual machines on the ESX or ESXi host must be migrated to another host or shut down before the host is placed into maintenance mode. If there are orphaned virtual machines in the inventory, the host cannot be put into maintenance mode. Ensure that the orphaned virtual machines are removed before attempting to put the host into maintenance mode. For more information, see Determining if a virtual machine is orphaned (1003742).

2. If an ESX host is a part of VMware High Availability (HA) or DR'S cluster, check the Admission Control settings. You may have to disable this option if there are not enough resources to ensure fail over capacity. For more information, see:

3. o o

ESX host in a VMware High Availability cluster fails to enter maintenance mode and stops at 2% (1007156) ESX host unable to enter maintenance mode in a two node cluster (1017330) Note: If you want vCenter Server to migrate your running virtual machines automatically to other hosts when placing your host in maintenance mode, DR'S must be enabled in your cluster in the Fully Automated mode.

4. If the ESX host is a part of a fully automated DR'S cluster, but the Enter Maintenance Mode task hangs at 2%, try motioning the virtual machines between the hosts in the cluster. This test helps you to determine the reason for the maintenance mode issues. For more information about troubleshooting motion issues, see Diagnosing VMware motion failure at 10% (1003734). Note: If your problem still exists after trying the steps in this article:  

Gather VMware Support Script Data. For more information, see Collecting diagnostic information for VMware products (1008524). File a support request with VMware Support and note this KB Article ID in the problem description. For more information, see How to Submit a Support Request.

---------------------------

Creating a local VMFS volume generates the error: Error during the configuration of the host: Failed to update the disk partition information Details You are experiencing these issues: 

Creating a VMFS volume on your local hard disk fails with the error: o o

Solution

Error during the configuration of the host:Failed to update the disk partition information Device or Resource Busy

These issues occur because the VMFS volume and partition you are creating is on the same controller as the local storage, and that volume is continually locked by the service console. The command line tool does not check locking state, so it succeeds in creating the partition. Removing a datastore on local disk has the same restriction. You must use command line tools. The datastore can be created on local disk, but needs to be done via the command line. The following steps describes how to create the partition and VMFS filesystem manually:

1. Log into the ESX host as root via the console or SSH. For more information, see Unable to

2. 3.

connect to an ESX host using Secure Shell (SSH) (1003807). If you ware using ESXi, the partitioning steps must be done at the console. For more information on ESXi, see Tech Support Mode for Emergency Support (1003677). Identify the disk in which you wish to add the datastore. For more information on how to identify disks in ESX, see Identifying disks when working with VMware ESX (1014953). Run the command: fdisk -u

4.

where is the disk selected in step 2. Use the fdisk utility to create a new partition to house the VMFS filesystem or datastore. The partition should be type fb. For information on partitioning with fdisk, see Partitioning with fdisk. The following is an example of the steps you may need take during the partitioning phase: a. Type n and press Enter for new partition. b. Type p and press Enter for a primary partition. c. Type 1 and press Enter. d. For the First cylinder prompt, press Enter for the default. e. For last cylinder or +size..., press Enter for the default. f. Type t and press Enter to change start the process to change volume type. g. Type 1 and press Enter to select the first partition and. h. Type fb and press Enter. fb is the hexadecimal code for VMFS volumes. i. Type x and press Enter to go into Expert mode. j. Type b and press Enter to start the process of changing the starting block of the partition. k. Type 1 and press Enter to select the first partition. l. Type 128 and press Enter to set the starting block to 128. m. Type w and press Enter to save the changes and exit fdisk.

5. Record the partition number that was created as a result of step 4. 6. Since the ESX installation is actively using the disk, reboot the ESX host for the changes to take effect.

7. Log into the ESX host as root via the console or SSH. 8. Run the command: vmkfstools -C vmfs3 -b -S : where: o is the block size for the volume. This value determines the maximum file size on the volume. For more information, see Configuration Maximums for the appropriate VMware product version. o is the name for the datastore you are about to create.

o o

is the string used to identify the disk as used in steps 2 and 3. is the partition number as recorded in step 5. Example: vmkfstools -C vmfs3 -b 1m -S localVMFS vmhba0:0:0:7

Note: If the new partition is on the same disk as the Service Console OS and there is I/O going on, vmkfstools cannot reserve it for filesystem creation. This is particularly true with Dell and its PERC controllers.

1. During the installation of ESX, specify a VMFS partition. 2. Install ESX using two LUNS on the disk array using the PERC management utility. One LUN to be used by the ESX installation, and the other for the VMFS datastore.

Identifying disks when working with VMware ESX Purpose When performing troubleshooting with ESX storage, you may use command line tools which require you to identify a specific disk or LUN connected to ESX. This article explores different ways to identify these disks.

Resolution ESX 5.X Use these commands to collect disk and LUN information from within ESX: 

The command esxcli storage core path list generates a list of all LUN paths currently connected to the ESX host. The output appears similar to: fc.5001438005685fb5:5001438005685fb4fc.50060160c46036df:50060167446036dfnaa.6006016094602800e07ff528b73ae011 UID: fc.5001438005685fb5:5001438005685fb4fc.50060160c46036df:50060167446036dfnaa.6006016094602800e07ff528b73ae011 Runtime Name: vmhba0:C0:T0:L23 Device: naa.6006016094602800e07ff528b73ae011 Device Display Name: DGC Fibre Channel Disk (naa.6006016094602800e07ff528b73ae011) Adapter: vmhba0 Channel: 0 Target: 0 LUN: 23 Plugin: NMP State: active Transport: fc Adapter Identifier: fc.5001438005685fb5:5001438005685fb4 Target Identifier: fc.50060160c46036df:50060167446036df Adapter Transport Details: WWNN: 50:01:43:80:05:68:5f:b5 WWPN:

50:01:43:80:05:68:5f:b4 Target Transport Details: WWNN: 50:06:01:60:c4:60:36:df WWPN: 50:06:01:67:44:60:36:df fc.5001438005685fb5:5001438005685fb4fc.50060160c46036df:5006016f446036dfnaa.6006016094602800e07ff528b73ae011 UID: fc.5001438005685fb5:5001438005685fb4fc.50060160c46036df:5006016f446036dfnaa.6006016094602800e07ff528b73ae011 Runtime Name: vmhba0:C0:T1:L23 Device: naa.6006016094602800e07ff528b73ae011 Device Display Name: DGC Fibre Channel Disk (naa.6006016094602800e07ff528b73ae011) Adapter: vmhba0 Channel: 0 Target: 1 LUN: 23 Plugin: NMP State: active Transport: fc Adapter Identifier: fc.5001438005685fb5:5001438005685fb4 Target Identifier: fc.50060160c46036df:5006016f446036df Adapter Transport Details: WWNN: 50:01:43:80:05:68:5f:b5 WWPN: 50:01:43:80:05:68:5f:b4 Target Transport Details: WWNN: 50:06:01:60:c4:60:36:df WWPN: 50:06:01:6f:44:60:36:df Note: To detail path information for a specific device (Device: ), use the command esxcli storage core path list -d . 

The command esxcli storage core device list generates a list of LUNs currently connected to the ESX host. The output appears similar to: mpx.vmhba0:C0:T0:L0 Display Name: Local VMware Disk (mpx.vmhba2:C0:T0:L0) Has Settable Display Name: false Size: 286070 Device Type: Direct-Access Multipath Plugin: NMP Devfs Path: /vmfs/devices/disks/mpx.vmhba2:C0:T0:L0 Vendor: VMware Model: Block device Revision: 1.0 SCSI Level: 2 Is Pseudo: false Status: on Is RDM Capable: false Is Local: true Is Removable: false Is SSD: false Is Offline: false Is Perennially Reserved: false

Thin Provisioning Status: unknown Attached Filters: VAAI Status: unsupported Other UIDs: vml.0000000000766d686261323a303a30



The command esxcli storage vmfs extent list generates a list of extents for each volume as well as providing the mapping from device name to UUID. The output appears silmilar to: Volume Name VMFS UUID Device Name Partition ------------ ----------------------------------------------------------------- --------esxi-local 4e0d86e1-0db6f826-6991-d8d3855ff8d6 mpx.vmhba2:C0:T0:L0 3 datastore1 4d4ac840-c1386fa0-9f6d-0050569300a7 naa.6006016094602800364ce22e3825e011 1 vmfs5 4dad8f16-911648ca-d660-d8d38563e658 naa.600601609460280052eb8621b73ae011 1



Extent Number -------------

-----

0 0 0

The command esxcli storage filesystem list generates a compact list of the LUNs currently connected to the ESX host, including VMFS version. The output appears silmilar to: Mount Point Volume Name UUID Mounted Type Size Free ------------------------------------------------- ------------ ---------------------------------- ------- ------ ------------- ----------/vmfs/volumes/f98fbd51-d2efb396 ISOs f98fbd51-d2efb396 true NFS 581284225024 181569196032 /vmfs/volumes/4d4ac840-c1386fa0-9f6d0050569300a7 datastore1 4d4ac840-c1386fa0-9f6d-0050569300a7 true VMFS-3 9395240960 746586112 /vmfs/volumes/4e0d86e1-0db6f826-6991-d8d3855ff8d6 esxilocal 4e0d86e1-0db6f826-6991-d8d3855ff8d6 true VMFS-5 294473695232 293884395520 /vmfs/volumes/4dad8f16-911648ca-d660-d8d38563e658 vmfs5 4dad8f16-911648ca-d660-d8d38563e658 true VMFS-5 1879048192 220200960 /vmfs/volumes/4e303229-94dedb01-508cd8d3855ff8d6 4e303229-94dedb01-508c-d8d3855ff8d6 true vfat 4293591040 4290248704 /vmfs/volumes/f9618575-313f4ef5-943d-d5308d29e876 Hypervisor1 f9618575-313f4ef5-943d-d5308d29e876 true vfat 261853184 128241664 /vmfs/volumes/12e6c575-9a49251d-634c-1c34f28a0238 Hypervisor2 12e6c575-9a49251d-634c-1c34f28a0238 true vfat 261853184 163708928 /vmfs/volumes/2da668ef-40e5d96b-90bf-855ddb9c5547 Hypervisor3 2da668ef-40e5d96b-90bf-855ddb9c5547 true vfat 299778048 114704384



The command ls -alh /vmfs/devices/disks lists the possible targets for certain storage operations. The output appears similar to: lrwxrwxrwx 1 root root 19 Jul 27 16:40 vml.0000000000766d686261323a303a30 -> mpx.vmhba2:C0:T0:L0 lrwxrwxrwx 1 root root 21 Jul 27 16:40 vml.0000000000766d686261323a303a30:1 -> mpx.vmhba2:C0:T0:L0:1 lrwxrwxrwx 1 root root 21 Jul 27 16:40 vml.0000000000766d686261323a303a30:2 -> mpx.vmhba2:C0:T0:L0:2 lrwxrwxrwx 1 root root 21 Jul 27 16:40 vml.0000000000766d686261323a303a30:3 -> mpx.vmhba2:C0:T0:L0:3 lrwxrwxrwx 1 root root 21 Jul 27 16:40 vml.0000000000766d686261323a303a30:4 -> mpx.vmhba2:C0:T0:L0:4 lrwxrwxrwx 1 root root 21 Jul 27 16:40 vml.0000000000766d686261323a303a30:5 -> mpx.vmhba2:C0:T0:L0:5 lrwxrwxrwx 1 root root 21 Jul 27 16:40 vml.0000000000766d686261323a303a30:6 -> mpx.vmhba2:C0:T0:L0:6 lrwxrwxrwx 1 root root 21 Jul 27 16:40 vml.0000000000766d686261323a303a30:7 -> mpx.vmhba2:C0:T0:L0:7 lrwxrwxrwx 1 root root 21 Jul 27 16:40 vml.0000000000766d686261323a303a30:8 -> mpx.vmhba2:C0:T0:L0:8 lrwxrwxrwx 1 root root 36 Jul 27 16:40 vml.02000600006006016094602800364ce22e3825e011524149442030 -> naa.6006016094602800364ce22e3825e011 lrwxrwxrwx 1 root root 38 Jul 27 16:40 vml.02000600006006016094602800364ce22e3825e011524149442030:1 -> naa.6006016094602800364ce22e3825e011:1 lrwxrwxrwx 1 root root 36 Jul 27 16:40 vml.02000e0000600601609460280052eb8621b73ae011524149442030 -> naa.600601609460280052eb8621b73ae011 lrwxrwxrwx 1 root root 38 Jul 27 16:40 vml.02000e0000600601609460280052eb8621b73ae011524149442030:1 -> naa.600601609460280052eb8621b73ae011:1

The following are definitions for some of identifiers and their conventions: 

naa.: or eui.: NAA stands for Network Addressing Authority identifier. EUI stands for Extended Unique Identifier. The number is guaranteed to be unique to that LUN. The NAA or EUI identifier is the preferred method of identifying LUNs and the number is generated by the storage device. Since the NAA or EUI is unique to the LUN, if the LUN is presented the same way across all ESX hosts, the NAA or EUI identifier remains the same. For more information on these standards, see the SPC-3 documentation from the InterNational Committee for Information Technology Standards (T10). The represents the partition number on the LUN or Disk. If the is specified as 0, it identifies the entire disk instead of only one partition. This identifier is generally used for operations with utilities such as vmkfstools. Example: naa.6090a038f0cd4e5bdaa8248e6856d4fe:3 = Partition 3 of LUN naa.6090a038f0cd4e5bdaa8248e6856d4fe.



mpx.vmhba:C:T:L or mpx.vmhba:C:T:L: Some devices do not provide the NAA number described above. In these circumstances, an MPX Identifier is generated by ESX to represent the LUN or disk. The identifier takes the form similar to that of the canonical name of previous versions of ESX with the mpx. prefix. This identifier can be used in the exact same way as the NAA Identifier described above.



vml. or vml.: The VML Identifier can be used interchangeably with the NAA Identifier and the MPX Identifier. Appending : works in the same way described above. This identifier is generally used for operations with utilities such as vmkfstools.



vmhba:C:T:L This identifier is now used exclusively to identify a path to the LUN. When ESX detects that paths associated to one LUN, each path is assigned this Path Identifier. The LUN also inherits the same name as the first path, but it is now used an a Runtime Name, and not used as readily as the above mentioned identifiers as it may be different depending on the host you are using. This identifier is generally used for operations with utilities such as vmkfstools. Example: vmhba1:C0:T0:L0 = Adapter 1, Channel 0, Target 0, and LUN 0. Note: Generally, multi-port fiber channel adapters are equipped with dedicated controllers for each connection, and therefore each controller is represented by different vmhba#. If the adapter supports multiple connections to the same controller, it is represented by a different channel number. This representation is directly dependant on the capability of the adapter.



The is a unique number assigned to a VMFS volume upon the creation of the volume. It may be included in syntax where you need to specify the full path of specific files on a datastore.

ESX 4.X Use these commands to collect disk and LUN information from within ESX: 

The command esxcfg-mpath -b generates a compact list of LUNs currently connected to the ESX host. The output appears similar to: naa.6090a038f0cd4e5bdaa8248e6856d4fe : EQLOGIC iSCSI Disk (naa.6090a038f0cd4e5bdaa8248e6856d4fe) vmhba33:C0:T1:L0 LUN:0 state:active iscsi Adapter: iqn.199801.com.vmware:bs-tse-i137-35c1bf18 Target: IQN=iqn.200105.com.equallogic:0-8a0906-5b4ecdf03-fed456688e24a8da-bs-tse-vc40-250g Alias= Session=00023d000001 PortalTag=1



The command esxcfg-scsidevs -l generates a list of LUNs currently connected to the ESX host.

The output appears similar to: mpx.vmhba0:C0:T0:L0 Device Type: Direct-Access Size: 139890 MB Display Name: Local ServeRA Disk (mpx.vmhba0:C0:T0:L0) Plugin: NMP Console Device: /dev/sdb Devfs Path: /vmfs/devices/disks/mpx.vmhba0:C0:T0:L0 Vendor: ServeRA Model: 8k-l Mirror Revis: V1.0 SCSI Level: 2 Is Pseudo: false Status: on Is RDM Capable: false Is Removable: false Is Local: true Other Names: vml.0000000000766d686261303a303a30 

The command esxcfg-scsidevs -m generates a compact list of the LUNs currently connected to the ESX host. The output appears silmilar to: mpx.vmhba0:C0:T0:L0:2 /vmfs/devices/disks/mpx.vmhba1:C0:T0:L0:2 4c715e5f-48aabce9-2d18005055860001 0 datastore1 naa.60060160b4111600624c5b749c7edd11:1 /dev/sdd1 4b178971-55673b38-1285-00235edc7ee5 0 LUN01



The command ls -alh /vmfs/devices/disks lists the possible targets for certain storage operations. The output appears similar to: lrwxrwxrwx 1 root root 19 Oct 16 13:00 vml.0000000000766d686261303a303a30 -> mpx.vmhba0:C0:T0:L0 lrwxrwxrwx 1 root root 21 Oct 16 13:00 vml.0000000000766d686261303a303a30:1 -> mpx.vmhba0:C0:T0:L0:1 lrwxrwxrwx 1 root root 21 Oct 16 13:00 vml.0000000000766d686261303a303a30:2 -> mpx.vmhba0:C0:T0:L0:2 lrwxrwxrwx 1 root root 21 Oct 16 13:00 vml.0000000000766d686261303a303a30:3 -> mpx.vmhba0:C0:T0:L0:3 lrwxrwxrwx 1 root root 21 Oct 16 13:00 vml.0000000000766d686261303a303a30:5 -> mpx.vmhba0:C0:T0:L0:5 lrwxrwxrwx 1 root root 36 Oct 16 13:00 vml.020000000060060160b4111600624c5b749c7edd11524149442035 -> naa.60060160b4111600624c5b749c7edd11 lrwxrwxrwx 1 root root 38 Oct 16 13:00 vml.020000000060060160b4111600624c5b749c7edd11524149442035:1 -> naa.60060160b4111600624c5b749c7edd11:1

The following are definitions for some of identifiers and their conventions: 

naa. or eui. NAA stands for Network Addressing Authority identifier. EUI stands for Extended Unique

Identifier. The number is guaranteed to be unique to that LUN. The NAA or EUI identifier is the preferred method of identifying LUNs and the number is generated by the storage device. Since the NAA or EUI is unique to the LUN, if the LUN is presented the same way across all ESX hosts, the NAA or EUI identifier remains the same. For more information on these standards, see the SPC-3 documentation from the InterNational Committee for Information Technology Standards (T10). 

naa.: or eui.: The represents the partition number on the LUN or Disk. If the is specified as 0, it identifies the entire disk instead of only one partition. This identifier is generally used for operations with utilities such as vmkfstools. Example: naa.6090a038f0cd4e5bdaa8248e6856d4fe:3 = Partition 3 of LUN naa.6090a038f0cd4e5bdaa8248e6856d4fe.



mpx.vmhba:C:T:L or mpx.vmhba:C:T:L: Some devices do not provide the NAA number described above. In these circumstances, an MPX Identifier is generated by ESX to represent the LUN or disk. The identifier takes the form similar to that of the canonical name of previous versions of ESX with the mpx. prefix. This identifier can be used in the exact same way as the NAA Identifier described above.



vml. or vml.: The VML Identifier can be used interchangeably with the NAA Identifier and the MPX Identifier. Appending : works in the same way described above. This identifier is generally used for operations with utilities such as vmkfstools.



vmhba:C:T:L This identifier is now used exclusively to identify a path to the LUN. When ESX detects that paths associated to one LUN, each path is assigned this Path Identifier. The LUN also inherits the same name as the first path, but it is now used an a Runtime Name, and not used as readily as the above mentioned identifiers as it may be different depending on the host you are using. This identifier is generally used for operations with utilities such as vmkfstools. Example: vmhba1:C0:T0:L0 = Adapter 1, Channel 0, Target 0, and LUN 0. Note: Generally, multi-port fiber channel adapters are equipped with dedicated controllers for each connection, and therefore each controller is represented by different vmhba#. If the adapter supports multiple connections to the same controller, it is represented by a different channel number. This representation is directly dependant on the capability of the adapter.



/dev/sd or /dev/sd This naming convention is not VMware specific. This convention is used exclusively by the service console and open source utilities which come with the service console. The represents the LUN or Disk and is assigned by the service console during boot. The optional represents the partition on the LUN or disk. These naming conventions may vary from ESX host to ESX host and may change if storage hardware replaced. This identifier is generally used for operations with utilities such as fdisk and dd.

Note: VMware ESXi does not have a service console; disks are referred to by the VML Identifier.



The is a unique number assigned to a VMFS volume upon the creation of the volume. It may be included in syntax where you need to specify the full path of specific files on a datastore.

ESX 3.X Use these commands to collect disk and LUN information from within ESX. 

The command esxcfg-mpath -l generates a compact list of the LUNs currently connected to the ESX host. The output appears similar to: Disk vmhba32:0:0 /vmfs/devices/disks/vml.020000000060060160c0521501065cacf13f9fdd1152414 9442035 (512000MB) has 2 paths and policy of Most Recently Used iScsi sw iqn.1998-01.com.vmware:esxhost-41e85afeiqn.199204.com.iscsi:a0 vmhba32:0:0 Standby preferred iScsi sw iqn.1998-01.com.vmware:esxhost-41e85afeiqn.199204.com.iscsi:b0 vmhba32:1:0 On active



The command esxcfg-vmhbadevs -m generates a compact list of the LUNs currently connected to the ESX host. The output appears similar to: vmhba1:0:0:3 /dev/sda3 48f85575-5ec4c587-b856-001a6465c102 vmhba2:0:4:1 /dev/sdc1 48fbd8e5-c04f6d90-1edb-001cc46b7a18 vmhba2:0:3:1 /dev/sdb1 48fbd8be-b9638a60-aa72-001cc46b7a18 vmhba32:0:1:1 /dev/sde1 48fe2807-7172dad8-f88b-0013725ddc92 vmhba32:0:0:1 /dev/sdd1 48fe2a3d-52c8d458-e60e-001cc46b7a18



The command ls -alh /vmfs/devices/disks lists the possible targets for certain storage operations. The output appears similar to: lrwxrwxrwx 1 root root 58 Oct 16 12:54 vmhba2:0:3:0 -> vml.0200030000600805f300124a90ca40a0bcd05c00294d5341313030 lrwxrwxrwx 1 root root 60 Oct 16 12:54 vmhba2:0:3:1 -> vml.0200030000600805f300124a90ca40a0bcd05c00294d5341313030:1 lrwxrwxrwx 1 root root 58 Oct 16 12:54 vmhba2:0:4:0 -> vml.0200040000600805f300124a9006d5bbdeb08b002a4d5341313030 lrwxrwxrwx 1 root root 60 Oct 16 12:54 vmhba2:0:4:1 -> vml.0200040000600805f300124a9006d5bbdeb08b002a4d5341313030:1 lrwxrwxrwx 1 root root 58 Oct 16 12:54 vmhba2:1:3:0 -> vml.0200030000600805f300124a90ca40a0bcd05c00294d5341313030 lrwxrwxrwx 1 root root 60 Oct 16 12:54 vmhba2:1:3:1 ->

vml.0200030000600805f300124a90ca40a0bcd05c00294d5341313030:1 lrwxrwxrwx 1 root root 58 Oct 16 12:54 vmhba2:1:4:0 -> vml.0200040000600805f300124a9006d5bbdeb08b002a4d5341313030 lrwxrwxrwx 1 root root 60 Oct 16 12:54 vmhba2:1:4:1 -> vml.0200040000600805f300124a9006d5bbdeb08b002a4d5341313030:1 The following are definitions for some of the identifiers and their conventions: 

vmhba:: This identifier can be used to identify either a LUN or a path to the LUN. When ESX detects that paths associated to one LUN, each path is assigned this identifier. The entire LUN then inherits the same name as the first path. When using this identifier for an entire LUN, the identified is called the canonical name. When this identifier is used for a path it is called the path name. These naming conventions may vary from ESX host to ESX host, and may change if storage hardware replaced. This identifier is generally used for operations with utilities such as vmkfstools. Example: vmhba1:0:0 = Adapter 1, Target 0, and LUN 0.



vmhba::: This identifier is used in the context of a canonical name and is used to identify a partition on the LUN or disk. In addition to the canonical name, there is a : appended to the end of the identifier. The represents the partition number on the LUN or Disk. If the is specified as 0, then it identifies the entire disk instead of only one partition. These naming conventions may vary from ESX host to ESX host, and may change if storage hardware replaced. This identifier is generally used for operations with utilities such as vmkfstools. Example: vmhba1:0:0:3 = Adapter 1, Target 0, LUN 0, and Partition 3.



vml. or vml.: The VML Identifier can be used interchangeably with the canonical name. Appending the : works in the same way described above. This identifier is generally used for operations with utilities such as vmkfstools.



/dev/sd or /dev/sd This naming convention is not VMware specific. This convention is used exclusively by the service console and open source utilities which come with the service console. The represents the LUN or Disk and is assigned by the service console during boot. The optional represents the partition on the LUN or disk. These naming conventions may vary from ESX host to ESX host, and may change if storage hardware replaced. This identifier is generally used for operations with utilities such as fdisk and dd. Note: VMware ESXi does not have a service console; disks are refered to by the VML Identifier.



The is a unique number assigned to a VMFS volume upon the creation of the volume. It may be included in syntax where you need to specify the full path of specific files on a datastore.

Datastore creation error: VmkfsLib_ScanAdapter: Failed to do a volume cache rescan Status: 0xbad003f Symptoms 

When you try to create a datastore using VMware Infrastructure Client connected to VirtualCenter, an error indicates that the host cannot access /vmfs/volumes/, where is a datastore that does not exist. If you refresh the datastore list, you see that the datastore was successfully created.



If you run vmkfstools -V you see the error: # vmkfstools -V VmkfsLib_ScanAdapter: Failed to do a volume cache rescan Status: 0xbad003f Failed scanning adapter 'vmfs-only' : No connection Error: Opaque service console status



If you run esxcfg-rescan, you see the error: # esxcfg-rescan vmhba32 Doing iSCSI discovery. This can take a few seconds ... Rescanning vmhba32 ... On scsi3, removing: 6:0 9:1 9:2 9:3 9:4. On scsi3, adding: 1:0 6:0 9:1 9:2 9:3 9:4. VmkfsLib_ScanAdapter: Failed to do a volume cache rescan Status: 0xbad003f Failed scanning adapter 'vmfs-only' : No connection Error: Opaque service console status Done.

Resolution This issue occurs when a datastore used by running virtual machines is unavailable. A datastore can be unavailable if, for example, a LUN is unpresented or if the paths leading to the datastore’s LUN are dead.

Problem with migrate to new datastore (error the method is disabled)

I have a problem when I try to move a Virtual machines storage to another datastore. All servers moved fine but for one server. When I try o move it I get Validation succeded and hits the button to start the migration. Then I get an error:

The method is disabled by "SYM-INCR 25-09-2011 23:03" (that is the name my snapshots get when running the Symantec Backup Exec backups). Error stack Call "VirtualMachine.Relocate" for object "sfsvargbroweb" on vCenter Server "ADMVC03" failed. (sfsvargbroweb is the name of the virtual machine and ADMVC03 is the vCenter server).

If I go to Snapshot manager there are no snapshots on the server.

Is there some kind of lockfile that is the problem so the server thinks that there are a snapshot? How do I fix it? Haven't tried moving the server when having it powered off.

/Freddie Tags: migration, snapshot, vcenter, esx_4

fringie 7 posts since Jan 18, 2011 1. Oct 6, 2011 1:52 AM in response to: fringie Re: Problem with migrate to new datastore (error the method is disabled) Tried this: http://kb.vmware.com/kb/1002310

Took a snapshot and then deleted all. Didn't solve the problem. Also I can't see any snap-shot files in the filestructure.  

Report Abuse

fringie 7 posts since Jan 18, 2011 2. Oct 6, 2011 2:47 AM in response to: fringie Re: Problem with migrate to new datastore (error the method is disabled) When turned off the migrateoption is greyed out.  

Report Abuse

Virtualinfra 108 posts since Mar 7, 2010 3. Oct 6, 2011 3:41 AM in response to: fringie Re: Problem with migrate to new datastore (error the method is disabled) What is the hardware version of the VM? What is the vCenter version your using.? Is vMotion is successfully happening.

Try to vMotion the virtual Machine to another host and try SVmotion if that works or not? Thanks & Regards Dharshan S VCP 4.0  

Report Abuse

a.p. 8,193 posts since May 18, 2007 4. Oct 6, 2011 3:42 AM in response to: fringie Re: Problem with migrate to new datastore (error the method is disabled) Did you already take a look at the vmx file? Maybe Symantec added some settings which were not removed!?

André  

Report Abuse

Virtualinfra 108 posts since Mar 7, 2010 5. Oct 6, 2011 3:56 AM in response to: Virtualinfra Re: Problem with migrate to new datastore (error the method is disabled) good option to check the vmx also... Thanks & Regards Dharshan S VCP 4.0  

Report Abuse

fringie 7 posts since Jan 18, 2011 6. Oct 6, 2011 4:15 AM in response to: Virtualinfra Re: Problem with migrate to new datastore (error the method is disabled) Didn't find anything strange as far as I can see in the vmx-file.

On that vm it's hw version 4 (Guest OS Suse Linux 32-bit). vCenter is 4.1.0 vMotion works fine. Tried storage vmotion afterwards and still no success.  

Report Abuse

fringie 7 posts since Jan 18, 2011 7. Oct 6, 2011 4:17 AM in response to: fringie Re: Problem with migrate to new datastore (error the method is disabled) Is there any other file one should look into? I mean VC must read this name from the Backupsnapshot (SYM-INCR 25-09-2011 23:03) from somewhere/some file.  

Report Abuse

Virtualinfra 108 posts since Mar 7, 2010 8. Oct 6, 2011 4:22 AM in response to: fringie Re: Problem with migrate to new datastore (error the method is disabled) There was a good clue to take a look at the vmx file. so Please plaste the vmx file information to further help.

# cat xxxxx.vmx and copy the content and paste here.. Thanks & Regards Dharshan S VCP 4.0  

Report Abuse

fringie 7 posts since Jan 18, 2011 9. Oct 6, 2011 5:05 AM in response to: Virtualinfra Re: Problem with migrate to new datastore (error the method is disabled) [root@biggreen sfsvargbroweb]# ll total 55069184 -rw------- 1 root root 42949672960 Oct 6 13:51 sfsvargbroweb_1-flat.vmdk -rw------- 1 root root 509 Oct 6 13:51 sfsvargbroweb_1.vmdk -rw-r--r-- 1 root root 37 Oct 6 13:10 sfsvargbroweb-5d8d7427.hlog -rw------- 1 root root 536870912 Oct 6 13:51 sfsvargbroweb-5d8d7427.vswp -rw------- 1 root root 13 Oct 6 10:42 sfsvargbroweb-aux.xml -rw------- 1 root root 12884901888 Oct 6 13:54 sfsvargbroweb-flat.vmdk -rw------- 1 root root 8684 Oct 6 13:52 sfsvargbroweb.nvram -rw------- 1 root root 507 Oct 6 13:51 sfsvargbroweb.vmdk -rw------- 1 root root 605 Oct 6 13:46 sfsvargbroweb.vmsd -rwxr-xr-x 1 root root 2787 Oct 6 13:51 sfsvargbroweb.vmx -rw------- 1 root root 268 Oct 6 13:51 sfsvargbroweb.vmxf -rw-r--r-- 1 root root 3960779 Aug 23 11:21 vmware-32.log -rw-r--r-- 1 root root 3376008 Aug 23 11:21 vmware-33.log -rw-r--r-- 1 root root 1803750 Oct 5 12:02 vmware-34.log -rw-r--r-- 1 root root 227947 Oct 6 11:38 vmware-35.log -rw-r--r-- 1 root root 87709 Oct 6 13:10 vmware-36.log -rw-r--r-- 1 root root 168639 Oct 6 13:49 vmware-37.log -rw-r--r-- 1 root root 81305 Oct 6 13:52 vmware.log [root@biggreen sfsvargbroweb]# cat sfsvargbroweb.vmx cat: sfsvargbroweb.vmx: Device or resource busy [root@biggreen sfsvargbroweb]# cat sfsvargbroweb.vmx #!/usr/bin/vmware .encoding = "UTF-8" config.version = "8" virtualHW.version = "4" nvram = "sfsvargbroweb.nvram" deploymentPlatform = "windows" virtualHW.productCompatibility = "hosted" unity.customColor = "|23C0C0C0" tools.upgrade.policy = "manual" powerType.powerOff = "default" powerType.powerOn = "default" powerType.suspend = "default" powerType.reset = "default" displayName = "sfsvargbroweb" extendedConfigFile = "sfsvargbroweb.vmxf" floppy0.present = "true" scsi0.present = "true" scsi0.sharedBus = "none"

scsi0.virtualDev = "lsilogic" memsize = "512" scsi0:0.present = "true" scsi0:0.fileName = "sfsvargbroweb.vmdk" scsi0:0.deviceType = "scsi-hardDisk" sched.scsi0:0.shares = "normal" scsi0:1.present = "true" scsi0:1.fileName = "sfsvargbroweb_1.vmdk" scsi0:1.deviceType = "scsi-hardDisk" sched.scsi0:1.shares = "normal" ide0:0.present = "true" ide0:0.clientDevice = "true" ide0:0.deviceType = "atapi-cdrom" ide0:0.startConnected = "false" floppy0.startConnected = "false" floppy0.fileName = "/dev/fd0" floppy0.clientDevice = "true" ethernet0.present = "true" ethernet0.networkName = "dmz" ethernet0.addressType = "vpx" ethernet0.generatedAddress = "00:50:56:bb:21:56" guestOSAltName = "Other Linux (32-bit)" guestOS = "linux" uuid.bios = "50 3b 6b 28 64 51 87 60-15 b9 84 cd 1b b1 cb d0" vc.uuid = "50 2f 49 3f 96 8b 40 e7-4a bb 6f 83 2d cf 59 50" log.fileName = "vmware.log" snapshot.action = "keep" sched.cpu.min = "0" sched.cpu.units = "mhz" sched.cpu.shares = "normal" sched.mem.minsize = "0" sched.mem.max = "512" sched.mem.shares = "normal" cpuid.1.ecx = "R----R----R--R-0-----------H-R--" cpuid.1.ecx.amd = "R---------------------------R---" cpuid.80000001.ecx.amd = "------------------RR-RR---------" cpuid.80000001.edx = "----R---------------------------" cpuid.80000001.edx.amd = "--------------------------------" evcCompatibilityMode = "FALSE" guestCPUID.0 = "0000000b756e65476c65746e49656e69" guestCPUID.1 = "000106d100010800000822010febfbff" guestCPUID.80000001 = "00000000000000000000000120100800" hostCPUID.0 = "0000000b756e65476c65746e49656e69" hostCPUID.1 = "000106d108080800000ce3bdbfebfbff" hostCPUID.80000001 = "00000000000000000000000120100800" scsi0:0.redo = ""

scsi0:1.redo = "" tools.remindInstall = "false" userCPUID.0 = "0000000b756e65476c65746e49656e69" userCPUID.1 = "000106d108080800000822010febfbff" userCPUID.80000001 = "00000000000000000000000120100800" vmware.tools.requiredversion = "8194" replay.supported = "TRUE" vmotion.checkpointFBSize = "4194304" tools.syncTime = "FALSE" uuid.location = "56 4d ee 50 51 84 6a 92-0f 02 7e cd c5 bd 67 13" cleanShutdown = "TRUE" sched.swap.derivedName = "/vmfs/volumes/4a1a78f7-4068b92f-573700151720c901/sfsvargbroweb/sfsvargbroweb-5d8d7427.vswp" migrate.hostlog = "./sfsvargbroweb-5d8d7427.hlog" [root@biggreen sfsvargbroweb]# --------------Strange I got device or resource busy when I tried cat while the server was powered on, worked fine when I powered it off. Think it worked when I checked earlier today, or was It perhaps powered off when I looked earlier (have turned it off and on so many times now when trying different things to make storage vmotion work).

Also I put in the list of files that I got if there is some file that needs to be checked or if there is some file that should not be there.

Oh btw I also made a clone of the server and storage vmotion works fine on the clone. One solution might be to power it off, clone it to a new server and delete the original server and run the clone instead?

Though it would be good to solve why this happens so it doesn't happen again or if there is something else needing to be fixed.  

fringie

Report Abuse

7 posts since Jan 18, 2011 10. Oct 6, 2011 5:08 AM in response to: fringie Re: Problem with migrate to new datastore (error the method is disabled) Btw just out of curosity why does it say: deploymentPlatform = "Windows" in the vmx-file when it runs on a esx and the guest os is Suse Linux?  

Report Abuse

Virtualinfra 108 posts since Mar 7, 2010 11. Oct 6, 2011 5:25 AM in response to: fringie Re: Problem with migrate to new datastore (error the method is disabled) This is because : deploymentPlatform = "Windows, When you created a VM hardware you created under windows platform instead of linux platform Thanks & Regards Dharshan S VCP 4.0  

Report Abuse

Virtualinfra 108 posts since Mar 7, 2010 12. Oct 6, 2011 5:38 AM in response to: fringie Re: Problem with migrate to new datastore (error the method is disabled) I See the virtual Machine is in DMZ Network.

further troubleshooting

Please try svmotion and capture the error message in the vcenter task and events here,

followed by the error message from virtual machine latest vmware.log file and also from vmkernel.log in the ESX where the VM is register file.

Hopefully we will try to figure out this.. Thanks & Regards Dharshan S VCP 4.0  

Report Abuse

a.p. 8,193 posts since May 18, 2007 13. Oct 6, 2011 9:51 AM in response to: fringie Re: Problem with migrate to new datastore (error the method is disabled) Maybe it helps to restart the Management Agents (http://kb.vmware.com/kb/1003490) on the ESX host to "synchronize" the current state with vCenter Server.

André  

Report Abuse

clogs1312 1 posts since May 22, 2008 14. Oct 10, 2011 5:02 AM in response to: fringie Re: Problem with migrate to new datastore (error the method is disabled) Had a similar problem. Wasn't able to migrate a VM to another datastore, but after removing and readding to inventory the vm moved without error ----------------

vSphere Client does not open on any Windows operating systems with the error: parsing the server "" "clients.xml" file Symptoms  

vSphere Client does not open on any Windows operating systems. When trying to launch the vSphere Client you receive errors similar to: Error parsing the server "" "clients.xml" file. The type initializer for VirtualInfrastructure.Utils.HttpWebRequestProxy' threw an exception.

Resolution You cannot use vSphere Clients prior to the Update 1 to access the vCenter Server or ESX hosts because of a Microsoft update that targets the .NET Framework released on June 9th 2010. For more information, see the Microsoft Knowledge Base article 980773. Perform one of these two options to resolve the issue: 

Download and install vSphere Client 4.0 Update 1 (build 208111) or Update 2 (build 258672) depending on your environment: o

To download and install the vSphere Client for ESX, ESXi (paid version), and vCenter Server: a. Go the vSphere Download Center. Note: The vSphere Client .exe is part of the ESX, ESXi, or vCenter Server download binaries.

b. c. d. e.

Click Download next to your ESX, ESXi, or vCenter Server edition. Log in with your VMware Account credentials. Click Yes to agree to the EULA. Click the .exe link next to vSphere Client and Host Update Utility. Note: You do not need to download the entire vSphere suite, only the vSphere Client.

f. Follow the on-screen instructions to install the updated vSphere Client. o

To download and install the vSphere Client Update 1 for ESXi (free version): a. Go to the ESXi Download Center. b. Click Download for VMware ESXi 4.0 Update 1 c. Log in with your account credentials, or register for free.

d. Click Download next to vSphere Client and Host Update Utility. e. Follow the on-screen instructions to install the updated vSphere Client. 

Remove the Microsoft update from your Windows operating system. The vSphere Client works after the update is removed.

Note: This affects Windows XP, Windows 2003, Windows 2008, Windows Vista, and Windows 7. If the build number for your vSphere Client is 208111 or higher, then you have vSphere Client 4.0 Update 1 or later, and should not be affected by this issue. You can determine the version of vSphere Client by reviewing the build number located in the first line of a viclient.log file. Note: This file is located in:  

%USERPROFILE%\Local Settings\Application Data\VMware\vpx for Windows XP or 2003 %USERPROFILE%\AppData\Local\VMware\vpx for 64-bit Windows Vista/7 or 2008.

For example: 2010-05-19 03:08:58.508 Log for vSphere Client Launcher, pid=4756, version=4.0.0, build=build-208111, option=release

VMware vSphere Client error - Error parsing the server "[server]" "clients.xml" file Friday, 30 September 2011 13:51 Windows - Servers      

1 2 3 4 5

If you are trying to run older version of VMware vSphere Client on Windows 7, or fully updated Windows XP you may get following errors: VMware vSphere Client error - Error parsing the server "[server]" "clients.xml" file

The type initializer for VirtualInfrastructure.Utils.HttpWebRequestProxy' threw an exception. These errors are caused by an updated Microsoft .NET version. You have couple of options here. Probably easiest fix is to download latest version of vSphere Client from VMware. If for some reason you can't do this follow instructions below

1. Download system.dll file. This file is taken from older version of Microsoft .NET installation. 2. Copy this file to C:\Program Files\VMware\Infrastructure\Virtual Infrastructure Client\Launcher\lib On 64 bit OS path would be: C:\Program Files (x86)\VMware\Infrastructure\Virtual Infrastructure Client\Launcher\lib If lib folder doesn't exist then create it. 3. Open file C:\Program Files\VMware\Infrastructure\Virtual Infrastructure Client\Launcher\VpxClient.exe.config in a text editor and just before last line paste following code: 4. Control Panel > System > Advanced > Environment Variables In System Variables click New and add following system variable: Name: DEVPATH Value: C:\Program Files\VMware\Infrastructure\Virtual Infrastructure Client\Launcher\lib Remember that on 64 bit system instead of Program Files you have to use Program Files (x86) 5. Launch VMware vSphere Client again. This time it should run without any errors.

sqlcmd.exe on SQL 2008 fails - HResult 0x2, Level 16

Error ... what? I've built one of my test servers with SQL Server Express 2008 w/ Advanced Services. On a whole load of our servers I use SQL Express for local installations and SQLCMD.EXE works well as a good way to run scheduled backups. It's done this way because SQL Express doesn't support SSIS (SQL Server Integration Services), the component required for scheduled tasks. With SQL 2008, however, SQLCMD.EXE didn't work for me when I tried to setup scheduled backups. Here's how I fixed it ... The instance in question has the default name of SQLEXPRESS. Obviously this means that to connect to the server you need to use SERVER\SQLEXPRESS - that works fine from Management Studio and from the Java application installed on this particular server. When using SQLCMD.EXE from the command line the full error message looks like this: HResult 0x2, Level 16, State 1 Named Pipes Provider: Could not open a connection to SQL Server [2]. Sqlcmd: Error: Microsoft SQL Server Native Client 10.0 : A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online. Sqlcmd: Error: Microsoft SQL Server Native Client 10.0 : Login timeout expired.

Some Googling suggested that this problem could be caused by one of 2 things. Firstly, that the SQL Browser service isn't running - on my server it was running. Secondly, named pipes aren't enabled in the SQL Server Configuration Manager ... on mine they are. Hmmm. I've had problems in the past with named instances of SQL Server so I had a look at the properties of the named pipes configuration and, sure enough, it said \\.\pipe\MSSQL$SQLEXPRESS\sql\query. That *looks* ok, right? Yes but it's the cause of the problem. I changed the named pipe to the following. -------------------------------------------------------------------------------------------------------------------chitecture is important | code2plan Beta Release >>

Using SQL Server 2008 Express as a default instance Saturday, 13 September 2008 17:46 by jesse

Time to upgrade! After uninstalling SQL Server 2005, I decided that I would try running only the Express version of SQL Server 2008 on my laptop. After installing, I found that running SQLCMD against the (local) server no longer worked. For example, sqlcmd -s (local) -i SomeScript.sql Resulted in this error: HResult 0x2, Level 16, State 1 Named Pipes Provider: Could not open a connection to SQL Server [2]. Sqlcmd: Error: Microsoft SQL Server Native Client 10.0 : A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online. Sqlcmd: Error: Microsoft SQL Server Native Client 10.0 : Login timeout expired. I had installed SQL Server 2008 Express as the default instance (and I don't have any other version of SQL Server on this machine), and accepted the default instance name SQLEXPRESS. SQLCMD is connecting to the instance through named pipes, and the pipe name it connects on for the default instance is \\.\pipe\sql\query To connect to a named instance, it connects through the named pipe \\.\pipe\MSSQL$\sql\query Sure enough, after opening Sql Server Configuration Manager, I expanded the Sql Server Network Configuration, double-clicked on Named Pipes, and was able to see that the pipe name included the instance name SQLEXPRESS. After changing the pipe name to \\.\pipe\sql\query, my sqlcmd reference to (local) worked great. Whew.

--------------------------------------------

Error adding Fiber Channel Datastore to ESXi 4.1 Hello,

I have two ESXi 4.1 servers running on two HP BL460C G1 blade servers. I have a MSA2000 SAN that I am using for fiber channel based shared storage.

When I try to add a new datastore to my ESXi hosts via vCenter server, I receive the following error:

Call "HostDatastoreSystem.QueryVmfsDatastoreCreateOptions" for object "datastoreSystem-32" on vCenter Server "VCNTR.domain.com" failed.

When I try to add the new datastore to an individual ESXi host directly, I receive this error:

Call "HostDatastoreSystem.QueryVmfsDatastoreCreateOptions" for object "hadatastoresystem" on ESXi "192.168.248.112" failed.

I have two 100GB datastores hosted on this same SAN that I was going to replace with a 500GB datastore. The two 100GB datastores were added without any problems. I have since deleted those datastores, but they are still presented to my ESXi hosts by the SAN, so I can re-add them without any errors.

I have searched for other solutions or explanations to these errors, but hopefully someone here will be able to give me some insight.

Thanks,

Chris Tags: datastore, fiberchannel, esxi_4.1, esxi4.1, hostdatastoresystem

Carrottm 2 posts since Mar 10, 2010 1. Aug 10, 2010 3:07 AM in response to: ChristopherL Re: Error adding Fiber Channel Datastore to ESXi 4.1 Hi All,

I am experieincing the same issue when trying to add storage to my ESXi 4.1 Host.

Any help would be appreciated.

Thanks, Mark Carrott  

Report Abuse

Carrottm 2 posts since Mar 10, 2010 2. Aug 10, 2010 4:27 AM in response to: ChristopherL Re: Error adding Fiber Channel Datastore to ESXi 4.1 Hi All,

I have resolved my issue by going into the RAID configuration on the the VM Host Server and performing a "fast initialize" on the disk. After rebooting the server I successfully added my new datastore.

Hope this helps.

Thanks, Mark  

Report Abuse

rohanpower 5 posts since Feb 12, 2010 3. Aug 26, 2010 9:25 PM in response to: ChristopherL Re: Error adding Fiber Channel Datastore to ESXi 4.1

I am seeing the same error at the moment, and I can't seem to find a way to fix it.

Error Message: Call "HostDatastoreSystem.QueryVmfsDatastoreCreateOptions" for object "datastoreSystem67" on vCenter Server "***SVR-VC01" failed.

Background: We have a MSA2012i SAN connected via iSCSI. We just recently deleted an existing vdisk that held a couple of VMDK volumes, then created a new vdisk exactly as before, with the same sized volumes. (This was done to change the RAID level) Nothing changed on the ESXi 4 & 4.1 Hosts between deleting and recreating the vdisks and volumes.

I have rebooted the VC Server and 1 of the two ESXi hosts but still no go. Haven't rebooted the SAN or the other host. Planning on rebooting the other host out of hours. Have disabled and re-enabled software iSCSI on the ESXi 4.1 host.

One thing to note is that this issue only happens on the first volume (LUN1). When I go to create a datastore on the second or third volumes, ESXi has no issues?

 

Report Abuse

ChristopherL 2 posts since Aug 4, 2010 4. Aug 27, 2010 5:16 AM in response to: ChristopherL Re: Error adding Fiber Channel Datastore to ESXi 4.1 My error was actually due to the drive controller on the SAN. The management interface of the SAN became unavailable and all the LUNs that were being newly provisioned were not accessible. All other LUNs that were already provisioned continued to work fine, just new disks were not able to be used. Finally after some scheduled downtime, we were able to reboot the SAN's drive controller, and the error was resolved.  

Report Abuse

rohanpower 5 posts since Feb 12, 2010 5. Aug 30, 2010 5:55 PM in response to: rohanpower Re: Error adding Fiber Channel Datastore to ESXi 4.1

Just in case anyone else has the same issue, and couldn't be resolved using the other posts above, this is what I did:

     

Disabled the SW iSCSI Adapter then rebooted the host Removed the vDisk on the SAN Updated the SAN to the latest firmware (J210P23–02) Created a new vDisk in Offline rather than Online mode Created two new volumes Enabled the SW iSCSI Adapter on the host, and then added the devices again.

After that everything came good. Obviously some of these steps may not be required (updating the firmware for example), but it may help someone in the future to know.

 

Report Abuse

aafeefi 1 posts since Jan 31, 2008 6. Sep 24, 2010 12:44 PM in response to: ChristopherL Re: Error adding Fiber Channel Datastore to ESXi 4.1

hi all,

actually i faced this problem when i tried to add any iscsi storage with size more than 2TB. please advise if there is any limitation for iscsi storage size with esx 4.1.

regards, Amjad Afeefi  

Report Abuse

MichaelW007 29 posts since Jan 27, 2010 7. Jan 26, 2011 1:58 PM in response to: aafeefi Re: Error adding Fiber Channel Datastore to ESXi 4.1 Yes, as per vSphere 4.1 maximums the size of a datastore is limited to 2TB - 512b.

-------------------------------

Call fails for “HostDatastoreSystem.QueryVmfsDatastoreCreateOptions” for object “ha-datastoresystem” 

VMWare

by Erwin Recently I had to reinstall an existing Dell PowerEdge 2970 server. This server was running SUSE Linux Enterprise Server 10 SP2 (32bit) with Xen 3.2.0 as virtualization software (host), but I decided to put VMWare ESXi 4.1.0 on this box. This server has a single Dell PERC 5/I RAID Controller, with six 72GB SAS disks attached to it. I started by destroying the existing two RAID sets, and created 2 new RAID sets. One for the OS (RAID1) and the other for the datastore (RAID10). After the RAID sets where intialized I started the installation of VMWare ESXi and the server was up-and-running in less than 20 minutes. As part of the post installation configuration of the VMWare ESXi server I tried to add a second (local) datastore by using the Add Storage wizard in the vSphere Client. In the Add Storage Wizard, I specified Disk/LUN as Storage Type and selected the available disk (the RAID10 set).

After pressing Next, the Add Storage Wizard returned the following error:

An error occured during host configuration. Call “HostDatastoreSystem.QueryVmfsDatastoreCreateOptions” for object “ha-datastoresystem” on ESXi “192.168.1.61″ failed.

On a side note, if you close this error message and go a step backwards and continue forward it wil not give an error anymore. This is not a solution or workaround, because when the Wizard gets at the Formatting property sheet, it wil give the following error:

"De objectverwijzing is niet op een exemplaar van een object ingesteld" is dutch for "Object reference not set to an instance of an object"

I couldn‟t find more detailed information about this error in the eventlog of the vSphere client, located in the directory: %USERPROFILE%\Local Settings\Application Data\VMware\vpx. And a search on the Internet didn‟t gave a answer either. So I was forced to troubleshoot on my own. I was certain that the volume was consistent , because I had performed a full initialization and integrity check when the RAID10 disk was created. As shown in the second screenshot of this post, the Add Storage wizard indicated that the disk was blank. I further noticed that there was no Available space on this disk. Normally speaking it would show how much space is available on the disk, so I assumed that there was something wrong with the partition table of this disk. I couldn‟t find an option in the vSphere client to remove the existing partition info on this volume. So I logged on to the ESXi server by using a SSH connection and issued the command fdisk -l. The following output was shown:

The disk /dev/disks/naa.6001e4f023eb110013ffed36d40990cb is the disk on which I tried to create a datastore. I directly knew which disk I should look for, but it can be easily identified through the Add Storage wizard of the vSphere client. (See the first screenshot in this post) As you can see, this disk contains 3 partitions (p1, p2 and p3)! How is this possible? Probably a leftover of the previous installation, although I had destroyed the previous RAID set. Fdisk also gave some warnings, indicating a non-consistent partition table on this disk. I decided to remove the partitions manually and used the following commands to remove them: 1. fdisk /dev/disks/naa.6001e4f023eb110013ffed364d40990cb. fdisk is started in the context of this disk. 2. Press d and enter 1 for the partition number, partition 1 on this disk will be deleted 3. Press d again and enter 2, which will delete partition 2 4. Because partition 2 is an extended partition, partition 3 will be deleted as well. There is no need or even possible to delete the third partition. 5. Press w, to write the changes to disk

To verify the deletion of the partitions on this disk, I issued the fdisk -l command again:

After this procedure, I was able to create a datastore on this disk through the Add Storage wizard!

Cannot boot or start a virtual machine converted by VMware Converter Symptoms After a successful conversion, you may experience these symptoms:    

The resulting virtual machine fails to start the operating system The resulting virtual machine fails to boot and does not issue a STOP error The boot process halts at a black screen with a blinking cursor You may receive these errors: o Windows could not start because the following file is missing or corrupt: C:\windows\system32\hal.dll Please re-install a copy of the above file. o Windows could not start because the following file is missing or corrupt C:\windows\system32\config\system o Operating System not found o Non-system disk or disk error o NTLDR is missing Press any key to restart o A disk read error occurred o The Master boot record (MBR) of this virtual machine's hard disk does not contain valid bootstrap code. It is likely that the MBR was corrupted by an incorrect guest operating system installation or some other reason. The virtual machine cannot continue and will now power off. (For advanced user: The MBR is the last disk sector accessed by the virtual machine before the error.) Please consult our Web site at http://www.vmware.com/info?id=18 for common troubleshooting help. o STOP 0x79 MISMATCHED_HAL

Resolution Validate that each troubleshooting step below is true for your environment. These steps provide instructions or a link to a document. These steps are ordered in the most appropriate sequence to isolate the issue and identify the proper resolution. Do not skip a step. 1. Ensure that during step 3 of the conversion (View/Edit Options), you choose the appropriate disk controller for your operating system. o For Windows XP, choose Bus Logic. o For Windows 2003, choose LSI Logic.

o

For Windows 2008 and Windows 7, choose LSI Logic SAS. For more information on adjusting the adapter after conversion, see Adjusting the virtual SCSI adapter type in a VMDK file (1006621).

2. If the source contained more than one hard disk, try changing the virtual hard disk order on the resulting virtual machine by removing both virtual hard disks, then adding them in the reverse order. For information on adding secondary disks, see the Virtual Machine Administration Guide . 3. Rewrite the master boot record on the virtual hard disk and restore the Windows boot files using the FIXMBR and FIXBOOT commands from the Recovery Console. For more information, see Repairing boot sector problems in Windows NT-based operating systems (1006556). 4. Verify that no data was corrupted during the conversion process. Corrupted data can affect the virtual machine's ability to boot successfully. For more information, see Checking for disk corruption on the destination after using VMware Converter (1006559). 5. Verify that the hardware abstraction layer (HAL) library file is correct and that it is not corrupted for the destination virtual machine type. For more information, see HAL cannot be identified upon conversion of a virtual machine using VMware Converter and fails to boot (1038576). Note: If your problem still exists after trying the steps in this article, file a support request with VMware Support and note this KB Article ID in the problem description. For more information on filing a request, see How to Submit a Support Request.

NETWORK:

Troubleshooting virtual machine network connection issues Symptoms   

Virtual machines fails to connect to the network There is no network connectivity to or from a single virtual machine A TCP/IP connection fails to and from a single virtual machine

cannot-access-vm connect-vm manage-vm no-connectivity

Purpose This article troubleshoots virtual machine network communication problems.

Resolution Validate that each troubleshooting step below is true for your environment. The steps will provide instructions or a link to a document, for validating the step and taking corrective action as necessary. The steps are ordered in the most appropriate sequence to isolate the issue and identify the proper resolution. Do not skip a step. 1. Ensure that the virtual machine has no underlying issues with storage or it is not in resource contention. As this might result in networking issues with the virtual machine. You can do this by logging to ESX/ESXi or Virtual Center/vCenter Server using VI/vSphere Client and logging to virtual machine console. For more information, see Troubleshooting a virtual machine that has stopped responding (1007819).

2. 3. 4. 5. 6. 7.

Note: If you are experiencing network connectivity issues with the ESX/ESXi host or with multiple virtual machines, see ESX/ESXi host has intermittent or no network connectivity (1004109). Verify that the virtual network adapter is present and connected. For more information, see Verifying virtual network adapter is present and connected to virtual machine (1003786). Verify that the networking within the virtual machine's guest operating system is correct. For more information, see Verifying the networking within the guest operating system (1003899). Verify the TCP/IP stack function. For more information, see Troubleshooting virtual machine TCP/IP connection issues (1007842). If this virtual machine was converted from a physical system, verify that there are no hidden network adapters present. For more information, see Networking Error, IP Address Already Assigned to Another Adapter (1179). Verify that the vSwitch has enough ports for the virtual machine. For more information, see Network cable of a virtual machine appears unplugged (1004883) and No network connectivity if all ports are in use (1009103). Verify that the virtual machine's IPSec configuration is configured correctly and that it is not corrupted. For more information, see IPSec error within a Windows virtual machine (1000797).

Note: If the problem still exists after trying the steps in this article:

 

Gather the VMware Support Script Data. For more information, see Collecting dianostic information for VMware products (1008524). File a support request with VMware Technical Support and note this Article ID (1003893) in the problem description. For more information, see How to Submit a Support Request.

-------------------------

Troubleshooting a virtual machine that has stopped responding Symptoms A virtual machine running on VMware ESX/ESXi does not respond to any external input or exhibit any activity. Specifically:     

Guest OS does not respond to keyboard or mouse activity at the console. Guest OS does not respond to network communication, including ping, RDP, SSH, etc. Virtual machine console screen is static, and does not change or refresh. Tasks performed on the virtual machine fail, timeout, or do not start. Virtual machine does not produce network or disk traffic.

Purpose This article provides steps to isolate possible causes of a vSphere virtual machine becoming unresponsive. An unresponsive virtual machine does not respond to any connection attempts and may be unable to respond to any attempts to power cycle it. There are a variety of reasons a virtual machine can end up in an unresponsive state. This article enables you to identify and resolve these common causes, and, when resolved, return the virtual machine to an operational state. It is possible to hard power off a virtual machine without troubleshooting the cause, but this will prevent collection and analysis of information which could assist with determining the root cause of the outage. For more information about shutting down the virtual machine, see Powering off a virtual machine on an ESXi host (1014165) and Powering off an unresponsive virtual machine on an ESX host (1004340).

This article assumes that the issue is currently occurring. If troubleshooting an issue that occurred in the past, some required information may be unavailable. Resolution The services a virtual machine provides may become unresponsive or unreachable due to a number of causes, including problems with the applications or guest OS within the virtual machine, problems with the virtual machine monitor or virtual devices, resource contention on the host, or issues with underlying storage or networking infrastructure. If the guest OS is producing any activity, it is successfully running. In this case, unresponsiveness is likely due to a connectivity problem or resource contention, or is specific to a higher-level component such as an application or service running within the guest OS.

Validate the scope It is important to have accurate symptoms and an understanding of the scope of a problem. To confirm the scope of the problem, work through these checks: 1. Confirm that the virtual machine is actually unresponsive. It is possible that the virtual machine is not responding via one interface, but is functioning correctly on others. For more information on testing whether a virtual machine is genuinely unresponsive, see Confirming whether virtual machine is unresponsive (1007802). If a virtual machine is responsive, but performing poorly, see Troubleshooting ESX virtual machine performance issues (2001003). 2. Verify that the virtual machine is powered on. If the virtual machine has been powered off unexpectedly, power it back on and then troubleshoot the cause of the unexpected shutdown. For more information, see: o o

Powering on an ESX/ESXi host's virtual machine (1003738) Determining why a virtual machine was powered off or restarted (1019064).

Note: If a virtual machine is powered off and cannot be powered back on, see Troubleshooting a virtual machine that is unable to power on (2001005). 3. Determine whether this issue is affecting multiple virtual machines or just one. If multiple virtual machines are affected, consider the similarities between the affected virtual machines when attempting to narrow the potential scope. In particular, focus on shared infrastructure which the group of affected virtual machines depend on, and whether all virtual machines depending on that common infrastructure are affected. For more information, see Assessing commonalities of an outage affecting multiple virtual machines (1019000). 4. Determine whether the gest OS is responsive to interaction at the virtual machine console. If an issue has been isolated to the guest OS or applications within the virtual machine, and the guest

OS is responsive at the console, interact with the guest OS at the console to address the problem. For more information, see Troubleshooting virtual machine network connection issues (1003893). 5. Determine whether the guest OS or its application services are responsive to interaction via the network. If the guest OS or services respond to network communication but the console is unresponsive or non-functional, see Cannot open the virtual machine console (749640) or Ensuring that a virtual machine is not inaccessible due to a VMware vCenter or VirtualCenter issue (1007808). 6. Determine whether the guest OS has reported any critical errors to the console, and is sitting in a halted state. For more information, see Identifying critical Guest OS failures within virtual machines (1003999). 7. Determine whether the ESX/ESXi host is unresponsive too. If the host is unresponsive as well, the scope is larger than initially assumed. For more information, see Determining why an ESX/ESXi host does not respond to user interaction at the console (1017135).

Identify the cause At this point, you have established that one or more virtual machines are unresponsive at both the virtual console and via the network. The host itself is responsive. A problem may exist with resource accessibility or contention, or with underlying storage or networking infrastructure. To identify the cause: 1. Determine whether the problem is triggered by an operation or task being performed on the virtual machine. For example, snapshot and vMotion operations both stun a virtual machine for brief periods of time while memory state is copied across the network or to disk. For more information, see Taking a snapshot with virtual machine memory stuns the virtual machine while the memory is written to disk (1013163). 2. Some common configuration errors can lead to a virtual machine becoming unresponsive, such as while waiting for a resource. Review the virtual machine and host configuration. For more information, see: o o

Common ESX/ESXi host configuration issues which can cause virtual machines to become unresponsive (1007813) Common ESX/ESXi virtual machine configuration issues which can cause virtual machines to become unresponsive (1007814)

3. Virtual machines depend on functional backing infrastructure. If there is an issue with the backing storage or networking infrastructure which the virtual machine depends on, the virtual hardware which a virtual machine presents to the guest OS may be impacted. Address the underlying storage or networking issue. For more information, see:

o o o

ESX Server virtual machines stop responding due to shared storage connectivity issues (1004144) Verifying that ESX/ESXi virtual machine storage is accessible (1003751) Troubleshooting virtual machine network connection issues (1003893)

4. Virtual machines depend on available host resources (CPU, Memory), and the guest OS consumes those resources. A problem with resource availability or scheduling inside or outside the virtual machine may cause it to become unresponsive. The virtual machine may also be blocking on unavailable resources or spinning at 100% vCPU utilization. For more information, see Troubleshooting a virtual machine that has stopped responding: VMM and Guest CPU usage comparison (1017926).

Action Plan At this point, you have established that the host running the virtual machine(s) is both responsive and not encountering any shared storage or networking infrastructure issues. The guest OS has not failed with a critical error, but remains unresponsive at the virtual machine console and via the network. Take action to recover or collect information about the unresponsive virtual machine based on the architectural layer which is suspect: 

If an issue has been isolated to the guest OS, or the %RUN is relatively high, but the virtual machine monitor is functioning correctly, move investigation to within the virtual machine's guest OS or applications. A guest OS can become unresponsive inside a virtual machine in the same way it can on physical hardware. For more information, see Troubleshooting unresponsive guest operating system issues (1007818). 1. Collect performance data while the problem is happening. For more information, see Using performance collection tools to gather data for fault analysis (1006797). 2. Attempt to manually induce a panic of the kernel inside the guest OS to collect additional information about its internal state. For more information, see:    

Using virtual NMI facilities to troubleshoot unresponsive virtual machines on ESX/ESXi (1009187) Microsoft article 927069: How to generate a complete crash dump file or a kernel crash dump by using an NMI on a Windows-based system Microsoft article 303021: How to generate a memory dump file when a server stops responding Linux Documentation Project article: Magic SysRq key Note: The preceding links were correct as of August 31, 2011. If you find a link is broken, provide feedback and a VMware employee will update the link.

If useful diagnostic information is produced by the guest OS in response to one of these events, engage the guest OS vendor to investigate further. 3. If step 2 does not produce useful information, suspend the virtual machine to collect information about its internal state and open a case with VMware Support. For more information, see: a. Suspend the virtual machine and collect the .vmss suspend state file. For more information, see Suspending a virtual machine on ESX/ESXi to collect diagnostic information (2005831). b. Collect logs from the host running the virtual machine. For more information see Collecting diagnostic information for VMware products (1008524). c. Power the virtual machine back on, then reset it. d. Engage VMware Support, providing the information collected in steps 1, 3a and 3b. For more information, see How to File a Support Request.

Note: If the virtual machine cannot be suspended because another management task is in progress, see Collecting information about tasks in VMware ESX and ESXi (1013003) and Restarting the Management agents on an ESX or ESXi Server (1003490). If attempts to suspend the virtual machine fail and no management task appears to be present, skip to the next section and attempt to crash the virtual machine. 

If an issue has been isolated to the virtual machine monitor, or the %WAIT is relatively high, or attempts to suspend the virtual machine have failed, collect performance data and forcefully crash the virtual machine to collect additional information about its internal state. 1. Collect performance data while the problem is happening. For more information, see Using performance collection tools to gather data for fault analysis (1006797). 2. Crash the virtual machine to collect information about its internal state. For more information, see Crashing a virtual machine on ESX/ESXi to collect diagnostic information (2005715). Note: If attempts to crash the virtual machine fail, skip to the next section and attempt to crash the host. 3. Engage VMware Support, providing the information collected in steps 1 and 2. For more information, see How to File a Support Request.



If an issue has been isolated to the virtual machine monitor, but attempts to suspend or crash the virtual machine fail, this reflects a problem with the VMkernel. Collect a log bundle from the

host, evacuate all unaffected virtual machines from the host, and use an NMI to intentionally generate a purple diagnostic screen. 1. Collect performance data while the problem is happening. For more information, see Using performance collection tools to gather data for fault analysis (1006797). 2. Move all unaffected virtual machines off of the host using vMotion. If possible, use Maintenance Mode to prevent additional virtual machines from being started on the host. 3. Configure the host to panic on receiving a non-maskable interrupt, and then issue an NMI to trigger a panic. For more information, see Using hardware NMI facilities to troubleshoot unresponsive hosts (1014767). 4. After the host has generated a purple diagnostic screen and completed dump of diagnostic information, take a screenshot or photograph of the console and restart the host. 5. Collect diagnostic information from the host. For more information, Collecting diagnostic information from an ESX or ESXi host that experiences a purple diagnostic screen (1004128). 6. Engage VMware Support, providing the information collected in steps 1, 4 and 5. For more information, see How to File a Support Request.

----------

ESX/ESXi hosts have intermittent or no network connectivity Symptoms  

An ESX/ESXi host is experiencing intermittent connectivity issues An ESX/ESXi host cannot connect to the network.

Purpose This article provides steps to verify the configuration of the network and to correct any issues that may be causing the lack of network connectivity.

Resolution Note: If you are experiencing a complete lack of network connectivity from a particular virtual machine, see Troubleshooting virtual machine network connection issues (1003893). For information on how to test

network connectivity to and from the ESX/ESXi host or virtual machines, see Testing network connectivity with the Ping command (1003486). Validate that each troubleshooting step below is true for your environment. The steps provide instructions or a link to a document, for validating the step and taking corrective action as necessary. The steps are ordered in the most appropriate sequence to isolate the issue and identify the proper resolution. Please do not skip a step.

1. Verify that the network adapter and server hardware are supported. For more information, see 2. 3. 4.

5. 6. 7. 8. 9. 10.

Verifying ESX/ESXi host hardware (System, Storage, and I/O devices) are supported (1003916). Verify that the network link is up. For more information, see Verifying a network link (1003724). Verify that proper VLAN IDs exist on the portgroup. For more information, see Configuring a VLAN on a portgroup (1003825). If you are using NIC teaming on the virtual switch, verify that the physical switch ports are configured consistently for each teamed network adapter and that the proper load balancing policy is configured on the virtual switch. VMware recommends you to use the default Route based on the originating virtual port ID load balancing policy. If link aggregation on the physical switch is configured, use the Route based on ip hash load balancing policy. For more information, see NIC teaming in ESXi and ESX (1004088) and ESX/ESXi host requirements for link aggregation (1001938). Verify that the speed and duplex of the network links are consistent. For more information, see Configuring the speed and duplex of an ESX/ESXi Server network adapter (1004089). Verify the ESX host networking configuration. For more information, see Verifying ESX Server host networking configuration on the service console (1003796). Verify that port security is not configured on the physical switch ports. For more information, see Loss of network connectivity when port security is configured on the physical switch (1002811). Verify that portfast (or equivalent) is enabled on all of the ESX host's physical switch ports. For more information, see STP may cause temporary loss of network connectivity when a failover or failback event occurs (1003804). Verify the integrity of the physical network adapter. For more information, see Verifying the integrity of the physical network adapter (1003686). Verify that no duplicate IP addresses exist on the network. For more information, see Warning for Duplicate IP Address for VMware VMotion Port Group (10165) or Duplicate IP address detected (1020647).

Note: If your problem still exists after trying the steps in this article, please:  

Gather the VMware Support Script Data. For more information, see Collecting Diagnostic Information in a VMware Virtual Infrastructure Environment (1003689). File a support request with VMware Technical Support and note this KB Article ID in the problem description. For more information, see How to Submit a Support Request.

-----------

Storage vMotion fails with the error: virtual machine has virtual disk in link-clone mode Symptoms  

Storage vMotion fails on a virtual machine that has been moved previously using Storage vMotion. If you try to perform Storage vMotion, you see this error for each disk: Virtual machine has virtual disk in link-clone mode



Linked clones are not in use for this virtual machine.

Purpose This article is applicable if you experience these issues on virtual machines that do not use linked clones. When linked clones is enabled on the virtual machine, you cannot perform Storage vMotion of the virtual machine and this article does not apply.

Note: Linked clones is a VMware technology that is used primarily in VMware View and Lab Manager environments. For information on moving linked clone desktops, see How to delete and re-deploy a persistent desktop virtual machine without losing the persistent data (1026202). Resolution This issue can occur if vCenter Server has cached an incorrect reference to the virtual machine disk backing information, and no linked clones are in use. There are several different options to workaround this issue. Each workaround causes vCenter Server to refresh its cache of the virtual machine's disk backing information:    

Migrate the virtual machine between hosts. Create and delete a snapshot for the virtual machine. Unregister and re-register the virtual machine. Restart the VMware Management agents.

Migrating the virtual machine between hosts To migrate the virtual machine between hosts using vMotion or cold migrate: 1. Open the vSphere Client and connect to the vCenter Server as a user with administrative privileges.

2. 3. 4. 5. 6.

Go to the Hosts and Clusters Inventory view. Right-click the virtual machine. Click Migrate.... Migrate the virtual machine to an alternate host. Repeat the storage vMotion operation.

Creating and deleting a snapshot for the virtual machine To create and delete a snapshot for the virtual machine: 1. Open the vSphere Client and connect to the vCenter Server as a user that has administrative privileges. 2. Go to the Hosts and Clusters Inventory view. 3. Right-click the virtual machine. 4. Click Snapshots > Take Snapshot.... 5. Create a snapshot and provide all necessary information. 6. Right-click the virtual machine. 7. Click Snapshot > Snapshot Manager.... 8. Select the snapshot created in step 5 and click Delete All. 9. Click Close. 10. Repeat the Storage vMotion operation. Note: There may be redolog files on disk from former snapshots, even if the vSphere Client does not display any snapshots in the Snapshot Manager. For more information, see Committing snapshots when there are no snapshot entries in the snapshot manager (1002310).

Unregistering and registering the virtual machine To unregister and reregister the virtual machine: 1. Open the vSphere Client and connect to the vCenter Server as a user that has administrative privileges. 2. Shut down the virtual machine. 3. Remove the virtual machine from the Inventory. 4. Re-add the virtual machine to the Inventory. For more information, see Registering or adding a virtual machine to the inventory (1006160). 5. Start the virtual machine. 6. Repeat the Storage vMotion operation.

Restarting the VMware Management agents For information on how to restart the management agents on a VMware ESX or ESXi host, see Restarting the Management agents on an ESX or ESXi Server (1003490).

-------------

Let’s troubleshoot VMware December 29, 2010 gunnalag Leave a comment 1. Scenario#1: Logon to Virtual Center Server Fails with generic error “Unable to connect to remote server” 1. Isolate if it‟s a Networking issue 1. adfind – Ensure the server name is typed correctly and is a valid Windows server in AD using adfind 2. ping – Ensure the remote vCenter server is reachable using ping 3. nslookup – Ensure the DNS entries are correct for the server by doing a nslo0kup 4. FQDN – use FQDN JIC if some DNS Zones entries issues is around causing the network connectivity 2. Isolate if it‟s a Windows server issue 1. telnet to 443 – Check the connectivity to the vCenter web service to ensure web service at least responding 2. sc query “vmware virtualcentre server” – Ensure that the VMware virtualcentre server service is running fine 3. eventvwr – check if there are any vcenter errors in application log or service/system errors in system log 4. vpxd.exe – check the vpxd log file %ALLUSERSPROFILE%\VMware\vmware virtualcentre\Logs\vpxd#.log” 5. Attempt to start the service if it‟s not running 6. If service fails with error, correlate it with the log statements in vpxd log files 3. Isolate if it‟s a DB connectivity issue 1. In case if there are any ODBC errors in vpxd log file follow below steps according to the error message “usually authentication errors” 2. if the DB used is a locally installed SQlExpress version or Access DB, verify that the SQL Server DB service is running fine 3. If the DB is a remote SQL/Oracle DB, check the reachability to the server 4. Open the ODBC Manager and very the DSN entry and test the connectivity to the server 5. In case of authentication issues, try to use 1. the other login account or 2. attempt with “Integrated Windows authentication”

6. If still there is any issue, correlate the issue with SQL authentication/security and DB logging 7. Once all the issues are isolated and if it‟s indeed DB connection issues that got fixed, then attempt to start the “vmware virtualcentre server” service on vCenter server

---------------

Expanding a drive within a VMWare image November 20th, 2006 Sean Leave a comment Go to comments There are many ways to expand a drive within a VMWare image and I have outlined two approaches below depending on your setup. I have used both these methods successfully on a Windows Server 2003 Vmware image (Vmware Workstation 5.0)

Method 1 Advantages: You get to keep your Vmware snapshots Limitations: You can‟t extend all volumes, I had trouble trying to extend the system/boot volume. According to a Microsoft Knowledge Base article you can‟t extend a volume if the system page file is located on the partition. You could try to move the page file to a partition that you do not wish to extend. My setup: C: System Drive 10 GB D: Data 4GB Let‟s say that we want to expand the D drive from 4GB to 10GB 1. Shutdown the VMWare Image 2. Add a new disk to the VMWare image (the size should be that which you want to allocate to another volume, in this example 6GB). 3. Boot up the VMWare image into Windows and go to Disk Management (Administrative Tools -> Computer Management -> Disk Management) 4. A wizard should be displayed to initialize and convert the new disk. If you are not prompted then you may need to Rescan. Do not assign a new volume at this stage. 5. You need to convert the D drive and the new drive to both be dynamic disks. 6. Once both the drives are dynamic, then right click on the D drive and select Extend Volume. 7. Follow the wizard to allocate the new space to the D drive. Once you complete the wizard, the D drive should now be 10GB. That‟s method 1 complete. If this worked for you, great! If not then take a look at Method 2 below.

Method 2 Advantages: You can successfully expand a system/boot drive. Limitations: You have to remove all your snapshots (the vmware-vdiskmanager utility requires this.) My Setup: C: System Drive 8GB D: Data 4GB E: Data 4GB

Let‟s say that we want to expand the C volume from 8GB to 12GB. Assume that the C drive is full to capacity (that‟s why we are expanding it right!) 1. If the Vmware image is running, shut it down. 2. You need to find the name of the Vmware (.vmdk) file that represents the virtual disk that you want to expand. Go to VM->Settings and locate the drive that you want to expand. The disk file field on the right hand side will display the name

of the .vmdk file.

3. You need to remove any snapshots present in the Vmware image. Note: By deleting the snapshots the system still remains in its current state. 4. Open up a command prompt and issue the following command: vmware-vdiskmanager -x 12GB “Windows Server 2003 Standard Edition.vmdk” where 12GB is the desired size of the expanded volume.

5. This expansion may take a while but when it is complete, boot up into Windows and go to Disk Management (Administrative Tools -> Computer Management -> Disk Management) You should see something similar to the following screenshot. The disk has been expanded but the new space is as yet unallocated. Now in theory you should be able to use the Windows diskpart utility to allocate this space but this would not work on a

boot/system disk for me.

6. So with diskpart not up to the task, what are your options?: o Buy some Windows partition software (i.e. Partition Magic) at a cost of USD200USD300 for a Server edition or o You could use Knoppix (free) to allocate the new space to the C drive. Knoppix is a bootable Live Linux CD and it happens to have a utility called QTparted. 7. Since everyone likes free stuff, I went with Knoppix in this case. Download Knoppix (the iso image) to the machine which hosts the VMware image. 8. Now you need to point your CD/DVD drive in your VMWare image to the iso image. How do I do this? 9. Reboot your VMware image and have it boot from the cd drive. You may need to modify the BIOS settings in order to change the boot order to have the system boot from the CD drive. The exact settings that you need to modify will depend on the exact BIOS that you have. It should be straightforward to change this. 10. After Knoppix boots , start at the large letter K at bottom left. Select K | System | QTParted to launch the utility. You will see the list of disks within the VMware on the left hand side. Selecting one of these disks will provide you with more details about that disk. In the following screenshot you can see the C drive with the 8GB allocated and the

4GB unallocated.

11. Right click on the drive that you want to expand and select Resize.

12. A dialog will be displayed and you can expand the allocated space by dragging the partition to the right. You could also type in the values into the textboxes. After allocating all the new space you now see that the total size is approx. 12GB. 13. Click OK to exit the dialog. At this stage you have not commited the changes yet. Go to the main menu of QTParted and select the Commit option. The following prompt will be displayed.

14. Choose Yes to commit the changes. You will see the following two screenshots as it commits the changes.

15. After the operation completes, you should see that the new volume now is 12GB in size with only 8GB in use (we now have 4GBs free )

16. Now, shutdown the Vmware image and boot back into Windows (you may need to change the boot order to accomplish this).

17. Don‟t be alarmed if CHKDSK wants to run some tests on your drive. It detected the changes you made in Knoppix and wants to verify that everything is ok.

18. You may need to restart Windows again because it will have detected new hardware.

19. Once Windows is back up, go to Disk Management (Administrative Tools -> Computer Management -> Disk Management) and you should now see that the C drive has the extra 4GB as promised.

20. Also, in Windows Explorer, right click on the C drive and choose Properties. You will see now that there is over 4GB of free space. )

Job Done! Lessons for the future: When you are creating future Vmware image make sure to allocate plenty of space to the drives. You can set that they only grow as required so they only ever take up the space that you require at any point in time.

---------------

How to Increase VMware Hard Disk Space 89

rate or flag this page By expectus

See all 6 photos

Ads by Google

Increase Virtual Machine Partition A common issue that people run into when using VMware is that once they have created there virtual machine and installed there OS and everything else they eventually run out of room and decide to increase the Virtual machines hard disk / partition. For this example I am running VMware server on Windows Vista , WIth one virtual machine running Linux Ubuntu 8.04 Hardy and will go through the steps to increase my harddrive size from 15GB to 25GB

Increasing Hard Disk Size on your Virtual Machine ( VMware ) Step 1. Firs thing to do is locate the location of vmware.exe on your PC typically it will be in C:\Program Files\VMware\VMware Server or VMware Workstation Once you have located your vmware.exe file open up command prompt on Windows. [Start -> Run -> type "cmd" in the window then press ENTER Now navigate into the vmware.exe directory , in this case simply C:\Program Files\VMware\VMware Server From here type vmware-vdiskmanager -x 25GB image.vmdk - 25GB in this case we would like to make the NEW size 25GB , this method will not remove any exisiting files - image.vmdk , Name of your Virtual Machine disk my example was Ubuntu.vmdk, if for some reason it doesn't work type the entire path of the vmdk file for example vmware-vdiskmanager -x 25GB D:\Virtual Machines\Ubuntu\Ubuntu.vmdk

Extending Partition / Hard Disk Now that you have used the VMware diskmanager to increase size of the Virtual Disk Space , this will create a new parition that 10GB in this case ( original 15GB + 10GB increase). This increased size will not automatically show up when we reboot our Virtual Machine. We need to

carry out a few extra steps to make use of this newly created space , this involves extending the partition or merging two partitions together. Step 2. Before trying to extend your harddrive to include the newly created partition, you will need to open vmdk file in a second virtual machine. The reason for this is that you cannot extend a partition on a drive that you are actually using system files on, It has to be set as the secondary drive. So go through the steps of making a new virtual machine and give it 4-5GB or so it doesn't need to be huge. You wont actually need to load any Operating system on it if you use the LiveCD. Before you run your second virtual machine you will need to add the harddrive you wish to increase onto that system , see screenshot

Select your 2nd/Other Virtual Machine, then select Edit virtual machine settings, Click Add, Select Harddrive and use exisiting virtual then load your original harddisk , in my case Ubuntu.vmdk

Choose to start Ubuntu without installing it, Run it off the CD (LiveCD)

Gparted Now that you have created your secondary virtual machine, Boot it up and in order to extend our virtual hard disk we will use a inbuilt linux tool called gparted , there are also many other partitioning programs out there including fdisk and many others. Now we want to load up our ubuntu LiveCD instead of having to install an Operating System, To get your virtual machine to boot up from your LiveCD do the following. - Use Daemon Tools or any other mounting program and mount your Ubuntu.iso image. - Start your virtual machine and soon as it starts press ESC to enter the boot menu and from here select CD-ROM - This will start your secondary virtual machine with the Ubuntu LiveCD

Running Gparted ( Partitioning Program ) To run gparted simply enter the terminal window and type "gparted" , and this will open the gparted gui window. ( Remember to issue this command with root privledges ) Once gparted has started you will see a window similar to below, We see two parition that we want to join together below circled in red, To extend the exisitng partition to the unallocated partition use the resize button to increase the size of the exisiting partition to increase the size. ( Ensure the harddrive is unmounted, and swapoff )

Not my original screenshot for this case but you get the drift:)

Resizing Partition , simply drag the handle across to pick a size or type in desired size

Finishing Up Now once you have resized your partition to include the unallocated partition size click apply. This may take sometime for it to resize the partition size. Once this is complete shutdown the virtual machine and remove the secondary harddrive you added in the previous section. Boot your original virtual machine and you should have successfully extended your hard disk space. Gparted LiveCD There is also a gparted LiveCD which can be run if you have trouble deleting and creating partitions. Simply mount the gparted LiveCD and follow the prompts.

Link: http://sourceforge.net/project/showfiles.php?group_id=115843&package_id=271779

Other Methods If you have made your way through this howto and still unable to get it to work you could try the Vmware Converter which can do all this tasks a lot easier, ( yeaa should've told you a lot earlier) its a free download and can perform partition extensions easily. VMware Converter Windows Method After you have loaded the secondary harddrive on the secondary virtual machine (windows in this case) to fix the partitions. Run the command prompt and enter "diskpart.exe" or if that doesn't work locate its location and run it through cmd using the directory locations. Steps - type - show volume select volume 2 ( in this case , double check to make sure you have the right one) extend exit Then simply shutdown this virtual machine, remove the secondary hard disk, then startup the original virtual machine. Any QUESTIONS let me know I will try and help you:)

VMware P2V Converter Best Practices Posted on 8 January 2010

The best/easiest approach to converting a Windows operating system from a physical machine to a virtual machine is to perform a hot migration with VMware Converter installed locally on the source (physical machine) operating system. Below an overview of the different steps involved in a P2V (Physical to Virtual) conversion. Note that these steps only apply to Windows operating systems.

BEFORE CONVERSION      

Confirm that the source server has at least 200 MB free disk space on its system volume. This space is required to operate the disk snapshot features in VMware Converter Confirm that the source machine has at least 364 MB RAM If you have software mirroring, break the mirror (but not the data) since VMware converter does not support software mirrors. Clean-up any temporary files and un-needed data Change all hardware related services to "disabled" startup mode Download the following utilities/scripts and install them to the directory c:\Temp\P2V on the source server. The following files are included:

o o o o o o o o o o o o o o o o o o o o

 

 

p1-HWPhysical.bat p2-installP2VConverter.bat p3-SystemConfigUtil.bat v1-HWVirtual.bat v2-vmprofile.bat v3-enablehdwacc.bat v3-enablehdwacc.vbs v4-renameNICs.bat v4-renameNICs.vbs v5-setip.bat v6-uninstallP2Vconverter.bat v7-SystemConfigUtil.bat v8-hiddendevices.bat v9-HALupdate.bat PSPCleaner.exe comm.exe libiconv2.dll libintl3.dll devcon.exe VMware vCenter Converter Standalone 4.01 build 161434 (See also http://www.vmware.com/download/converter/) Log on to the source machine (mstsc -v: servername ?F -console) with a local administrator account and open a command window Run c:\Temp\P2V\p1-HWPhysical.bat o The script creates a list of all devices of the physical machine (including non-present devices) Install the VMware vCenter Converter Standalone software by executing c:\Temp\P2V\p2installP2VConverter.bat Run the System Configuration Utility on the source server by executing c:\Temp\P2V\p3SystemConfigUtil.bat to reduce the number of services and applications running on startup (all software except for all Microsoft Services and VMware Converter Service). o On the General tab  Select "Selective Startup  Uncheck "Load Startup Items" o On the Services tab  Select "Hide All Microsoft Services"  Click "Disable All"  Mark the VMware vCenter Converter Agent  Mark the VMware vCenter Converter Server o Click "Apply" o Click "Close" o Click "Restart

CONVERSION 

Log on to the source machine with a local administrator account

  





Start the VMware vCenter Converter Standalone application (A shortcut should be available on the desktop) Select "Convert Machine" Specify Source o Select source Type: Powered-on machine o Specify the powered-on machine: This local machine o Click NEXT Specify Destination o Destination Type  Select destination type: VMware Infrastructure virtual machine  VMware Infrastructure details  Server: type in the IP address of the ESX server you want to convert to (or of your Virtual Center server)  User name: type the user name of an administrative account for the above server  Password: type the password of the above user name  Click NEXT o Host/Resource  Select the ESX host/group you want to convert to  Virtual machine name: type the name of the destination virtual machine (this is normally already set)  Datasource: select the datastore in which to place the destination virtual machine  Click NEXT View/Edit Options o Data to copy  Keep the defaults unless you need to re-size the partitions  Select "Ignore page file and hibernation file" o Devices  Select the number of processors: adapt if needed (Remember to revert the HAL after conversion if you are changing from a multi-processor to a uniprocessor machine)  Disk controller: select SCSI LSI logic for Windows 2003 and beyond  Memory for this virtual machine: adapt if needed o Network adapters  Choose the number of network adapters you need  Select the appropriate VLAN  Select Connect at power-on o Services  Go to tab "Destination Services"  Change starting mode to "Disabled" for all services you will not need in the virtual machine o Advanced options  De-select "Synchronize changes that occur to the source during cloning"  De-select "Power on target machine"  De-select "Install VMware Tools on the imported machine"  Select "Remove System Restore checkpoints on destination"  Select "Reconfigure destination virtual machine"



o Click NEXT Ready to Complete o Review the summary information o Click NEXT

AFTER CONVERSION   

    

      

 



Shut down the physical machine Use the VI client to log on to your virtual center server or to your ESX server Review and adjust the virtual hardware settings: o Adjust the number of NICs, CPUs o Remove any unnecessary devices such as serial ports, USB controllers, COM ports, floppy drives, … Start the virtual machine Verify if virtual machine boots correctly Stop the physical machine Log on to the virtual machine Run c:\Temp\P2V\v1-HWVirtual.bat o Creates a list of all devices that matches the virtual machine o Compares the list of all devices of the physical machine (created in the step "before conversion" with the list of all devices that matches the virtual machine (created in the previous step) o Removes all phantom harware o Rescans hardware o Reboots the server Change registry key regarding profile problem in VMWare (see VMware article) by executing c:\Temp\P2V\v2-vmprofile.bat Enable Video Hardware Acceleration by executing c:\Temp\P2V\v3-enablehdwacc.bat (this command calls the vbs script c:\Temp\P2V\enablehdwacc.vbs) Rename your network connections by executing c:\Temp\P2V\v4-renameNICs.bat (this command calls the vbs script c:\Temp\P2V\renameNICs.vbs) Set IP information on the first network connection by executing c:\Temp\P2V\v5-setip.bat %1 %2 %3 %4 %5 %6 %7 From this step on you should be able to connect to the virtual server via RDP Uninstall VMware vCenter Converter Standalone by executing c:\Temp\P2V\v6uninstallP2Vconverter.bat If you are converting from HP Proliant hardware you can clean up the HP hardware related drivers, utilities, agents using the HP Proliant Support Pack Cleaner from Guillermo Musumeci. Execute c:\Temp\P2V\PSPCleaner.exe Run the System Configuration Utility on the virtual server by executing c:\Temp\P2V\v7SystemConfigUtil.bat, select the "normal startup" option and reboot your server Show all hidden devices and uninstall any unused devices by executing c:\Temp\P2V\v8hiddendevices.bat o Select show hidden devices o Check if there are still unused devices present and uninstall them manually Update the HAL if you changed from multi to uniprocessor by executing the script c:\Temp\P2V\v9-HALupdate.bat

 

Reboot your server Install VMware Tools

----------------------------------------------------

A lot has been written about P2V‟ing Windows Domain controllers. The preferred way is to build a new domain controller based on a virtual machine and demote the physical domain controller. Rebuilding a new domain controllers may not always be possible due to time constraints or other dependencies like additional software running on the domain controller which can not be easily migrated. However, P2V‟ing of a domain controller is possible under the right conditions. These conditions are the same for any other transactional service like a Microsoft SQL server or a Microsoft Exchange server and comes to the following point: ‘Make sure all transactional processes are stopped during the P2V process’ For a Windows domain controller the transactional system consists of the Active Directory database. For a Windows 2000 and Windows 2003 domain controller the Active Directory service can‟t be stopped. To solve this problem, boot the system in the Directory Service Restore Mode. Directory Services Restore Mode (DSRM) is a special boot mode for repairing or recovering Active Directory. It is used to log on to the computer when Active Directory has failed or needs to be restored. In DSRM mode Active Directory is not running and no transactions are occurring. In this state a hot clone can be made. After the hot clone is made, cleanup the system and restore the original ip address of the server. The following steps describe the complete P2V process for a Windows 2000/Windows 2003 domain controller: 1. If any other transactional services are running on the domain controller, stop and disable these services 2. Boot the physical domain controller in Directory Services Restore Mode (DSRM) 3. Clone the physical domain controller with VMware Converter 4. After the conversion is completed, disable the nic(s) of the physical domain controller (through the physical console, iLO, DRAC or other consol tool). This prevents the physical server from being online on the network ever again. 5. Shutdown the physical domain controller 6. Start the virtual domain controller in Directory Services Restore Mode (DSRM). This prevents the directory service from starting and gives room to cleanup the system.

7. Remove any unnecessary software, like hardware monitor agents (e.g. HP Insight Agents) and unnecessary hardware drivers. See this post for more information. 8. Remove inactive devices in the device manager. See this post for more information. 9. Configure the original IP address(es) to the virtual domain controller 10. Enable any other transactional services you disabled in step 1 11. Boot the virtual domain controller in normal mode. The virtual domain controller should now be operational. Be warned: from now on the physical domain controller can never be brought back online! This will result in USN rollback and Active Directory replication will not work correctly. See Microsoft Knowledge Base Article KB875495 for more information. Perform the following checks to validate the domain controller: 1. 2. 3. 4. 5.

Check the eventlogs Check for NTDS Replication errors Run dcdiag from the windows support tools Run netdag from the windows support tools Create a test user on another domain controller and validate it‟s replicated to the virtual domain controller 6. Delete the test user on the virtual domain controller and validate the deletion is replicated to another domain controller If you do resize the disk where the Active Directory replica root path is placed (e.g. %SystemRoot%\Sysvol) during the P2V process, you will get the following error: The File Replication Service has detected that the replica root path has changed from "d:\sysvol\domain" to "d:\sysvol\domain". If this is an intentional move then a file with the name NTFRS_CMD_FILE_MOVE_ROOT needs to be created under the new root path. Creating an empty file with the name NTFRS_CMD_FILE_MOVE_ROOT recreate the sysvol folder and solve this problem. --------------------------

Testing port connectivity with Telnet Purpose For troubleshooting purposes, it may be necessary to test connectivity to the different ports on your servers. This article provides you with the steps to use the Telnet utility to test connectivity to different ports on your servers from either a Windows or Linux host.

Resolution Testing port connectivity with Telnet from Windows Note: The Telnet client is not installed by default on Windows 7, Windows Server 2008 R2, Windows Vista, and Windows Server 2008. For more information, see http://technet.microsoft.com/enus/library/cc771275(WS.10).aspx.

The preceding link was correct as of June 2, 2011. If you find the link is broken, provide feedback and a VMware employee will update the link.

To initiate a Telnet test to a port from Windows:

1. Open a command prompt. For more information, see Opening a command or shell prompt (1003892).

2. In the command prompt window type: telnet where is the hostname or IP address of the server, and is the port that you want to connect to.

3. Press Enter. Note: To exit out of the Telnet application type Ctrl + ], then type quit. Depending on the application that uses the port, you may only see a blank screen with a cursor in the corner, this is normal. Two common outputs of a successful connection attempt are:

1. Connecting to port 902 on an ESX/ESXi host: C:\>telnet server 902 Connecting... 220 VMware Authentication Daemon Version 1.10: SSL Required, ServerDaemonProtocol:SOAP, MKSDisplayProtocol:VNC

2. Connecting to port 25 on a mail server:

C:\>telnet server 25 Connecting... 220 server ESMTP Sendmail 8.13.3/8.13.3;

3. Connecting to port 443 on the vCenter Server: C:\>telnet server 443 Connecting... 220 VMware Authentication Daemon Version 1.10: SSL Required, ServerDaemonProtocol:SOAP, MKSDisplayProtocol:VNC If Telnet is unable to connect to the port, the output is similar to:

C:\>telnet server 902 Connecting To server...

Could not open connection to the host, on port 902: Connect failed

C:\>

Testing port connectivity with Telnet from Linux or MacOS To initiate a Telnet test to a port from Linux or MacOS:

1. Open a shell prompt. For more information, see Opening a command or shell prompt (1003892). 2. In the shell prompt window type: telnet where is the hostname or IP address of the server, and is the port that you want to connect to

3. Press Enter. Note: To exit out of the Telnet application type CTRL + ], then type quit. Depending on the application that uses the port, you may only see a blank screen with a cursor in the corner, this is normal. Two common outputs of a successful connection attempt are:

1. Connecting to port 902 on an ESX/ESXi host: [root@server]$ telnet server 902 Trying server... Connected to server. Escape character is '^]'.

220 VMware Authentication Daemon Version 1.10: SSL Required, ServerDaemonProtocol:SOAP, MKSDisplayProtocol:VNC

2. Connecting to port 25 on a mail server: [root@server]$ telnet server 25 Trying server... Connected to server. Escape character is '^]'. 220 server ESMTP Sendmail 8.13.3/8.13.3; If Telnet is unable to connect to the port, the output is similar to:

[root@server]$ telnet server 902 Trying server... telnet: connect to address server: Connection refused

If the connection is refused, a firewall may be blocking that port from your source to the destination server. For more information, see Required ports for configuring an external firewall to allow ESX and vCenter Server traffic (1005189).

Note: Several distributions of Linux do not have a Telnet client installed by default. Refer to the website of your distribution for details on whether one is available and how to install the package.