I have a HP DL 385 G7 server running ESXi 5.1 via a bootable USB drive. The datastore is all local storage (OBR10 - 16 300 GB 10K SAS drives) and contains 18 VMs currently. There are 8 NICs in the server (4 on mobo and 4 on PCI card), 132 GB RAM, and 8-core Opteron processors.
We needed to shut down the host today to upgrade its RAM. Before upgrading the RAM, I shut down the host from within vSphere. We sat and watched as all VMs were shut down since they were all setup to shutdown with the ESXi host per the vSphere config. The server powered off with no issues. We put the RAM in per HP's instructions and attached a USB hub to the server for multiple dongle connections to some of our VMs that are used for Engineering. We connected all network cables as well as the KVM and then powered up the server.
ESXi booted fine based on the console view (showing the proper amount of RAM post-upgrade), but the VMs did not. All VMs had been configured to start with the host (some in a particular order, some in any order with 15 second delays between startups), but once the host had booted successfully, I checked in vSphere and saw only a few VMs had booted. No more than 4 VMs were listed in inventory, but we have 18! There was a VM named Unknown that appeared to be stuck trying to boot, which I ended up removing from Inventory before looking at the settings to see which VMX it was attached to (oops). All of the startup / shutdown configuration except for those few VMs that showed up in inventory was gone as well. My Elastix server was set to boot first, but it did not boot at all (nor did it show in inventory). However, the VMs set to boot 2nd and 3rd booted with no issues and were running fine. Besides those two VMs that were running and the one named Unknown, there was one other VM shown in inventory that was powered off. It was not one of the VMs set to boot in a specific order but rather one in the "startup in any order" pool. That struck me as very odd too since I had about 7 VMs setup to boot in a specific order when the host boots.
I had to open the datastore browser and add the VMX file for each VM to Inventory again. After that, I could boot each VM with no huge issues.
The Elastix server for some reason booted with a DHCP address rather than the static ip I had given it. I was able to login to the web interface and change it to what it should be (and all phone calls and office phones worked fine after the ip change), but that was really odd. I wonder if this has anything to do with the fact that I did not shut down the Elastix server from a SSH console or that it is running VMWare Tools but shows to be the 3rd-party/Independent version of VMWare Tools (had issues installing through vSphere and had to install manually when I setup the VM using http:/
I will also say that there was a custom firewall rule on the ESXi host that I had setup to allow vnc connections to a particular VM we use for press brake programming (which I had setup to be persistent after a host reboot) that I had to recreate post-reboot (/etc/profile.local had nothing in it post reboot). It may have been something I did wrong, but I wanted to mention it.
The USB drive that boots ESXi 5.1 in this server was once upgraded from 5.0 and caused some strange issued with VMWare Tools missing (http:/
So is there any good explanation for what happened? Would it be better if I just installed ESXi 5.1 fresh on a different USB drive and started from scratch (and add all VMs in the datastore to inventory, re-create vSwitches, etc.) to prevent these kind of issues moving forward?
