VMWARE HA summary

Book vSphere 6.0 U2 HA Deepdive
by Duncan Epping


Three major components that form the foundation for HA as of vSphere 6.0:
- FDM - This is the HA agent.
- HOSTD - responsible for many of the tasks we take for granted like powering on virtual machines.
- vCenter
  -- Deploying and configuring HA Agents
  -- Communication of cluster configuration changes
  -- Protection of virtual machines

These resources can be carved up with the use of vSphere __DRS__ Distributed Resource Scheduler (DRS) into separate
pools of resources or used to increase availability by enabling HA.

-There is only one master HA agent in a cluster.
-A master agent is in charge of monitoring the health of virtual machines for which it is responsible and restarting any that fail.
-The slaves are responsible for forwarding information to the master agent and restarting any virtual machines at the direction of the master.
-A master will claim responsibility for a virtual machine by taking “ownership” of the datastore on which the virtual machine’s configuration file is stored.
-The host that is participating in the election with the greatest number of connected datastores will be elected master.

- Network heartbeats are sent by default every second.
- Datastore heartbeating enables a master to more determine the state of a host that is not
reachable via the management network.

- Each hosts in a vSphere HA cluster keeps track of the power state of the virtual machines it is hosting. This set of powered on virtual machines is stored the “poweron” file.
 - Directory .vSphere-HA/<FDM cluster ID>“, file host-<id>-poweron
- Isolation will be validated through the “poweron” file which, as mentioned earlier,
will be updated by the host when it is isolated.

-This file is also used by the slaves to inform the master that it is isolated from the management network: the top line of the file will either contain a “0” (zero) or a “1”.  A “0” means not-isolated and a “1” means isolated.

/vmfs/volumes/57568ce5-75726414-ecd1-001999d8f733/.vSphere-HA/FDM-3408E9EB-A69F-4610-8FC2-4E9A717781D6-18-b0c4ab3-server # ls -la
drwx------    1 root     root           980 Nov 30 15:41 .
drwx------    1 root     root           420 Jun 29 06:12 ..
-rw-r--r--    1 root     root             0 Nov 30 15:41 host-3715-hb
-rw-r--r--    1 root     root           524 Nov 30 15:41 host-3715-poweron
-rw-r--r--    1 root     root             0 Nov 30 15:40 host-9-hb
-rw-r--r--    1 root     root         12180 Nov 30 15:41 host-9-poweron
-rw-r--r--    1 root     root       2229248 Nov 30 12:27 protectedlist
/vmfs/volumes/57568ce5-75726414-ecd1-001999d8f733/.vSphere-HA/FDM-3408E9EB-A69F-4610-8FC2-4E9A717781D6-18-b0c4ab3-server # cat host-3715-poweron
28055712863839
0
5
81 /vmfs/volumes/57568ce5-75726414-ecd1-001999d8f733/Asterisk(FTP)/Asterisk(FTP).vmx
...
99 /vmfs/volumes/57568ce5-75726414-ecd1-001999d8f733/Asterisk13_template_8G/Asterisk13_template_8G.vmx

 - By default, HA selects 2 heartbeat datastores, it will select datastores that are available on all hosts, or as many as possible.
 - This still leaves the question open how the master differentiates between a Failed, Partitioned, or Isolated host.

When the master stops receiving network heartbeats from a slave, it will check for host “liveness” for the next 15 seconds.
Before the host is declared failed, the master will validate if it has actually failed or not by doing additional liveness checks.
1) the master will validate if the host is still heartbeating to the datastore.
2) Second, the master will ping the management IP address of the host.
If both are negative, the host will be declared Failed.

 - HA will trigger an action based on the state of the host. When the host is marked as Failed, a restart of the virtual machines will be initiated. When the host is marked as Isolated, the master might initiate the restarts.

 - Affinity Rules for VM - specify either selected VM be placed on the same host (affinity) or on different hosts (anti-affinity)

 - EVC -  Enhanced vMotion Compatibility - feature to help ensure vMotion compatibility for the hosts in a cluster.
EVC ensures that all hosts in a cluster present the same CPU feature set to virtual machines, even if the actual CPUs on the hosts differ.
Using EVC prevents migrations with vMotion from failing because of incompatible CPUs.

- By "tagging" an VM as an "Agent VM" we ensure that this virtual machine gets powered on first in the case of a host failure. We do this to ensure the service is available before all other VMs start.

This is the current order which HA uses in case of a host failure and restarts need to occur:
1 Agent virtual machines
2 FT secondary virtual machines
3 Virtual Machines configured with a restart priority of high,
4 Virtual Machines configured with a medium restart priority
5 Virtual Machines configured with a low restart priority

 - VM Restart Retries - The default value is 5.
 HA will try to start the virtual machine on one of your hosts in the affected cluster; if this is unsuccessful on that host, the restart count will be increased by 1.

 -When it comes to restarts, one thing that is very important to realize is that HA will not issue more than 32 concurrent power-on tasks on a given host.

 - The Failure of a Slave
 --with datastore heartbeat
T0 – Slave failure.
T3s – Master begins monitoring datastore heartbeats for 15 seconds.
T18s – If heartbeat datastores are configured, the host will be declared dead.

 --without datastore heartbeat
T0 – Slave failure.
T10s – The host is declared unreachable and the master will ping the management  network of the failed host. This is a continuous ping for 5 seconds.
T15s – If no heartbeat datastores are configured, the host will be declared dead.


 - The Failure of a Master
 T0 – Master failure.
T10s – Master election process initiated.
T25s – New master elected and reads the protectedlist.
T35s – New master initiates restarts for all virtual machines on the protectedlist which
are not running.


 - Isolation Response and Isolation Detection
 -- Disabled (default) – When isolation occurs on the host, the state of the virtual machines remains unchanged.
 -- Shutdown – When isolation occurs, all virtual machines running on the host will be shut down using a guest-initiated shutdown through VMware Tools. If this is not successful within 5 minutes, a “power off” will be executed. This time out value can be adjusted by setting the advanced option das.isolationShutdownTimeout. If VMware Tools is not installed, a “power off” will be initiated immediately.
  -- Power off – When isolation occurs, all virtual machines are powered off. It is a hard stop, or to put it bluntly, the “virtual” power cable of the virtual machine will be pulled out!


In previous versions (prevSphere 6.0) there was also an isolation response called "leave powered on", this has been
renamed to "disabled" as "leave powered on" means that there is no response to an isolation event.

 In a converged network environment with iSCSI storage, for instance, it would be impossible to validate host isolation during a full isolation as the validation would fail due to the inaccessible datastore from the perspective of the isolated host.

 -  das.isolationaddress
 Right click on Cluster Icon -> Edit Settings -> vSphere-HA -> Advanced Options:
 das.isolationaddress 10.x.x.93

 - Admission Control guarantees that capacity is available for an HA initiated failover by reserving resources within a cluster.

 - VM and Application Monitoring
An example of a single virtual machine failure would, for instance, be the infamous “blue screen of death”. In the case of App Monitoring the type of failure that triggers a response is defined by the application developer or administrator.

When an error occurs, HA will trigger a restart of the virtual machine when all three conditions are met:
1.  No VMware Tools heartbeat received
2.  No network I/O over the last 120 seconds
3.  No storage I/O over the last 120 seconds