VM HA and Host HA #12400
-
|
How enable VM HA and Host HA? |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 1 reply
-
|
From what I understand, VM HA is enabled by default as long as you use a compute offering with HA enabled or you can create a compute offering by checking Offer HA toggle. To enable Host HA, however, you need to configure Out-of-Band Management by going to Infrastructure -> Hosts -> Configure Out-of-Band Management That said, it’s important to first understand the difference between VM HA and Host HA before going down the rabbit hole. Within the CloudStack community, it has even been suggested to avoid using Host HA in some scenarios #12139 (comment) According to the CloudStack documentation:
You can verify this in the official docs: https://docs.cloudstack.apache.org/en/latest/adminguide/reliability.html In my understanding VM HA is a framework that monitors virtual machines and starts or restarts them when they are detected as down. This can happen if: The hypervisor host goes down, The VM crashes, The main challenge with any HA system is reliable failure detection. If you cannot accurately determine that a component is truly down, you risk serious issues such as split-brain scenarios. This is where Host HA comes in. To reliably fence a host (the hypervisor, not the VM), CloudStack uses OOBM like IPMI or Redfish. In theory, If CloudStack detects that a host is unreachable, It first tries to recover the host using oobm (for example, issuing a reboot command). If the host still does not recover, CloudStack fences it by issuing a shutdown command through OOBM. Only after fencing does CloudStack restart the VMs on another host. Without a reliable fencing mechanism, the following can happen, lets say you have three KVM hosts in a cluster. HA-enabled VM runs on KVM1. The network to KVM1 goes down, but the host and VM are actually still running. CloudStack thinks KVM1 is down and restarts the VM on KVM2 or KVM3. When the network to KVM1 is restored, the same VM is now running in two places(I acutally have tested this). In theory with Host HA and OOBM, CloudStack ensures that KVM1 is properly powered off before restarting the VM elsewhere, so only one active instance of the VM exists and accesses the shared storage. Since i have tested VM HA more let me give you my observation.
For a host running a HA-enabled VM, CloudStack monitors the VM and logs activity such as c.c.a.m.A.MonitorTask. When the VM is down c.c.a.m.A.MonitorTask reports the VM is beind ping and the HighAvailabilityManagerExtImpl triggers a series of investigators SimpleInvestigator, XenServerInvestigator, KVMInvestigator, HypervInvestigator, VMwareInvestigator, PingInvestigator, ManagementIPSysVMInvestigator. Ovm3Investigator. Eventually, the host goes into Alert state with a message In an environment without NFS primary storage, CloudStack cannot reliably determine whether the host is truly down, so, The host transitions to Alert, VMs are not restarted or migrated to another host
When NFS is available, CloudStack creates a VMHA directory on the NFS primary storage. Each compute host writes a heartbeat file there. If the host running a HA-enabled VM goes down The KVMInvestigator detects the failure via the heartbeat mechanism. The host is marked as Down and After a short time, CloudStack restarts the VM on another host. In this case, VM HA works as expected because the shared NFS storage provides a partially reliable way to detect host failure. This is based on the below threads i read and hands-on experience
|
Beta Was this translation helpful? Give feedback.
-
|
When using Linstor as the primary storage, since Linstor ensures that only one host can write to the storage, enabling VM HA alone is sufficient to meet the high availability requirements of virtual machines. Moreover, I've noticed that this approach allows skipping the host HA check steps, enabling faster recovery of virtual machines. |
Beta Was this translation helpful? Give feedback.
-
Have you tested VM HA in conjunction with Linstor? If yes, could you share your experience. |
Beta Was this translation helpful? Give feedback.
From what I understand, VM HA is enabled by default as long as you use a compute offering with HA enabled or you can create a compute offering by checking Offer HA toggle. To enable Host HA, however, you need to configure Out-of-Band Management by going to Infrastructure -> Hosts -> Configure Out-of-Band Management
That said, it’s important to first understand the difference between VM HA and Host HA before going down the rabbit hole. Within the CloudStack community, it has even been suggested to avoid using Host HA in some scenarios #12139 (comment)
According to the CloudStack documentation: