Happy 2015!! I’m seriously backed up and delinquent on blog posts so prepare for a deluge of interesting topics soon.

As painful and hopefully unlikely as this scenario may be, better to have a planned and controlled outage vs unplanned with potential data loss, or worse. This post covers the cluster shutdown process for Nutanix with Hyper-V. In case you didn’t know, Nutanix is not dependent on Windows or vSphere for clustering, as it handles its own HA mechanism. This does require, however, that you consider the independent Nutanix cluster services to ensure a graceful shutdown.

In the references at the bottom, you’ll notice that for vSphere clusters, the method described there is to shut down the Nutanix cluster first, then the vSphere cluster. For Hyper-V the procedure is reversed, requiring that the Hyper-V cluster be shut down first, then the Nutanix cluster. If you shut down your Nutanix cluster services first, you won’t be able to connect to your Failover Cluster to properly shut it down.

Shut Down Failover Cluster

1. Gracefully shut down all VMs (Not Nutanix CVMs, yet!!)
2. Open Failover Cluster Manager (cluadmin.msc) and connect to the Failover Cluster
3. Right-click the cluster name, select “more actions” –> “shut down cluster”, select “yes” to confirm
• Alternatively, in PowerShell enter: “Stop-Cluster”

Once the Hyper-V cluster is shut down, you can begin to shut down the Nutanix cluster.

Shut Down Nutanix Cluster

1. SSH or connect to any CVM console, log in
2. Enter “cluster stop” (root not required)
3. Once completed successfully, you will see confirmation that all services have stopped on all CVMs:
4. Shut down each CVM on all hosts:
5. Lastly, shut down each physical Hyper-V node - This can be done of number of ways, onesy twosy, PowerShell, or via the Remote Shutdown UI (shutdown /i) for bulk operation:

Startup is the reverse, ensuring that your Nutanix cluster is up and running properly before starting your failover cluster.

References:

http://blogs.msdn.com/b/clustering/archive/2013/08/23/10443912.aspx

http://port5480.blogspot.com/2014/10/shut-down-nutanix-cluster.html

Disaster recovery (DR) is always a hot topic that many companies do not do at all for one reason or another or do badly. Providing DR for a virtual environment can be a particularly challenging and expensive endeavor. In my enterprise I am running ESX4 U1 on top of Brocade fiber-channel connected NetApp FAS arrays (1 production, 1 DR) with a 1Gb Metro-Ethernet link between sites. I do not currently have the luxury of using VMware’s Site Recovery Manager (SRM) in my environment so my process will be completely manual. SRM removes many of the monotonous tasks of turning up a virtual DR environment including testing your plan, but this comes with a heavy price tag. I am fortunate enough, however, to have the full suite of NetApp tools at my disposal.

Snap Manager for Virtual Infrastructure (SMVI) is NetApp’s answer to vSphere VM backups for use with their storage arrays. SMVI does require a license and requires some specific array-level licenses, such as: SnapRestore, applicable protocol license (NFS, FCP, iSCSI), SnapMirror (optional), and FlexClone. There are some specific instances in which FlexClone is not required such as for NFS VM in-place VMDK restores. All by itself SMVI can be used instead of VCB or Ranger type products to backup/restore VMs, volumes, or individual files within the guest VM OS. SnapMirror can be used in conjunction with SMVI to provide DR by sending backed up VM volumes offsite to another Filer.

Here is the backup process:

1. Once a backup is initiated, a VMware snapshot is created via vCenter for each powered-on VM in a selected volume, or for each VM that is selected for backup. You can choose either method but volume backups are recommended.
2. The VM snapshots preserve state and are used during restores. Windows application consistency can be achieved by using VMware’s VSS support for snapshots.
3. Once all VMs are snapped, SMVI initiates a snapshot of the volume on the Filer.
4. Once the volume snaps are complete, the VM snapshots are deleted in vCenter.
5. If you have a SnapMirror in place for your backed up volumes, it is then updated by SMVI.

NetApp fully supports, and recommends, running SMVI on the vCenter server for simplicity. Setup is very straight forward and only requires the vCenter server/ credentials and the storage array names or IPs/ credentials. Best practice is to set up a dedicated user in vCenter as well as on the arrays for SMVI. The required vCenter permissions for this service account are detailed in the best practices guide in the references section at the bottom of this post.

SMVI Setup

Once SMVI can communicate with vCenter, you will see the Vi datacenters, datastores and VMs on the inventory tab.

Backup configuration is simple. Name the job, choose the volume, specify the desired backup schedule, how long to keep the backups, and where to send alerts. If you’ll be using SnapMirror, make sure to check the “Initiate SnapMirror update” option.

By default the SMVI job alerts include a link to the job log but the listed address may be unreachable. In my case the links sent were to an IPv6 address even though IPv6 was disabled on the server. This can be changed by editing the smvi.override file in \Program Files (x86)\NetApp\SMVI\server\etc and adding the following lines:

smvi.hostname=vcenter.domain.com
smvi.address=10.10.10.10

Once you successfully run a backup job in SMVI you will be able to see the available snapshots for the volume on the source Filer. Note the SMVI snaps vs snaps used by the SnapMirror.

I my scenario, I am backing up the 2 LUNS called VMProd1_FC and VMProd2_FC which exist in volumes called ESXLUN1 and 2, respectively. Both of these volumes have corresponding SnapMirrors configured between the primary and DR Filers.

A couple of things to keep in mind about timing:

• It is a good idea to sync your SnapMirrors outside of SMVI which will reduce the time it takes when SMVI updates the mirrors. Just make sure to do this at a time other than when your SMVI jobs are running!
• If you are using deduplication on these volumes (you should be), schedule it to run at a time when you are not running SMVI backups or syncing SnapMirrors.

Once SMVI successfully updates the SnapMirror, you will see the replicated snapshots on the destination side of the mirror as well. DR for your ESX environment is now in effect!

Testing the DR Plan

Here comes the fun part and where having SRM would be extremely helpful to automate most of this. Thanks to NetApp’s FlexClone technology we can test our DR plan without breaking the mirrors, so you could test whenever and as often as necessary without affecting production.

First step, create a FlexClone Volume of the replicated snapshot you want to be able to mount and test with. Choose the appropriate volume then select Snapshop—>Clone from the Volumes menu. Important to note is that you must use a File space guarantee or the FlexClone creation will fail! This can be done via System Manager, CLI, or FilerView:

Your new volume will now be visible in the Filer’s volumes list, which also creates a corresponding LUN that you’ll notice is offline by default:

Bring the new LUN online and then you will be able to present it to your DR ESX hosts:

The LUN should instantly appear on the ESX hosts in your iGroup but if it does not run a rescan on the appropriate storage adapter:

ESX sees the LUN, now it needs to be added to the cluster. Switch to the Storage page in vCenter and select “add storage.” Select the new disk and be sure to select “Assign a new signature” to the disk or ESX will not be able to write to it! Only a new UUID is written, all existing data will be preserved.

The new LUN will now be accessible to all ESX hosts in the cluster. Note the default naming convention used after adding the LUN:

This is where things get REALLY manual and you wish you ponied that $25k for SRM. Browse the newly mounted datastore and register each VM you need to test, one by one.

Now your organization’s policies take over as to how far you need to go to satisfy a successful DR test. Once the VM is registered it can be powered on, but if it’s counterpart is still running in production disable its vNIC first. If you need to go further, then shut down the production VM, re-IP the DR clone and then bring it online. If you need to have users connect to the DR clone then there are other implications to consider such as DNS, DFS, ODBC pointers, etc. When your test is complete, power off all DR VM clones, dismount the datastore from within ESX, delete the FlexClone Volume on the DR Filer, bring the production VMs back up, check DNS, done. A beautiful thing and best of all the prod—>DR mirror is still in sync!

References:

SMVI 2.0 Best Practices

SMVI 2.0 Admin Guide

NetApp KB52524