Happy 2015!! I’m seriously backed up and delinquent on blog posts so prepare for a deluge of interesting topics soon.

As painful and hopefully unlikely as this scenario may be, better to have a planned and controlled outage vs unplanned with potential data loss, or worse. This post covers the cluster shutdown process for Nutanix with Hyper-V. In case you didn’t know, Nutanix is not dependent on Windows or vSphere for clustering, as it handles its own HA mechanism. This does require, however, that you consider the independent Nutanix cluster services to ensure a graceful shutdown.

In the references at the bottom, you’ll notice that for vSphere clusters, the method described there is to shut down the Nutanix cluster first, then the vSphere cluster. For Hyper-V the procedure is reversed, requiring that the Hyper-V cluster be shut down first, then the Nutanix cluster. If you shut down your Nutanix cluster services first, you won’t be able to connect to your Failover Cluster to properly shut it down.

Shut Down Failover Cluster

1. Gracefully shut down all VMs (Not Nutanix CVMs, yet!!)
2. Open Failover Cluster Manager (cluadmin.msc) and connect to the Failover Cluster
3. Right-click the cluster name, select “more actions” –> “shut down cluster”, select “yes” to confirm
• Alternatively, in PowerShell enter: “Stop-Cluster”

Once the Hyper-V cluster is shut down, you can begin to shut down the Nutanix cluster.

Shut Down Nutanix Cluster

1. SSH or connect to any CVM console, log in
2. Enter “cluster stop” (root not required)
3. Once completed successfully, you will see confirmation that all services have stopped on all CVMs:
4. Shut down each CVM on all hosts:
5. Lastly, shut down each physical Hyper-V node - This can be done of number of ways, onesy twosy, PowerShell, or via the Remote Shutdown UI (shutdown /i) for bulk operation:

Startup is the reverse, ensuring that your Nutanix cluster is up and running properly before starting your failover cluster.

References:

http://blogs.msdn.com/b/clustering/archive/2013/08/23/10443912.aspx

http://port5480.blogspot.com/2014/10/shut-down-nutanix-cluster.html

Yay, last post of 2014! Haven’t invested in the hyperconverged Software Defined Storage model yet? No problem, there’s still time. In the meanwhile, here is how to cluster Server 2012 R2 using tried and true EqualLogic iSCSI shared storage.

EQL Group Manager

First, prepare your storage array(s), by logging into EQL Group Manager. This post assumes that your basic array IP, access and security settings are in place.  Set up your local CHAP account to be used later. Your organization’s security access policies or requirements might dictate a different standard here.

Create and assign an Access Policy to the VDS/VSS in Group Manager otherwise this volume will not be accessible. This will make subsequent steps easier when it’s time to configure ASM.

Create some volumes in Group Manager now so you can connect your initiators easily in the next step. It’s a good idea to create your cluster quorum LUN now as well.

Host Network Configuration

First configure the interfaces you intend to use for iSCSI on your cluster nodes. Best practice says that you should limit your iSCSI traffic to a private Layer2 segment, not routed and only connecting to the devices that will participate in the fabric. This is no different from Fiber Channel in that regard, unless you are using a converged methodology and sharing your higher bandwidth NICs. If using Broadcom NICs you can choose Jumbo Frames or hardware offload, the larger frames will likely net a greater performance impact. Each host NIC used to access your storage targets should have a unique IP address able to access the network of those targets within the same private Layer2 segment. While these NICs can technically be teamed using the native Windows LBFO mechanism, best practice says that you shouldn’t, especially if you plan to use MPIO to load balance traffic. If your NICs will be shared (not dedicated to iSCSI alone) then LBFO teaming is supported in that configuration. To keep things clean and simple I’ll be using 4 NICs, 2 dedicated to LAN, 2 dedicated to iSCSI SAN. Both LAN and SAN connections are physically separated to their own switching fabrics as well, this is also a best practice.

MPIO - the manual method

First, start the MS iSCSI service, which you will be prompted to do, and check its status in PowerShell using get-service –name msiscsi.

Next, install MPIO using Install-WindowsFeature Multipath-IO

Once installed and your server has been rebooted, you can set additional options in PowerShell or via the MPIO dialog under  File and Storage Services—> Tools.

Open the MPIO settings and tick “add support for iSCSI devices” under Discover Multi-Paths. Reboot again. Any change you make here will ask you to reboot. Make all changes once so you only have to do this one time.

The easier way to do this from the onset is using the EqualLogic Host Integration Tools (HIT Kit) on your hosts. If you don’t want to use HIT for some reason, you can skip from here down to the “Connect to iSCSI Storage” section.

Install EQL HIT Kit (The Easier Method)

The EqualLogic HIT Kit will make it much easier to connect to your storage array as well as configure the MPIO DSM for the EQL arrays. Better integration, easier to optimize performance, better analytics. If there is a HIT Kit available for your chosen OS, you should absolutely install and use it. Fortunately there is indeed a HIT Kit available for Server 2012 R2.

Configure MPIO and PS group access via the links in the resulting dialog.

In ASM (launched via the “configure…” links above), add the PS group and configure its access. Connect to the VSS volume using the CHAP account and password specified previously. If the VDS/VSS volume is not accessible on your EQL array, this step will fail!

Connect to iSCSI targets

Once your server is back up from the last reboot, launch the iSCSI Initiator tool and you should see any discovered targets, assuming they are configured and online. If you used the HIT Kit you will already be connected to the VSS control volume and will see the Dell EQL MPIO tab.

Choose an inactive target in the discovered targets list and click connect, be sure to enable multi-path in the pop-up that follows, then click Advanced.

Enable CHAP log on, specify the user/pw set up previously:

If your configuration is good the status of your target will change to Connected immediately. Once your targets are connected, the raw disks will be visible in Disk Manager and can be brought online by Windows.

When you create new volumes on these disks, save yourself some pain down the road and give them the same label as what you assigned in Group Manager! The following information can be pulled out of the ASM tool for each volume:

Failover Clustering

With all the storage pre-requisites in place you can now build your cluster. Setting up a Failover Cluster has never been easier, assuming all your ducks are in a row. Create your new cluster using the Failover Cluster Manager tool and let it run all compatibility checks.

Make sure your patches and software levels are identical between cluster nodes or you’ll likely fail the clustering pre-check with differing DSM versions:

Once the cluster is built, you can manipulate your cluster disks and bring any online as required. Cluster disks will not be able to be brought online until all nodes in the cluster can access the disk.

Next add your cluster disks to Cluster Shared Volumes to enable multi-host read/write and HA.

The new status will be reflected once this change is made.

Configure your Quorum to use the disk witness volume you created earlier. This disk does not need to be a CSV.

Check your cluster networks and make sure that iSCSI is set to not allow cluster network communication. Make sure that your cluster network is setup to allow cluster network communication as well as allowing client connections. This can of course be further segregated if desired using additional NICs to separate cluster and client communication.

Now your cluster is complete and you can begin adding HA VMs, if using Hyper-V, SQL, File or other roles as required.

References:

http://blogs.technet.com/b/keithmayer/archive/2013/03/12/speaking-iscsi-with-windows-server-2012-and-hyper-v.aspx

http://blogs.technet.com/b/askpfeplat/archive/2013/03/18/is-nic-teaming-in-windows-server-2012-supported-for-iscsi-or-not-supported-for-iscsi-that-is-the-question.aspx

A few minor things have changed since the RTM build of 2008 R2 dropped with regard to this feature. Here is a quick recap and overview.

First your domain must be running in Server 2008 R2 forest and domain functional level, which means all of your DCs must be running 2008 R2.

Once this is achieved you will be allowed to enable the recycle bin AD feature (which must be done from the DC holding the Domain Naming master role).

To verify that the feature was successfully enabled fire up ldp.exe in admin mode. In the 2008 R2 RC the deleted objects container was now visible but in RTM it isn’t. Hit View—>Tree—> and select the Configuration partition for the BaseDN. Expand down to the Partitions container and verify the recycle bin has been enabled under msDS-EnabledFeature.

Ok good, the feature is enabled but we still can’t see the deleted objects container nor any objects that we delete at this point, even though the container is visible in the well known objects list. This is because the container itself is marked as deleted. Deleted object restore or view operation attempts will fail at this point as well.

To change this, with the domain root selected as BaseDN, click Browse—>Search, enter CN=Deleted Objects,DC=domain,DC=com for the Base DN, change the filter from * to user, then click options. Change the search call type to extended then click the controls button. In the “Load Predefined” drop down select “return deleted objects” and double-click the 1.2.xx item in the active controls pane. Click Check in, then Ok. The TechNet link at the bottom goes into greater detail about this.

Refresh the tree and you should now see the deleted objects container. You should now be able to view and restore your deleted objects via ldp or PowerShell.

Reference:

TechNet

I have operated in production environments with AD accounts in full-time admin mode as well as non-privileged mode (2 accounts). In a domain setting UAC can make the later tolerable by prompting you for credentials when attempting to run an application requiring greater privileges. In the XP days this was accomplished by using “Run As”. The temptation is to just turn off UAC and run as a full-time admin but I’m trying to be good and resist this temptation, especially with Windows 7 and the new security levels UAC provides.

You can find the UAC settings able to be modified via group or local policy in Computer Configuration\Security Settings\Local Policies\Security Options:

By default, Microsoft PowerShell runs in non-privileged mode, with UAC enabled, as does the command prompt. This means that if you want to run elevated commands or applications then you would need to open another session in administrative mode (right-click—>run as administrator). In UNIX this issue is addressed with the sudo command that provides temporary elevation to “super user” status to execute an otherwise restricted process. One could then define elevated accounts via the sudoers group where you can define which accounts can be used for sudo, the goal here is to not needlessly give out the root password.

We can emulate this behavior in Windows with PowerShell with the help of UAC. If you’re only looking to executed elevated processes in PowerShell without entering alternate credentials then you can ignore the UAC piece. The first step is to create a profile in PowerShell, if you haven’t already. Enter $profile in PowerShell to show the path to the profile. The MSDN link at the bottom of this entry will give more information on the various locations profiles can exist. Test-path $profile will show you if that profile exists and if it’s operational.  If the test comes back false then you will need to create the profile first by executing:

To follow up on an earlier posting about how to enable the new recycle bin feature in AD, this post will outline how to perform an object restore. Once the recycle bin has been successfully enabled you can use the ldp.exe tool or PowerShell to view the deleted objects container (hidden in ADUC) as well as recover anything contained within.

Via LDP

Get connected and expose the domain tree: Type ldp into the start search bar, click Connection—>Connect—>enter applicable DC information—>once connected click Connection—>Bind. Next click View—>Tree and enter the AD root as BaseDN you want to work within: dc=domain,dc=com. Drill down into your AD tree and look for CN=Deleted Objects… This is where all your objects go once deleted in AD. Double-click this line and you will see every object currently in a deleted state. Double click on a child object to reveal more details. The CN’s of each should be recognizable as their friendly names and in the details pane the Name value will be visible as well, in my case I deleted a user account named Temp.

To restore, right click the object you want to recover and click Modify. In the “Edit Entry Attribute” box type isDeleted, leave “Values” empty, click the “Delete” radial under Operation, then click the Enter button. You will see the new entry in the Entry List box. Next, go back to the “Edit Entry Attribute” box and type distinguishedName, in the values box type the original DN of the object before it was deleted. The DN of the object in it’s deleted state will not work here. (Look for the “lastKnownParent” line of the deleted object in the details pane and add the object’s CN to it). Click the “Replace” radial under operation then click Enter. You should now have 2 entries in the Entry List box. Tick the “Extended” box at the bottom left and click run. The output window should show that the object was modified with no errors. Go back to ADUC and verify that your object is back from the dead.

Via PowerShell

The PowerShell method is a one line command that pipes a get object operation to a restore object operation. This method is much faster but less gratifying for some reason. The filter criteria can be modified but going by display name should be easy enough. You will get no confirmation of success or failure in the PowerShell display.

Get-ADObject -Filter 'displayName -eq "temp"' -IncludeDeletedObjects | Restore-ADObject

Resource:

Technet