Yay, last post of 2014! Haven’t invested in the hyperconverged Software Defined Storage model yet? No problem, there’s still time. In the meanwhile, here is how to cluster Server 2012 R2 using tried and true EqualLogic iSCSI shared storage.

EQL Group Manager

First, prepare your storage array(s), by logging into EQL Group Manager. This post assumes that your basic array IP, access and security settings are in place.  Set up your local CHAP account to be used later. Your organization’s security access policies or requirements might dictate a different standard here.

Create and assign an Access Policy to the VDS/VSS in Group Manager otherwise this volume will not be accessible. This will make subsequent steps easier when it’s time to configure ASM.

Create some volumes in Group Manager now so you can connect your initiators easily in the next step. It’s a good idea to create your cluster quorum LUN now as well.

Host Network Configuration

First configure the interfaces you intend to use for iSCSI on your cluster nodes. Best practice says that you should limit your iSCSI traffic to a private Layer2 segment, not routed and only connecting to the devices that will participate in the fabric. This is no different from Fiber Channel in that regard, unless you are using a converged methodology and sharing your higher bandwidth NICs. If using Broadcom NICs you can choose Jumbo Frames or hardware offload, the larger frames will likely net a greater performance impact. Each host NIC used to access your storage targets should have a unique IP address able to access the network of those targets within the same private Layer2 segment. While these NICs can technically be teamed using the native Windows LBFO mechanism, best practice says that you shouldn’t, especially if you plan to use MPIO to load balance traffic. If your NICs will be shared (not dedicated to iSCSI alone) then LBFO teaming is supported in that configuration. To keep things clean and simple I’ll be using 4 NICs, 2 dedicated to LAN, 2 dedicated to iSCSI SAN. Both LAN and SAN connections are physically separated to their own switching fabrics as well, this is also a best practice.

MPIO - the manual method

First, start the MS iSCSI service, which you will be prompted to do, and check its status in PowerShell using get-service –name msiscsi.

Next, install MPIO using Install-WindowsFeature Multipath-IO

Once installed and your server has been rebooted, you can set additional options in PowerShell or via the MPIO dialog under  File and Storage Services—> Tools.

Open the MPIO settings and tick “add support for iSCSI devices” under Discover Multi-Paths. Reboot again. Any change you make here will ask you to reboot. Make all changes once so you only have to do this one time.

The easier way to do this from the onset is using the EqualLogic Host Integration Tools (HIT Kit) on your hosts. If you don’t want to use HIT for some reason, you can skip from here down to the “Connect to iSCSI Storage” section.

Install EQL HIT Kit (The Easier Method)

The EqualLogic HIT Kit will make it much easier to connect to your storage array as well as configure the MPIO DSM for the EQL arrays. Better integration, easier to optimize performance, better analytics. If there is a HIT Kit available for your chosen OS, you should absolutely install and use it. Fortunately there is indeed a HIT Kit available for Server 2012 R2.

Configure MPIO and PS group access via the links in the resulting dialog.

In ASM (launched via the “configure…” links above), add the PS group and configure its access. Connect to the VSS volume using the CHAP account and password specified previously. If the VDS/VSS volume is not accessible on your EQL array, this step will fail!

Connect to iSCSI targets

Once your server is back up from the last reboot, launch the iSCSI Initiator tool and you should see any discovered targets, assuming they are configured and online. If you used the HIT Kit you will already be connected to the VSS control volume and will see the Dell EQL MPIO tab.

Choose an inactive target in the discovered targets list and click connect, be sure to enable multi-path in the pop-up that follows, then click Advanced.

Enable CHAP log on, specify the user/pw set up previously:

If your configuration is good the status of your target will change to Connected immediately. Once your targets are connected, the raw disks will be visible in Disk Manager and can be brought online by Windows.

When you create new volumes on these disks, save yourself some pain down the road and give them the same label as what you assigned in Group Manager! The following information can be pulled out of the ASM tool for each volume:

Failover Clustering

With all the storage pre-requisites in place you can now build your cluster. Setting up a Failover Cluster has never been easier, assuming all your ducks are in a row. Create your new cluster using the Failover Cluster Manager tool and let it run all compatibility checks.

Make sure your patches and software levels are identical between cluster nodes or you’ll likely fail the clustering pre-check with differing DSM versions:

Once the cluster is built, you can manipulate your cluster disks and bring any online as required. Cluster disks will not be able to be brought online until all nodes in the cluster can access the disk.

Next add your cluster disks to Cluster Shared Volumes to enable multi-host read/write and HA.

The new status will be reflected once this change is made.

Configure your Quorum to use the disk witness volume you created earlier. This disk does not need to be a CSV.

Check your cluster networks and make sure that iSCSI is set to not allow cluster network communication. Make sure that your cluster network is setup to allow cluster network communication as well as allowing client connections. This can of course be further segregated if desired using additional NICs to separate cluster and client communication.

Now your cluster is complete and you can begin adding HA VMs, if using Hyper-V, SQL, File or other roles as required.

References:

http://blogs.technet.com/b/keithmayer/archive/2013/03/12/speaking-iscsi-with-windows-server-2012-and-hyper-v.aspx

http://blogs.technet.com/b/askpfeplat/archive/2013/03/18/is-nic-teaming-in-windows-server-2012-supported-for-iscsi-or-not-supported-for-iscsi-that-is-the-question.aspx

Hyperconverged Web-scale Software Defined Storage (SDS) solutions are white hot right now and Nutanix is leading the pack with their ability to support all major hypervisors (vSphere, Hyper-V and KVM) while providing nearly unlimited scale. Dell partnering with Nutanix was an obvious mutually beneficial choice for the reasons above plus supplying a much more robust server platform. Dell also provides a global reach for services and support as well as solving other challenges such as hypervisors installed in the factory.

Nutanix operates below the hypervisor layer so as a result requires a lot of tightly coupled interaction with the hardware directly. Many competing platforms in this space sit above the hypervisor so require vSphere, for example, to provide access to storage and HA but they are also limited by the hypervisor’s limitations (scale).  Nutanix uses its own algorithm for clustering and doesn’t rely on a common transactional database which can cause additional challenges when building solutions that span multiple sites. Because of this the Nutanix Distributed Filesystem (NDFS) supports no known limits of scale. There are current Nutanix installations in the thousands of nodes across a contiguous namespace and now you can build them on Dell hardware.

Along with the Dell XC720xd appliances, we have released a number of complementary workload Product Architectures to help customers and partners build solutions using these new platforms. I’ll discuss the primary architectural elements below.

Wyse Datacenter Appliance Architecture for Citrix

Wyse Datacenter Appliance Architecture for VMware

Wyse Datacenter Appliance Architecture for vWorkspace

Nutanix Architecture

Three nodes minimum are required for NDFS to achieve quorum so that is the minimum solution buy in, then storage and compute capacity can be increased incrementally beyond by adding one or more nodes to an existing cluster. The Nutanix architecture uses a Controller VM (CVM) on each host which participates in the NDFS cluster and manages the hard disks local to its own host. Each host requires two tiers of storage: high performance/ SSD and capacity/ HDD. The CVM manages the reads/writes on each host and automatically tiers the IO across these disks. A key value proposition of the Nutanix model is data locality which means that the data for a given VM running on a given host is stored locally on that host as apposed to having reads and writes crossing the network. This model scales indefinitely in a linear block manner where you simply buy and add capacity as you need it. Nutanix creates a storage pool that is distributed across all hosts in the cluster and presents this pool back to the hypervisor as NFS or SMB.

You can see from the figure below that the CVM engages directly with the SCSI controller through which it accesses the disks local to the host it resides. Since Nutanix sits below the hypervisor and handles its own clustering and data HA, it is not dependent upon the hypervisor to provide any features nor is it limited by any related limitations.

From a storage management and feature perspective, Nutanix provides two tiers of optional deduplication performed locally on each host (SSD and HDD individually), compression, tunable replication (number of copies of each write spread across disparate nodes in the cluster) and data locality (keeps data local to the node the VM lives on). Within a storage pool, containers are created to logically group VMs stored within the namespace and enable specific storage features such as replication factor and dedupe. Best practice says that a single storage pool spread across all disks is sufficient but multiple containers can be used. The image below shows an example large scale XC-based cluster with a single storage pool and multiple containers.

While the Nutanix architecture can theoretically scale indefinitely, practicality might dictate that you design your clusters around the boundaries of the hypervisors, 32 nodes for vSphere, 64 nodes for Hyper-v. The decision to do this will be more financially impactful if you separate your resources along the lines of compute and management in distinct SDS clusters. You could also, optionally, install many maximum node hypervisor clusters within a single very large, contiguous Nutanix namespace, which is fully supported. I’ll discuss the former option below as part of our recommended pod architecture.

Dell XC720xd platforms

For our phase 1 launch we have five platforms to offer that vary in CPU, RAM and size/ quantity of disks. Each appliance is 2U, based on the 3.5” 12-gen PowerEdge R720XD and supports from 5 to 12 total disks, each a mix of SSD and HDD. The A5 platform is the smallest with a pair of 6-core CPUs, 200GB SSDs and a recommended 256GB RAM. The B5 and B7 models are almost identical except for the 8-core CPU on the B5 and the 10-core CPU on the B7. The C5 and C7 boast a slightly higher clocked 10-core CPU with doubled SSD densities and 4-5x more in the capacity tier. The suggested workloads are specific with the first three targeted at VDI customers. If greater capacity is required, the C5 and C7 models work very well for this purpose too.

For workload to platform sizing guidance, we make the following recommendations:

Here is a view of the 12G-based platform representing the A5-B7 models. The C5 and C7 would add additional disks in the second disk bay. The two disks in the rear flexbay are 160GB SSDs configured in RAID1 via PERC used to host the hypervisor and CVM, these disks do not participate in the storage pool. The six disks in front are controlled by the CVM directly via the LSI controller and contribute to the distributed storage pool across all nodes.

Dell XC Pod Architecture

This being a 10Gb hyperconverged architecture, the leaf/ spine network model is recommended. We do recommend a 1Gb switch stack for iDRAC/ IPMI traffic and build the leaf layer from 10Gb Force10 parts. The S4810 is shown in the graphic below which is recommended for SFP+ based platforms or the S4820T can be used for 10GBase-T.

In our XC series product architecture, the compute, management and storage layers, typically all separated, are combined here into a single appliance. For solutions based on vWorkspace under 10 nodes, we recommend a “floating management” architecture which allows the server infrastructure VMs to move between hosts also being used for desktop VMs or RDSH sessions. You’ll notice in the graphics below that compute and management are combined into a single hypervisor cluster which hosts both of these functions.

Hyper-V is shown below which means the CVMs present the SMBv3 protocol to the storage pool. We recommend three basic containers to separate infrastructure mgmt, desktop VMs and RDSH VMs. We recommend the following feature attributes based on these three containers (It is not supported to enable compression and deduplication on the same container):

You’ll notice that I’ve included the resource requirements for the Nutanix CVMs (8 x vCPUs, 32GB vRAM). The vRAM allocation can vary depending on the features you enable within your SDS cluster. 32GB is required, for example, if you intend to enable both SSD and HDD deduplication. If you only require SSD deduplication and leave the HDD tier turned off, you can reduce your CVM vRAM allocation to 16GB. We highly recommend that you disable any features that you do not need or do not intend to use!

For vWorkspace solutions over 1000 users or solutions based on VMware Horizon or Citrix XenDesktop, we recommend separating the infrastructure management in all cases. This allows management infrastructure to run in its own dedicated hypervisor cluster while providing very clear and predictable compute capacity for the compute cluster. The graphic below depicts a 1000-6000 user architecture based on vWorkspace on Hyper-V. Notice that the SMB namespace is stretched across both of the discrete compute and management infrastructure clusters, each scaling independently. You could optionally build dedicated SDS clusters for compute and management if you desire, but remember the three node minimum, which would raise your minimum build to 6 nodes in this scenario.

XenDesktop on vSphere, up to 32 nodes max per cluster, supporting around 2500 users in this architecture:

Horizon View on vSphere, up to 32 nodes max per cluster. supporting around 1700 users in this architecture:

Network Architecture

Following the leaf/ spine model, each node should connect 2 x 10Gb ports to a leaf switch which are then fully mesh connected to an upstream set of spine switches.

On each host there are two virtual switches: one for external access to the LAN and internode communication and one private internal vSwitch used for the CVM alone. On Hyper-V the two NICs are configured in a LBFO team on each host with all management OS vNICs connected to it.

vSphere follows the same basic model except for port groups configured for the VM type and VMKernel ports configured for host functions:

Performance results

The tables below summarize the user densities observed for each platform during our testing. Please refer to the product architectures linked at the beginning of this post for the detailed performance results for each solution.

Resources:

http://en.community.dell.com/dell-blogs/direct2dell/b/direct2dell/archive/2014/11/05/dell-world-two-questions-cloud-client-computing

http://blogs.citrix.com/2014/11/07/dell-launches-new-appliance-solution-for-desktop-virtualization/

http://blogs.citrix.com/2014/11/10/xendesktop-technologies-introduced-as-a-new-dell-wyse-datacenter-appliance-architecture/

http://blogs.vmware.com/euc/2014/11/vmware-horizon-6-dell-xc-delivering-new-economics-simplicity-desktop-application-virtualization.html

http://stevenpoitras.com/the-nutanix-bible/

http://www.dell.com/us/business/p/dell-xc-series/pd

Yay! I just received word that my MVP status for RDS has been renewed for 2014. Happy and honored, as always!

http://mvp.microsoft.com/en-us/mvp/Peter%20Fine-5000341

That’s a mouthful! Here’s a quick taste of the new Dell + Nutanix appliance we’ve been working on.  Our full solution offering will be released soon, stay tuned. In the meantime, the Dell marketing folks put together a very sexy video:

Previously I took at look at a small three disk Storage Spaces configuration on Windows 8.1 for my Plex media server build. Looking at the performance of the individual disks, I grew curious so set out to explore Spaces performance in Server 2012 R2 with one extra disk. Windows 8.1 and Server 2012 R2 are cut from the same cloth with the Server product adding performance tiering, deduplication and means for larger scale.  At its core, Storage Spaces is essentially the same between the products and accomplishes a singular goal: take several raw disks and create a pool with adjustable performance and resiliency characteristics. The execution of this however, is very different between the products.

For my test bed I used a single Dell PowerEdge R610 which has 6 local 15K SAS disks attached to a PERC H700 RAID controller with 1GB battery-backed cache. Two disks in RAID 1 for the OS (Windows Server 2012 R2) and four disks left for Spaces. Spaces is a very capable storage aggregation tool that can be used with large external JBODs and clustered across physical server nodes, but for this exercise I want to look only at local disks and compare performance between three and four disk configurations.

Questions I’m looking to answer:

• Do 2 columns outperform 1? (To learn about columns in Spaces, read this great article)
• Do 4-disk Spaces greatly outperform 3-disk Spaces?
• What advantage, if any, does controller cache bring to the table?
• Are there any other advantages in Spaces for Server 2012 R2 that outperform the Windows 8.1 variant (aside from SSD tiering, dedupe and scale)?

No RAID

The first rule of Storage Spaces is: thou shalt have no RAID in between the disks and the Windows OS. Spaces handles the striping and resiliency manipulation itself so can’t have hardware or software RAID in the way. The simplest way to accomplish this is via the use of a pass-through SAS HBA but if you only have RAID controller at your disposal, it can work by setting all disks to RAID 0. For Dell servers, in the OpenManage Server Administrator (OMSA) console, create a new Virtual Disk for every physical disk you want to include, and set its layout to RAID-0. *NOTE – There are two uses of the term “virtual disk” in this scenario. Virtual Disk in the Dell storage sense is not to be confused with virtual disk in the Windows sense, although both are ultimately used to create a logical disk from a physical resource.

To test the native capability of Spaces with no PERC hardware cache assist, I set the following policy on each Virtual Disk:

Make some Spaces!

Windows will now see these local drives as assignable disks to a Storage Pool. From the File and Storage Services tab in Server Manager, launch the New Storage Pool Wizard. This is where creating Spaces in Windows 8.1 and Server 2012 R2 divest in a major way. What took just a few clicks on a single management pane in Win8.1 takes several screens and multiple wizards in Server 2012 R2. Also, all of this can be done in PowerShell using the new-storagepool and new-virtualdisk cmdlets. First name the new storage pool:

Add physical disks to the pool, confirm and commit.

You will now see your new pool created under the Storage Spaces section. Next create new virtual disks within the pool by right-clicking and selecting “New Virtual Disk”.

Select the pool to create the Virtual Disk within and name it. You’ll notice that this is where you would enable performance tiering if you had the proper disks in place using at least one SDD and one HDD.

Choose the Storage Layout (called resiliency in Win8.1 and in PowerShell). I previously determined that Mirror Spaces are the best way to go so will be sticking with that layout for these investigations.

Choices are good, but we want to save committed capacity, choose Thin Provisioning:

Specify the size:

Confirm and commit:

Now we have a pool (Storage Space) and a Virtual Disk, now we need to create a volume. Honestly I don’t know why Microsoft didn’t combine this next step with the previous Virtual Disk creation dialogs, as a Virtual Disk with no file system or drive letter is utterly useless.

Next select the size of the volume you want to create (smaller than or equal to the vDisk size specified previously), assign a drive letter and file system. It is recommended to use ReFS whenever possible. Confirm and commit.

Finally, a usable Storage Space presented to the OS as a drive letter!

Performance Baseline

First, to set a baseline I created a 4-disk RAID10 set using the integrated PERC H700 with 1GB cache (enabled). I copied a 6GB file back and forth between the C:\ and M:\ drives to test 100% read and 100% write operations. Write IO was fairly poor not even breaking 20 IOPS, the RAID10 double write penalty in full effect. Reads were much better, as expected, nearing 300 IOPS at peak.

3-Disk Mirror Space Performance

To force the server to go to disk for all IO operations (since I simply moved the same 6GB file between volumes), I used a little utility called Flush Windows Cache File by DelphiTools. This very simple CLI tool flushes the file cache and memory working sets along with the modified and purged memory standby lists.

Test constants: 3-disk Mirror space, PERC caching disabled, single column (default for 3 disks), 6GB file used to simulate a 100% read operation and a 100% write operation:

Here are a few write operations followed by a read operation to demonstrate the disk performance capabilities. Write IOPS spike ~30 with read IOPS ~170. Interestingly, the write performance is better than RAID10 on the PERC here with read performance at almost half.

Turning the PERC caching back on doesn’t help performance at all, for reads or writes.

4-Disk Space, 1 Column Performance

Test constants: 4-disk Mirror space, PERC caching enabled, single column, 6GB file used to simulate a 100% read operation and a 100% write operation:

When adding a 4th disk to a 3-disk Space, the column layout has already been determined to be singular, so this does not change. To use a two column layout with four disks, the space would need to be destroyed and recreated. Let’s see how four disks perform with one column.

You’ll notice in the vDisk output that the Allocated size is larger now with the 4th disk, but the number of columns is unchanged at 1.

Adding the 4th disk to the existing 3-disk Space doesn’t appear to have helped performance at all. Even with PERC caching enabled, there is no discernable difference in performance between 3 disks or 4 using a single column, so the extra spindle adds nothing here. Disabling PERC cache appears to make no difference so I’ll skip that test here.

4-Disk Space, 2 Column Performance

Test constants: 4-disk Mirror space, PERC caching enabled, two columns, 6GB file used to simulate a 100% read operation and a 100% write operation.

Using 4 disks by default will create Virtual Disks with 2 columns as confirmed in PowerShell, this is also the minimum number of disks required to use two columns:

WOW! Huge performance difference with four disks and two columns. The read performance is consistently high (over 700 IOPS) with the writes performing in a wider arc upwards of 300 IOPS. This is a 1000% performance increase for writes and 300% increase for reads. Literally off the chart! Spaces is allowing each of these participating disks to exhibit their full performance potential individually.

Thinking this is too good to be true, I disabled the PERC cache and ran the test again. This is no fluke, I got very close to the same results the second time with all disks participating. Incredible.

Plex Performance

Since I began this whole adventure with Plex in mind, fitting that I should end it there too. Streaming a compatible file across the network yielded the following results. Skipping around in the file, all four disks were read from at some point during the stream, half spiking ~110 IOPS, the other two ~80 IOPS. This is twice the amount of observed performance on my home 3-disk 7.2k SATA-based HTPC.

Conclusions:

Using 15K SAS disks here, which are theoretically capable of ~200 IOPS each, performance is very good overall and seems to mitigate much of the overhead typically seen in a traditional RAID configuration. Spaces provides the resilient protection of a RAID configuration while allowing each disk in the set to maximize its own performance characteristics. The 3-disk one column Spaces configuration works but is very low performing. Four disks with one column don’t perform any better, so there is significant value in running a configuration that supports two or more columns. Columns are what drive the performance differential and the cool thing about Spaces is that they are tunable. Four disks with a two column configuration should be your minimum consideration for Spaces, as the performance increase adding that second column is substantial. Starting with three disks and adding a fourth will persist your single column configuration so is not advised. Best to just start with at least four disks. PERC caching showed to neither add nor detract from performance when used in conjunction with Spaces, so you could essentially ignore it altogether. Save your money and invest in SAS pass-through modules instead. The common theme that sticks out here is that the number of columns absolutely matters when it comes to performance in Spaces. The column-to-disk correlation ratio for a two-way mirror is 1:2, so you have to have a minimum of four disks to use two columns. Three disks and one column work fine for Plex but if you can start with four, do that.

Resources:

Storage Spaces FAQs

Storage Spaces – Designing for Performance

This post originated at Exit.the.Fast.Lane | Weestro.blogspot.com