For whatever reason, sometimes VM snapshots get stuck and ultimately forgotten. As you know this can be disastrous as once that line in the sand has been drawn, the resulting redo log will continue to grow until its underlying disk is completely exhausted. One-off manual snaps can be easily forgotten but worse is when programmatic snaps, the likes of NetApp SMVI or VCB, don’t get removed cleanly and start to stack up. I just had this problem when my SMVI, for reasons unknown, stopped removing snaps from one of my volumes and started incrementing them. At the point I caught the problem, some of my VMs on this volume had as many as 5 SMVI snapshots! Not good. SMVI is a great solution overall that works really well, but its handling and reporting of VI snapshots could be a lot better.



So now I have exposed a problem in my environment. The performance of my VMs is suffering and the integrity of my backups could be questionable. The first step is to create a new alarm in vCenter that watches VM snapshot size. I will warn on anything over 1GB and alert on 2GB. While researching for solutions to this problem (and manually deleting snapshots) I came across a tool written by one of NetApp’s Architects last year. Even though it was written by NetApp specifically for SMVI, it is solid enough to be used in a number of other scenarios. CVMS (Cleanup VMware Snapshots) is an executable that can be run manually via CLI or called via a script. I my case I will call this tool via a script in SMVI.
Feed it the vCenter address, credentials, and snap name prefix, then you can scope by datastore, VM, or VM set. It will go in order and remove all snaps that match the defined prefix, one at a time. Manually-created user snaps will not be affected, unless of course the snap name matches your defined prefix. The tool is incredibly well written and provides just about every customization you would care to do in this scenario.


Sample command string and output:

C:\>cvms -vcuser administrator -vcpasswd passwd -vcip 10.2.1.2 snapname bar -ds test_ds -verbose
LOG REPORT FOR CVMS
-----------------------------------------------------
CVMS Version: 1.0
Log Filename: \NetApp\CVMS\Report\CVMS_20100131_143350.log
Start Time: Sun Jan 31 14:33:50 2010
Datastore(s) selected: test_ds
Command line arguments successful.
Initializing connectivity to Virtual Center and storage appliances.
Converting Virtual Center hostname to IP address ...
Attempting to ping Virtual Center 10.2.1.2 ...
Ping of Virtual Center 10.2.1.2 successful.
Creating new Virtual Center instance for 10.2.1.2 ...
Logging into Virtual Center server 10.2.1.2 ...
Virtual Center login successful.
Collecting VMware and storage appliance configuration data.
Collecting datacenter information ...
Found 2 Datacenter(s).
Collecting host system information ...
Host system information collected.
Looking on host system esx2.internal.net for datastore test_ds ...
Requested Datastore (test_ds) is available.
Saving virtual machine information for vm2.
Saving virtual machine information for vm1.
Cleaning up snapshots for all VMs listed ...
Checking snapshot capability of VM vm1 ...
Removing all snapshots with string 'bar' from VM vm1 ...
No VM snapshots found.
Checking snapshot capability of VM vm2 ...
Removing all snapshots with string 'bar' from VM vm2 ...
Removing VM snapshot 'bar2' ...
Removal of VM snapshot for vm2 successful.
Command completed successfully.
Backup End Time: Sun Jan 31 14:34:02 2010
Exiting with return code: 0

In my particular scenario, I backup per volume in SMVI so will add a custom script to each backup job to ensure that all snaps get properly cleaned up afterwards. To present a script to SMVI, it needs to exist in %PROGRAMFILES%\NetApp\SMVI\server\scripts (or <drive>:\Program Files (x86)). SMVI can use .bat, .cmd, .pl, etc. Here is the syntax of one of my volume clean scripts:

if not %BACKUP_PHASE% == POST_BACKUP goto end
set PATH="D:\Program Files (x86)\NetApp\CVMS"; %PATH%
cvms.exe -vcip vcenter –vcuser domain\account –vcpasswd password -ds Volume1 -snapname smvi -reportdir "D:\Program Files (x86)\NetApp\CVMS"
:end

Depending on how your backups are configured, you could run a script like this at the end of the day or like me, after every backup. I backup every 8 hours so have plenty of time for cleanup in between. The report directory will house text file outputs of each instance run with the same output you would see in the CLI using the –verbose switch. Refer to the SMVI 2.0 best practices guide for available variables that can be referenced in a script.
The NA community homepage for the tool is in the references below. CVMS is not in the NA Tool Chest however, I checked, so unless someone tells me otherwise I will host a mirror of the utility.

References:
CVMS (homepage)
CVMS (mirror)
Scripting SMVI cleanup
SMVI 2.0 Best Practices Guide

My set up for this scenario is simple: 1 x FAS2020  running HA with 12 x internal SAS disks (Ontap 7.3.2). I am adding a partially-populated external SATA shelf (DS14MK2AT) to provide expansion and an additional tier of storage. The process is relatively straight forward and should apply to most arrays in the NetApp family.

Hardware Installation

NetApp uses an inordinate amount of packing material to ship what ultimately amounts to 3U’s of occupied space in the rack. Better safe than RMA I guess.
If you’ve assembled other storage arrays or servers this part won’t be much of a challenge. One item of note is that the upper shelf controller goes in upside down, which may not be immediately obvious.
Once your shelf is securely installed in the rack with the drives inserted, install your SFPs in the “In” ports on both controllers, keeping in mind the upper SFP will go in upside down. NetApp will ship 2 sets fiber pairs with SC connectors, you will only need 1 set if you are installing a single shelf. Each pair will be labeled to match “1” and “2” on both ends. If you have additional shelves to install you will need to also install SFPs in the “out” ports to connect those shelves to the loop. Make sure to properly set your shelf ID which will be “1” if this is your first shelf.

 

FC Adapter Configuration

Ok, now the fun begins. Because my FAS2020 had no external shelves previously I had both FC ports on each controller connected to my Fiber Channel fabrics providing 4 paths to each storage target. Unfortunately I now need 2 of these ports to connect a loop to my new shelf. Any subsequent shelves added to the stack will attach to a prior shelf via the “Out” ports. The first step is to remove the 2 controller ports from my fabrics, both physically and in the Brocade switch configuration. I will be using the 0B interfaces on both controllers to connect to my shelf. My FC clients, vSphere and Server 2008 R2 clusters running DSM, are incredibly resilient and adjust to the dead paths immediately with no data interruption. Perform an HBA rescan in ESX and check the pathing just to be sure everything is ok.
Before the fiber from the shelf can be connected to the controller ports, we need to change the operation mode of the FC ports. Currently they are in “target” mode as they were being used to serve data via the FC fabric. To talk to an external drive shelf they need to be in “initiator” mode. This is done using the fcadmin command in the console. Fcadmin config will display the current state of a controller’s FC adapters. Notice that they are in target mode. The syntax to change the mode is fcadmin config –t <adapter mode> <adapter>. You must also first offline the adapter to be changed because Ontap will not allow the change to an active adapter.
Once the adapter mode has been changed you will need to reboot the controller before it will take effect. If you are running an HA cluster this can be done easily utilizing the takeover and giveback functions. From the console of the controller that will be taking over the cluster, run cf takeover. This will migrate all operations of the other controller to the node on which you issue the command. As part of this process the node that has been taken over, will be rebooted. Very clean.
Fas1 taking over the cluster:
The cluster will resume normal operation after the giveback which can be verified by issuing the cf status command, or via System Manager if you’d like a more visually descriptive display.

Disk Assignments

Now that Fas2 is back up, you can verify the operation mode the 0B adapters (fcadmin config) as well as check that the disks in the external shelf can now be seen by the array. Issue the disk show –n command to view any unassigned disks in the array (which should be every disk in the external shelf).
Because I am working with a partially populated shelf (8 out of 14 disks), I will configure a 3:3 split (+ 2 spares) between the controllers and create new aggregates on both. Performance is not a huge concern for me on this external shelf, I’m just looking for reserve capacity. Here is the physical disk design layout I’ll be working with:
*NOTE make sure that “disk auto assign” is turned off in the options if you want complete control on disk assignment. Otherwise the filer will likely assign all disks to a single controller for you. It is enabled by default and needs to be disabled on both nodes.
To fix this, enter advanced privilege mode on the filer and issue the disk remove_ownership <drive name> command for each drive you want to change.

Aggregates and Spares

Now that the disks are assigned to their respective controllers, we can create aggregates. If the disk type in the external shelf were the same as the internal disks, we could add them to an existing aggregate, but since I am adding a new disk type to my array I have to create a new aggregate. I’m going to switch over to System Manager for the remaining tasks.
Each controller will need its own aggregate comprised of the disks you just assigned to each (save the spare). I will be using the default NetApp naming standard and creating aggr1. This can be performed from the Disks or Aggregate page and is pretty self explanatory.
RAID 4 is the way to go here as I don’t have the spare disks to justify RAID DP + a hot spare. Although I will be married to this decision for the life of this aggregate, it’s a sacrifice I have to make. Repeat this process on the other node. *NOTE make sure to leave at least 1 spare of each disk type, per controller, in the array. NetApp’s recommendation is as follows for ensuring you have the proper number of spares given a common disk type:

There you have it. A new shelf added hot to a NetApp array with no disruption to the connected clients. Now you can create your volumes, LUNs, CIFS/NFS shares, etc. If I add another AT shelf at some point at least I won’t have to sacrifice any more disks to spares!

Disaster recovery (DR) is always a hot topic that many companies do not do at all for one reason or another or do badly. Providing DR for a virtual environment can be a particularly challenging and expensive endeavor. In my enterprise I am running ESX4 U1 on top of Brocade fiber-channel connected NetApp FAS arrays (1 production, 1 DR) with a 1Gb Metro-Ethernet link between sites. I do not currently have the luxury of using VMware’s Site Recovery Manager (SRM) in my environment so my process will be completely manual. SRM removes many of the monotonous tasks of turning up a virtual DR environment including testing your plan, but this comes with a heavy price tag. I am fortunate enough, however, to have the full suite of NetApp tools at my disposal.

Snap Manager for Virtual Infrastructure (SMVI) is NetApp’s answer to vSphere VM backups for use with their storage arrays. SMVI does require a license and requires some specific array-level licenses, such as: SnapRestore, applicable protocol license (NFS, FCP, iSCSI), SnapMirror (optional), and FlexClone. There are some specific instances in which FlexClone is not required such as for NFS VM in-place VMDK restores. All by itself SMVI can be used instead of VCB or Ranger type products to backup/restore VMs, volumes, or individual files within the guest VM OS. SnapMirror can be used in conjunction with SMVI to provide DR by sending backed up VM volumes offsite to another Filer.

Here is the backup process:

1. Once a backup is initiated, a VMware snapshot is created via vCenter for each powered-on VM in a selected volume, or for each VM that is selected for backup. You can choose either method but volume backups are recommended.
2. The VM snapshots preserve state and are used during restores. Windows application consistency can be achieved by using VMware’s VSS support for snapshots.
3. Once all VMs are snapped, SMVI initiates a snapshot of the volume on the Filer.
4. Once the volume snaps are complete, the VM snapshots are deleted in vCenter.
5. If you have a SnapMirror in place for your backed up volumes, it is then updated by SMVI.

NetApp fully supports, and recommends, running SMVI on the vCenter server for simplicity. Setup is very straight forward and only requires the vCenter server/ credentials and the storage array names or IPs/ credentials. Best practice is to set up a dedicated user in vCenter as well as on the arrays for SMVI. The required vCenter permissions for this service account are detailed in the best practices guide in the references section at the bottom of this post.

SMVI Setup

Once SMVI can communicate with vCenter, you will see the Vi datacenters, datastores and VMs on the inventory tab.

 

Backup configuration is simple. Name the job, choose the volume, specify the desired backup schedule, how long to keep the backups, and where to send alerts. If you’ll be using SnapMirror, make sure to check the “Initiate SnapMirror update” option.

By default the SMVI job alerts include a link to the job log but the listed address may be unreachable. In my case the links sent were to an IPv6 address even though IPv6 was disabled on the server. This can be changed by editing the smvi.override file in \Program Files (x86)\NetApp\SMVI\server\etc and adding the following lines:

smvi.hostname=vcenter.domain.com
smvi.address=10.10.10.10

Once you successfully run a backup job in SMVI you will be able to see the available snapshots for the volume on the source Filer. Note the SMVI snaps vs snaps used by the SnapMirror.

I my scenario, I am backing up the 2 LUNS called VMProd1_FC and VMProd2_FC which exist in volumes called ESXLUN1 and 2, respectively. Both of these volumes have corresponding SnapMirrors configured between the primary and DR Filers.

 

A couple of things to keep in mind about timing:

• It is a good idea to sync your SnapMirrors outside of SMVI which will reduce the time it takes when SMVI updates the mirrors. Just make sure to do this at a time other than when your SMVI jobs are running!
• If you are using deduplication on these volumes (you should be), schedule it to run at a time when you are not running SMVI backups or syncing SnapMirrors.

Once SMVI successfully updates the SnapMirror, you will see the replicated snapshots on the destination side of the mirror as well. DR for your ESX environment is now in effect!

Testing the DR Plan

Here comes the fun part and where having SRM would be extremely helpful to automate most of this. Thanks to NetApp’s FlexClone technology we can test our DR plan without breaking the mirrors, so you could test whenever and as often as necessary without affecting production.

First step, create a FlexClone Volume of the replicated snapshot you want to be able to mount and test with. Choose the appropriate volume then select Snapshop—>Clone from the Volumes menu. Important to note is that you must use a File space guarantee or the FlexClone creation will fail! This can be done via System Manager, CLI, or FilerView:

Your new volume will now be visible in the Filer’s volumes list, which also creates a corresponding LUN that you’ll notice is offline by default:

 

Bring the new LUN online and then you will be able to present it to your DR ESX hosts:

The LUN should instantly appear on the ESX hosts in your iGroup but if it does not run a rescan on the appropriate storage adapter:

ESX sees the LUN, now it needs to be added to the cluster. Switch to the Storage page in vCenter and select “add storage.” Select the new disk and be sure to select “Assign a new signature” to the disk or ESX will not be able to write to it! Only a new UUID is written, all existing data will be preserved.

The new LUN will now be accessible to all ESX hosts in the cluster. Note the default naming convention used after adding the LUN:

This is where things get REALLY manual and you wish you ponied that $25k for SRM. Browse the newly mounted datastore and register each VM you need to test, one by one.

 

Now your organization’s policies take over as to how far you need to go to satisfy a successful DR test. Once the VM is registered it can be powered on, but if it’s counterpart is still running in production disable its vNIC first. If you need to go further, then shut down the production VM, re-IP the DR clone and then bring it online. If you need to have users connect to the DR clone then there are other implications to consider such as DNS, DFS, ODBC pointers, etc. When your test is complete, power off all DR VM clones, dismount the datastore from within ESX, delete the FlexClone Volume on the DR Filer, bring the production VMs back up, check DNS, done. A beautiful thing and best of all the prod—>DR mirror is still in sync!

References:

SMVI 2.0 Best Practices

SMVI 2.0 Admin Guide

NetApp KB52524

Folder redirection has been around since Windows 2000 and has undergone significant changes since then. The core function is the same: take a local directory path and point it somewhere else, without the user knowing or caring that it isn’t local. The advantages of this are that your users can store their important documents “on the network” without having to map drives or instructing them to save in a certain location. You can selectively redirect documents and app data but exclude photos, music, etc if you choose. In this example, I am interested in redirecting My Documents for all users to a secure, redundant, and high performance NetApp Filer. Technologies involved are Windows 7 Enterprise, Server 2008 R2 DFS, and NetApp CIFS shares running on a FAS2020.

First thing, create your CIFS shares on the filer. The way NetApp NAS with the CIFS protocol works is that the filer actually becomes a member of your domain. You can even apply certain GPO settings to it! The stated domain type is incorrect as I’m running in 2008 R2 native mode, but this doesn’t affect anything functionally from what I can tell.

  

Your CIFS shares are then managed just like a regular Windows server. You can even connect to the filer via the computer management MMC.

Each share you create exists inside a volume and has an associated qtree. All the other NetApp goodies still apply: deduplication, snapshots, auto volume grow, and opportunistic locking. The rest of the options look very much like a regular Windows file server.

I have created a single hidden share called Users$ that sits in a 300GB volume. All of my user’s My Documents will live here. Following best practices, I have granted authenticated users full control to the share as I will control granular access permissions with NTFS. We’re ready to prepare the DFS namespace.

Now in my redirection GPO I could simply point all users to redirect to \\<filername>\users$\<username> but one of the values DFS provides is a consistent domain-based namespace: \\domain\<dfsRoot>\<redirection_root>\<username>. Everything that will exist as a file share in my environment will be accessible via a DFS namespace, much cleaner this way and much easier to change targets should I need to enact my DR plan which I would do via folder targets to my DR filer. I first create a new domain-based namespace in Server 2008 mode called Users (add a $ to the end to make it hidden):

This is simply the DFS root which lives in the local file system space on my namespace server (domain controller). To be able to point to my filer I need to create another folder inside of this DFS root, that can then be targeted to a matching folder on my NAS. So I will create a new folder called “root” on both NAS: \\cufas2\users$\ and DFS: \\domain.com\Users\. First create the new root folder on NAS then add a folder to DFS with a target that will point to the root folder on the filer. Additional targets can be configured and controlled for redundancy, replication, and DR.

Before we configure the GPO let’s set permissions on the Root folder. This is a critical step and is what will ultimately make or break this configuration. Since this is the root folder for the entire share, remove permission inheritance and set the following permissions:

• CREATOR OWNER - Full Control (Apply onto: Subfolders and Files Only)
• System - Full Control (Apply onto: This Folder, Subfolders and Files)
• Domain Admins - Full Control (Apply onto: This Folder, Subfolders and Files)
• Authenticated Users - Create Folder/Append Data (Apply onto: This Folder Only)
• Authenticated Users - List Folder/Read Data (Apply onto: This Folder Only)
• Authenticated Users - Read Attributes (Apply onto: This Folder Only)
• Authenticated Users - Traverse Folder/Execute File (Apply onto: This Folder Only)

This will allow all users to programmatically create their directory folders beneath the root folder as well as be granted full control to them without the ability to see anyone else’s folders. Domain Admins will have full control to all folders. Now we’re ready to set up our folder redirection GPO.

Folder redirection is a user configuration setting so the GPO that contains these settings must be linked to an OU that houses user accounts, or linked high enough in the AD tree so that user-housing OUs will inherit. Redirection can be set in basic or advanced mode, basic redirecting everyone to the same location. Advanced enables the opportunity to redirect users differently based on security group. In either case you can redirect to the user’s home directory, create a folder for each user under the specified root, redirect to a specific location or to the user’s local %userprofile% path. I will be using the “create a folder” option under the advanced mode and the path is the DFS root created earlier: \\domain.com\users$\root. For now the policy will apply to one group, domain users, but I will have future flexibility should I need to have additional groups redirected differently. The effect of this policy is that each user, once successfully redirected, will automatically have a new folder under the root directory named after their username, with the documents folder beneath it. Any other folders I choose to redirect will also live under this %username% directory.

Additionally, you can grant the user exclusive access to their folder which keeps out even domain admins and creates problems for backups. Since we’ve set very specific permissions on the root we don’t need to worry about this anyway. The other pertinent option is to move the contents of the source to the destination which I have had problems with in my environment. I’ll be leaving both of these unchecked.

Now when a user logs into any machine in the enterprise they will see the exact same documents folder which lives safe and sound on an enterprise storage array. Should the primary array fail, DFS will repoint their redirected documents to a DR filer in another datacenter.

 

Something else to consider for your laptop users is offline files in conjunction with folder redirection. I have this enabled by default for all laptops and expressly disable them for desktops. This is a good compromise so that laptop users will enjoy the benefits of redirection while in the office but will also be able to access and work on these documents while away. The next time they connect to the corporate network any changes they made to an offline file will sync back up with their redirected folders on the NAS.

I’m documenting the easy method to this process having just gone through the more round-about backdoor method. The proper process to mask and present LUNs on a NetApp array is through the creation of LUNs and iGroups (mask) which is wizard driven inside the NetApp System Manager. This can all be done, of course, via the CLI which I have done but this way is much simpler. This all assumes that you have already properly setup FCP or iSCSI on your array and that all requisite fiber-channel zoning is complete. My filers are running ONTAP 7.3.2 and I have 2 fiber-channel fabrics powered by Brocade switches.

Open System Manager and navigate to the controller node that will be hosting your LUN. Expand Storage and click LUNs. Click create under LUN Management. Enter a name for your new LUN along with the size and presentation type.

Next allow ONTAP to create a new volume or choose an existing container. I’ll create a new volume:

The next step will mask the LUN to a host of your choosing via the creation of an iGroup. Click Add Initiator Host and select the WWpN of the host HBA you wish to connect to the LUN. You can only add a single initiator in this step. Once complete move your new iGroup from “Known initiator hosts” to “Hosts to connect”.

Review the changes to be made in the summary and click next to start the process. When complete you will see your new LUN and iGroup with the host’s WWpN. By default NetApp likes to put each initiator port in its own iGroup. Because I am building a SQL cluster and need all hosts to see all of the same LUNs I will be adding all cluster members to the same iGroup. This can be done either by creating a new iGroup and selecting the WWpN you want to add followed by reassigning it to the proper iGroup, or you can copy/paste the port name into the iGroup you just created.

Make sure to enable ALUA if your array supports these features.

Verify the iGroup via the CLI if you wish by issuing the igroup show command:

Now, assuming you have ONTAP DSM (MPIO) installed, you should see the new LUN on your host.