This series has meandered down a long and winding path from basic concepts to theory to practice. In this final installment, I’m going to show a lot of really boring graphs and charts to answer the big question: What about performance?
Part 1 – Hyper-V storage fundamentals
Part 2 – Drive Combinations
Part 3 – Connectivity
Part 4 – Formatting and file systems
Part 5 – Practical Storage Designs
Part 6 – How To Connect Storage
Part 7 – Actual Storage Performance
I think what people really want to know is, out of all the various ways storage can be configured, what’s going to provide the best performance? I will, as always, caution against being that person with an unhealthy obsession with performance. I’ve met too many individuals who will stress over an extra millisecond or two of latency in a system where no one will think anything is wrong unless the delay passes several seconds. If you truly need ridiculously high storage speed and you are not already a storage expert, hire one.
What I’m going to do in this post is show you performance results that were all taken from the exact same hardware configured in several different ways. The test was configured exactly the same way each time. What I want you to understand from that is that the numbers themselves are largely irrelevant. The hardware that I used is not up to the challenges of anything more than a small business environment, and by small, I mean something like under twenty users. What’s important is how the numbers do or do not compare against each other across the various test types.
Materials and Methods
This section lists the hardware, software, and configurations used in the tests.
3x Hewlett-Packard N40L MicroServers with firmware 2013.10.01. With the exception of the hard drives, all hardware is configured as indicated in our sub-$2000 build paper.
In the server used as the storage server, two Seagate ST3000DM001 drives were added to the defined configuration. These were placed into a mirror Storage Spaces array.
Although it should not have impacted the test outcomes, both of the Hyper-V systems were expanded so that their internal drives are in a hardware RAID-1 configuration. No test was run from these drives.
The hardware switch used is the Netgear GS716T device indicated in the document, with firmware version 184.108.40.206.
This test used a virtual machine running Windows Server 2012 R2 Standard Edition in evaluation mode, build 6300.
The virtual machine used in testing was running Windows Server 2012 R2 Standard Edition, evaluation build 9600. All Windows Server patches were current as of February 10th, 2014.
The testing software used was IOMeter version 1.1.0 RC1.
The storage host is running Windows Storage Server 2012 R2 Standard Edition, same build and patch level as the guest.
The two Hyper-V hosts are running Hyper-V Server 2012 R2, same build and patch level as the guest.
No out-of-band hotfixes were applied to any system; only those available through Windows Updates were applied.
The test virtual machine was a generation 1 machine assigned 2 virtual CPUs and 2GB of RAM. Its C: drive was a dynamically expanding drive with a limit of 60 GB. This drive was placed on separate storage from the test volume. The test volume was attached to the virtual SCSI chain and set to 260 GB in size. It was dynamic in some tests but converted to fixed for others. The type used will be reported in each test result. Both disks were in the VHDX format.
The IOMeter test was configured as follows:
- Only one worker was allowed.
- The Maximum Disk Size was set to 65,536,000 on a disk with 4k sector size and 524,288,000 on disks with a 512b sector size; this results in a 250GB test file.
- A single Access Specification was designed and used: 16K; 75% Read; 50% Random
- The Test Setup was limited to a 30 minute run
- Results and configurations were saved on the virtual machine’s C: drive for tests run inside the VM
- All other settings were kept at default
Download the IOMeter ICF. (the current version doesn’t actually allow import of saved ICF files, but it is human-readable)
The hosts were configured as follows:
- Onboard (Broadcom NetXtreme I) adapters were left strictly for management
- All other adapters in the Hyper-V nodes (4x Realtek) were fully converged in LACP with Dynamic load-balancing in most tests
- Some tests, indicated in the text, broke the teams so that each node had 2x Realtek in a Dynamic load-balancing configuration passing all traffic besides management and storage. 2x Realteks were assigned their own IPs and participated directly in the storage network.
- The Realtek adapters do not support VMQ or RSS
- The two Hyper-V nodes communicated with the storage server using 2 vNICs dedicated to storage communications on a specific VLAN.
- MPIO was enabled across the nodes’ storage NICs for iSCSI access. SMB multichannel was in effect for SMB tests. This was true in both vNIC and pNIC configurations. The MPIO method was the default of round-robin.
- The two other adapters in the storage node (2x Realtek) were assigned a distinct IP and placed into the VLAN dedicated to storage
- The two Hyper-V nodes were joined into a failover cluster
Target configuration notes:
- All tests were conducted against data on the 3 TB Storage Space
- Between tests, data was moved as necessary to keep the test file/VHDX as close to the start of the physical drive as possible.
- No other data was on the disk beyond the MFT and other required Windows data.
- iSCSI target was provided by the built-in iSCSI Target of Windows Storage Server 2012 R2
- SMB 3.02 access was also provided by the built-in capability of Windows Storage Server 2012 R2
Hardware RAID-1 would have been preferred for the 3 TB Seagate drives, but the onboard RAID array cannot work with disks that large. It is possible that the extra CPU load negatively impacted the local test in a way that would not have impacted any of the remote tests. There was no way to control for that with the available hardware.
A truly scientific test would include more hardware variance than this. The switch presents the major concern, because another switch was not available to rule out any issues that this one might have.
The following charts show a graphical representation of the results of each test.
The first shows all results together in a single combined view:
The next three show the individual result sets.
Megabytes per Second Comparison
Average Response Time Comparison
Download the cumulative test results in Excel format. Each tab in the Excel workbook represents one test. All IOMeter results are contained here, not just the three metrics listed above.
With the low-end nature of the hardware, the variances of these results are more pronounced than they would be in higher-end hardware.
RSS support would likely have made a substantial difference.
There are a couple of things to be aware of when scanning the results. First, the “Local Disk, No VM” result was performed right on the empty Storage Space. This space uses 4k sector sizes. The virtual machine’s VHDX sets its cluster size to 512 bytes. This alone affected the outcome in a way that makes all the other tests look like the difference between virtual and physical is more dramatic than it is.
The test configuration presents a very difficult case for hard disks. Larger chunk sizes result in better performance. The chunk size was chosen arbitrarily, but can’t be used to predict real-world performance even with the same hardware. The goal was to use a single size to reduce variance across the tests. The 75% read percentage was chosen as most real-world operations are read-heavy. The breakdown of read vs. write performance can be found in the download results file.
Most of the remote tests had a few moments of very high delays that skewed the total results negatively. I wasn’t able to determine if these delays were legitimate, as they recorded lengths of sometimes approaching two minutes. They always occurred in the first couple of minutes of each remote test and then never again. Usually by the end of each test, the numbers had settled.
Obviously, not all possible permutations were tested, primarily due to time constraints. However, speeds increased substantially when the fully converged design was split out and two physical NICs were designated for storage communications.
For the “contention” test, another VM was running the same IOMeter load on the other node against the same CSV while data was collected on the test VM. Notice how little that affects the outcome.
Raw hardware access is always the fastest possible configuration. However, higher-end hardware would not experience the wide range of differences and would bring offloading technologies and optimizations that would close the gap. Local access is, as expected, faster than over-the-network access.
One thing the results are pretty unambiguous about is that dedicated physical NICs work better for storage communications than virtual NICs on a converged fabric. Again, higher-end hardware would reduce this impact dramatically.
It also appears that running on a non-owner node has an effect. Unfortunately, with a test group this small, there’s no way to be certain that there wasn’t a network issue. This warrants further study.
The multichannel SMB configuration showed that it performs better than an iSCSI configuration, all else being equal. This would also eliminate the penalty imposed by running on a non-CSV owner node.
The difference between the dynamic and fixed SMB results was surprisingly wide. This disparity warrants further study.
This series has talked at very great length. Now it’s time to actually get something done. What I’m going to do in this piece is show you how to connect to storage.
Part 1 – Hyper-V storage fundamentals
Part 2 – Drive Combinations
Part 3 – Connectivity
Part 4 – Formatting and file systems
Part 5 – Practical Storage Designs
Part 6 – How To Connect Storage
Part 7 – Actual Storage Performance
How you connect depends on entirely on how the storage is presented. You can use either PowerShell or the GUI. Since this post is likely to be long anyway, I’ve decided to show only GUI methods. I’ve asked Jeffery Hicks to write a matching post that shows the PowerShell equivalents. That work is in progress and will be published at a later date.
Notice: All of the screenshots and procedures in this post are from 2012 R2. 2012 shouldn’t be dramatically different, but I did not compare this to a 2012 system.
In this section, I intend to show you how to connect to and use local disks. However, once you’ve connected to a LUN on Fibre Channel or iSCSI storage, it is treated the same way.
You have the option of using the Disk Management MMC console, but this is the 2012 era, and Server Manager is the current tool. Server Manager can’t control CD/DVD drives or MPIO policies (unless I missed something) but it can do everything else. As you probably know, the systems I use for all the things I do in this blog are from our sub-$2000 cluster document. Two machines are running Hyper-V Server. The third is running a GUI version of Windows Server and hosts the storage for the cluster. In the following screenshot, I’m looking at the Disks tab on the File and Storage Services section of Server Manager on that Windows Server. It is also connected to both of the nodes (using item 2, Add other servers to manage on the welcome screen of Server Manager). Server Manager also comes in the RSAT tools for Windows desktops. You must use Windows 8.1 to manage 2012 R2 servers.
As you might have expected, I’ve modified the builds somewhat from the document. SVHV1 is still exactly as described in the document. In SVHV2, I’ve purchased another 250GB drive and converted it to a RAID 1 using the BIOS (I’ve got another for SVHV1 as well, I just haven’t gotten around to rebuilding that system yet). SVSTORE has the drives from the document, but I’ve also picked up a pair of 3 TB drives (the 2TB drive didn’t make it into the screenshot).
The 3TB drives are completely fresh, with no data. I want them to be in a mirror for the virtual machines. This will give them superior read performance and, more importantly, protect them from drive failure. If this system were intended to host virtual machines locally, I’d want this to be in a hardware RAID-1. That’s because any software mirroring takes processing power, and you really shouldn’t be stealing CPU time from your VMs. Unfortunately, I learned that the N40L’s RAID processor just can’t handle drives of this size. Apparently, I can buy an add-in card, but I just don’t want to deal with it. So, I’m going to use Storage Spaces to create a mirror. Before I get into that, I’ll show you how to set up a single disk.
Prepare a Local Disk for Usage
This section will use a local Storage Spaces virtual disk for its example. These are example the same steps you’d use for a single internal disk, another type of virtual disk (provided by an internal hardware RAID system), or an iSCSI or Fibre Channel disk.
I’m only going to show you one way to do this. There are quite a few others, even in the GUI. The way most of us are used to doing it is through Disk Management, and that process has not changed. I’m going to use Server Manager, because this is the new stuff.
In the File and Storage Services section of Server Manager, go to the Disks sub-section under Volumes. In the Disk list at the top, find your new disk. If it’s freshly installed, it’s probably Offline. If it’s online, it may show an Unknown partition.
To get started, right-click on the disk to work with and click Bring Online. You’ll get a warning message that lets you know that onlining a disk that is currently being used by another server might cause data loss. Acknowledge the message (I’m assuming, for this section, that the disk isn’t being used by another server). From this point, you’d traditionally Initialize the disk, then create a volume. With Server Manager, you can now get it all done in a single wizard. Right-click on your freshly onlined disk and choose New Volume. This will kick off the New Volume wizard. The first screen is informational, so click Next.
The second screen has you choose the server and disk to work with. Since you specifically started by right-clicking on a disk, you should already have the correct disk selected:
Disk and Server Selection
Once you click Next, you’ll get a pop-up dialog. This occurs because we did not take the Initialize step. It’s just telling us that it’s going to initialize the disk as a GPT. This shouldn’t be a problem, since our system is already bootable and Windows Server 2012+ has no issues with GPT. If you prefer the MBR method for any reason, you’ll have to use something other than Server Manager to initialize the disk. Click OK to proceed or Cancel to find another way.
The next few screens are modernized versions of Disk Management’s screens: capacity of the new volume, drive letter/mount selection, and format/label/allocation unit specification.
Depending on the options you have available/installed, the next screen will be for Deduplication. I don’t want to spend a lot of time on this, but the short introduction is that for Hyper-V, this is most appropriate for VDI. You can use it for hosting servers, but you’re sort of on your own if performance suffers or you don’t get the space savings boost you expect. Remember that Microsoft’s deduplication does not occur inline in real time. It occurs on a set schedule.
New Volume Deduplication Settings
After this, it’s the Confirmation and Results screens, where you can watch your disk get created. Unlike Disk Management, Server Manager’s wizard only performs quick formats, so this shouldn’t take long regardless of disk size. Once it’s done, your disk is ready to use.
Prepare a Storage Spaces Volume for Usage
The disk(s) that you want to use can be online or offline (right-click option on the Disks tab), but they must have at least some space that’s not already claimed by a volume or partition. For this demo, I’m going to use completely clean disks. To get started, go to the Storage Pools section in File and Storage Services. In the lower right, under Physical Disks, make sure that the disks you want to use appear. You can use the Tasks menu on the Storage Pools section at the top to Refresh or Rescan Disks if some are missing (why these aren’t on the Physical Disks Task menu is beyond me). To get started, open either of these menus and click New Storage Pool.
Storage Pool Screen
On the first screen of this wizard (not shown), you give the new Storage Space a name and, optionally, a description. I just called mine “SVSTORE Space”, because I’m not very creative. Click Next when you’re happy with the name (or at least accepting of it).
On the second screen, you select the disks you want to be part of the new Storage Space. On the right, each disks has a drop-down for Automatic, Manual, or Hot Spare. Automatic allows Storage Spaces to figure out the best way to use the disks. I haven’t spent any real time researching what Manual does, but using it allowed me to specify the interleave size during the creation of a virtual disk (that part comes later) . Hot Spare does just that, makes the disk available as a hot spare. If you’re not familiar with this term, a hot spare disk sits empty until an array disk fails. The data from that disk is copied to the hot spare and it then comes online in place of the failed disk. I’ve never seen this used in a two-disk system and not sure how it would work, perhaps active/passive. Usually, hot spares are used in a parity system or as a +1 for mirrors or RAID-10 configurations. I selected Automatic for both my drives. Click Next once you’ve set the selections as desired.
Storage Pool Wizard Disk Selection
Your hard work will be rewarded with a summary screen. If you click it, click Create. If you don’t, click Back or Cancel. These directions will assume you went with Create. In that case, you’ll get a screen with a few progress bars. Once they’re done, hopefully none of them turn red. These directions will also assume they stayed blue. Once the process completes, you can Close. Before you do that, you might want to check the box for Create a virtual disk when this box closes. If you don’t, then you’ll have to right-click on the new storage pool you just created and select Create Virtual Disk…, which is an entire extra click.
The first two screens of the virtual disk wizard are pretty easy. The first is just explanatory welcome text and the second has you choose your pool. My box only has one pool so it didn’t take me long to decide. On the third screen, you have to give your new virtual disk a name and, optionally, a description. I just called mine “Mirror”, because, well, there’s definitely a reason I write technical material and not popular fiction. You’ll notice there’s a checkbox to tier the storage. Mine is grayed because I have only spinning disks; you need both spins and solids in the same batch for tiering to work. Click Next when this page is good enough.
It’s on the fourth screen that you get your first real choice: Simple, Mirror, and Parity. These are effectively RAID-0, RAID-1, and RAID-5/6, respectively. As I understand it, there are some differences between these and the industry-standard versions, but I don’t know what all of them are. I know that Storage Spaces mirroring has the ability to create a three-way copy if you add a third disk. You can read part 2 of this series if you need a refresher on the various RAID types.
Virtual Disk Layout
The next screen has you choose between thin (dynamic) or fixed (thick) provisioning. I have done plenty of virtualization on thin-provisioned storage in the past, including thin-provisioned virtual disks (VHDs) on thin-provisioned SAN LUNs. The upside is that things like volume creation can be a lot faster and snapshots can be a lot smaller (on actual SANs, I don’t know about SS). The downside is fragmentation and other performance hits when the space is expanded. In larger storage systems with many spindles, the cost of these operations is usually lost in other latencies and is irrelevant to all but the most demanding systems (or neurotic administrators). In a two-spindle system, the hit is more noticeable… although the load is usually a lot lighter, so it still probably doesn’t really matter. But, I’ll never use this space for anything else, so I don’t really have a great reason to use thin provisioning.
Virtual Disk Provisioning
The final screen you can make changes on is the size screen. I’m just going to use it all. I’ve heard rumors that having a single VM on a CSV and keeping both owned by the same node improves performance, but I’ve never seen a published benchmark that corroborates that. If that’s the sort of thing you like to do, then feel free to split your Spaces up. You might also want to have separate Spaces so you can have some for Hyper-V storage and others for regular file shares or SOFS storage.
Virtual Disk Size
After this, there’s nothing left but to review the Confirmation screen and watch the progress bars on the Results page. Well… actually, there is more. I would definitely check the Create a volume when this wizard closes checkbox before clicking Finish. Otherwise, you’ll have to jump over to the Disks or Volumes tab to start the wizard, and that’s a lot more than just one extra click. It’s at least three.
I’m not going to talk you through the New Volume wizard again. Read the Prepare a Single Disk for Usage section if necessary.
I would really like to post a nice how-to guide on FibreChannel connections. Unfortunately, it’s really vendor-driven. The basic process involves loading drivers and software for your host bus adapter (HBA), then masking and connecting World Wide Names. Work with your storage vendor.
iSCSI works in a client/server configuration, but it uses the terms “initiator” instead of “client” and “target” instead of “server”. Perform any necessary configuration on the target. As with Fibre Channel, configuration of the target is determined by the storage vendor. The target needs to be set up first.
Before you get started configuring Windows or Hyper-V Server as an initiator, there are a few things you need to sort out. Don’t team NICs to be used in iSCSI. You can use vNICs that are connected to a Hyper-V virtual switch that is hosted to a team, but dedicated physical NICs are faster. The initiator and target should be in the same subnet(s) if at all possible. Routed iSCSI traffic can be noticeably slower.
There is a connection GUI for iSCSI on all modern versions of Windows and Hyper-V Server. It exists whether or not the GUI is installed. At any command or Run prompt, just type iscsicpl.exe. If you’ve never run it before, you’ll be presented with the following dialog:
iSCSI Service Message
You could answer No if you like, but you might find it difficult to connect to iSCSI targets if you do. For our purposes, Yes is the correct answer. Once you that’s done, you’ll be presented with the iSCSI Initiator Properties dialog. This is a busy dialog, and there is a lot going on. I want to first jump over to the Configuration tab, because it contains valuable information that usually gets skipped in iSCSI discussions.
iSCSI Initiator Configuration Tab and iSCSI Security
Let’s take a look at the Configuration tab:
iSCSI Configuration Tab
The vital data here is the Initiator Name. When this computer requests to connect to a target, this is the name it’s going to present. You’ll notice you can change this. I hope you’re asking, “Then, is it possible for me to lie and pretend I’m an entirely different computer?” The answer is yes, absolutely you can. All Microsoft initiators, by default, start with iqn.1991-05.com.microsoft: and then the FQDN of the computer. You could change the initiator in an attempt to confuse a potential attacker, but this is an approach called “security by obscurity” which is rightly derided because it’s only going to fool the laziest and most incompetent attackers. However, most targets do allow you to restrict access to specific initiators and IP addresses. IP addresses are a bit tougher to spoof than initiator names, since the network behaves poorly and has blatantly obvious problems when two systems try to use the same IP address at the same time, but this is still not a good security measure.
If iSCSI security is really a concern, then you have three choices. The first is CHAP (challenge-handshake authentication protocol). This is a simple security method that involves a pre-shared key. The CHAP button you see on the dialog above is used only for mutual or reverse CHAP. For this, the target will present a password to the initiator, because it’s just as easy to spoof a target as to spoof an initiator. Here is where you set that password. It is important to understand that the only security this provides is at the moment of connection. If CHAP fails, no connection is made. If CHAP succeeds, everything sends back and forth in the clear without further verification, at least until the next connection attempt.
The second iSCSI security method is IPSec. I’ve used IPSec just enough in my career to really hate it. What IPSec gives you is a completely encrypted communications chain, but at a high overhead cost, and that’s before you get into the administrative nightmares. I’m not going to talk about IPSec any further except to say that this is where you configure it on the initiator side. If you’re thinking about IPSec, I highly recommend you also consider iSNS as it might reduce your headaches a little. iSNS will be briefly revisited in the next section.
The third method is to isolate iSCSI onto its own network. We’ve already talked about using a dedicated subnet for performance reasons, but it’s also an easy way to get some security for the traffic. If you don’t put any gateway devices in your iSCSI network, then an attacker will be forced to compromise your network at the layer-2 level, which is difficult, to say the least. You can employ VLANs to add a bit more security. The best approach is to use dedicated switches and switch ports.
The Reports button needs a GUI edition of Windows Server to run. Otherwise, it just goes from gray to blue and back to gray. Neat!
iSCSI Initiator Discovery Tab
Discovery should be self-explanatory, but there is, unfortunately, a lot of confusion with it. Discovery merely finds out what is available on a given target. You provide the target IP address (or name, but of course, it will be resolved to an address), and Discovery goes to see what disks are there. Just because an initiator can discover disks doesn’t mean that it can connect to them!
Here is a shot of the Discovery tab with some portals added:
iSCSI Initiator Discovery Tab
Use the Discover Portal button to enter the portals you want to connect to. When you click, you’ll be given a small dialog that will allow you to enter the IP or DNS name of the target as well as the port to connect on. It is recommended that you use the IP, or your iSCSI infrastructure will be dependent upon DNS. It is also recommended that you not change the target’s port number without a compelling reasons (security by obscurity is not a compelling reason). You can click the Advanced button on this small dialog to configure more security options. This dialog will be shown and discussed in the next section, as it is the same dialog you see when connecting to a disk. However, the context is very important. This page is just for connecting to the discovery portal. Settings here are completely distinct from disks. Most initiators do not secure their portals against discovery attempts, so setting things here might actually cause unnecessary problems. The only thing I set here is ensuring that the source addresses for each portal is on the same subnet as the target, but that’s not strictly necessary. Once you’ve entered information for a portal, it will be displayed in the dialog as shown above. Please be aware that you can completely mangle the information and it will still be accepted, although it’s not going to be useful.
Unfortunately, there is no way to modify the settings for an existing portal item. You have to remove it. This will pop up a warning dialog about the item being on the Favorites tab. I’d deal with that before trying to re-add a portal, or you might be in for some confusion. Look to the relevant section below for more information.
The lower portion of this dialog is for iSNS settings. It requires an iSNS server, obviously. What you can do with such a thing is preconfigure just about everything, including IPSec settings, and manage them from a central location. I have never used an iSNS server, nor have I ever talked to anyone who has. However, its feature list is fairly compelling if you’re looking at a complicated iSCSI deployment.
iSCSI Initiator Targets Tab
The Targets tab is the first you see when you start the applet. Here is what one looks like after a target has been added (and maybe with a click of the Refresh button):
The Quick Connect button is for attaching to a new portal and target, not one that’s already in the Discovered targets box (I imagine this button is why people get confused about the purpose of discovery). It doesn’t allow for any modifications to security or the port, which is why I don’t usually guide people toward using it.
Target Tab’s Properties Sub-Dialog
By highlighting an item on the Discovered targets list with a status of Inactive, you might be able to use the Connect button to jump right onto it. However, you won’t want to do this if there are any special needs, such as multiple source and destination IP paths. This is because you’re only going to get one shot to create connections. Some initiator software may intervene and build up all the necessary IP paths, so this may not matter. For most of us, we’ll use the Properties button. You’ll get this box:
iSCSI Session Properties
If there are any active connections to the disk, the list box at the top will contain entries. Your ultimate goal is to have one session per path to storage. For the system I’m using, my initiator has IPs 192.168.50.10 and 192.168.51.10, while my target has IPs 192.168.50.100 and 192.168.51.100. So, I want two sessions, one on 192.168.50.0/24 and the other on 192.168.51.0/24. I start by clicking Add session, and I get the following (this is the same dialog you get when clicking Connect on the Targets tab):
iSCSI Connect to Target Dialog
In this screen capture, I’ve already selected the Enable multi-path box, because I intend to use two paths. But, I want to define the paths myself, because I find that Windows usually doesn’t figure out the connections. Click the Advanced button to set that up. You’ll be presented with the Advanced Settings dialog (which is the same dialog you get when adjusting Advanced Settings on the discover portal):
iSCSI Advanced Settings
I’ve already set the options here in accordance with my first connection. Don’t just arbitrarily select either Data digest and/or Header digest because they sound important. If the target isn’t configured to use them in the exact same pattern you select here, your connections won’t work at all. What they do is verify data validity at the expense of a bit of overhead. The boxes indicate whether you want to calculate a checksum on the data portion of the iSCSI TCP/IP packet or its header, or, of course, both.
The lower portion of this screen is for CHAP security. You need to make sure that this is configured the same way that your target is. The Target secret refers to the password that the initiator is expecting for this disk, NOT the discover portal. As mentioned elsewhere, mutual authentication is the password that the target will attempt to send back to the initiator and is set on the Configuration tab. Your target may refer to this is Reverse CHAP. The lower two boxes are for RADIUS authentication. I have never used RADIUS in an iSCSI context, but the basic approach is to ensure that both your target and initiator can contact the same RADIUS system. As an aside, I would highly recommend that you don’t use a RADIUS server that lives on a VM that is on this iSCSI target. I hope I don’t have to explain that.
Once I have this screen the way I like, I can click OK twice. I’ll be back at the Properties screen, and if all is well, I’ll have an entry in the list box with an undecipherable identifier made up of two gigantic hexadecimal numbers. Now, I’ll go back through the Add session dialog and repeat my earlier steps to set up a connection from initiator IP 192.168.51.10 to target IP 192.168.51.100.
The Devices button allows you to look at some connection details for this disk. Much of it will probably be pretty cryptic. However, there is an MPIO button that, as you will see, can be quite useful.
At the very bottom of the Properties dialog is the section on MCS (Multiple Connected Session). I never use this. Instead, I use MPIO. This will be discussed on the next section. For now, let’s go to the next tab in the iSCSI dialog.
iSCSI Initiator Favorite Targets Tab
I won’t include a screen capture of this tab because there’s not much to it. When the system reboots, it will try to connect to every iSCSI disk listed here. If you’re using multiple paths, then each path will be listed twice. The Details button will show how you’ve configured the connection. Connections you’ve made from anywhere else in this applet will automatically be placed here. If you don’t ever want to connect to a disk again (or if you want to reconfigure it), remove it from here.
iSCSI Initiator Volumes and Devices Tab
I’ll be perfectly honest: I’m not entirely certain what the practical use of this tab is. Maybe some apps benefit from it. Hyper-V Server will function just fine if you never use it or if you fill it all in. Use the Auto Configure button if it makes you feel better, but don’t expect fireworks.
iSCSI Initiator RADIUS Tab
This is for informing the initiator about RADIUS servers it should use if it’s going to participate in RADIUS authentication to targets. I won’t spend any time on this either; if you know how to set up RADIUS, this page will be easy for you.
iSCSI Additional Points
Once you have everything configured, you’re connected. Windows will now treat the disk as though it were local. Just look in Disk Management or DISKPART or Get-Disk if you don’t believe me. Read the Internal/Direct Attached Disks section above to continue.
The issue with iSCSI is it really doesn’t have a good “reconnect” mechanism. If it loses connectivity, it’s supposed to try to reconnect on its own. I couldn’t say how often that works, but I know it doesn’t always work. If there is a way to get it to reconnect after you’ve manually disconnected or determined that a disconnect occurred, I have no idea how to get it to reconnect without restarting the server (not the service) or rebuilding the connection from scratch. If you have an iSCSI connection made that is idle for a very long time, it seems like it quietly drops the connection but thinks it’s still active (this could be a target issue, for all I know). When does this happen? With Cluster Shared Volumes, mostly. There are a lot of people out there that will tell you to run one VM per CSV. I’ll investigate whether or not that’s good advice at some point in the future. But, the point to be aware of is that if you do that, and the VM sits on only one node for a very long time, the other node may disconnect from the disk underneath that CSV. If you’re lucky, the CSV will just go into Redirected Access mode if you try to move a VM to it. Otherwise, the VM won’t be able to talk to its disk. This condition seems to be very rare, but it’s something to be aware of.
Multi-Path I/O (MPIO)
Multi-path I/O is a topology-neutral technology for establishing more than one connection from the same host to the same storage across unique pathways. Whew, big mouthful that is. The key words are: multiple paths, multiple pathways, and topology-neutral. It’s mostly talked about in terms of iSCSI, but it’s absolutely not limited to that. If you have two internal SAS controllers and your internal disks have two SAS connectors, you can use MPIO. If you have two controllers to connect to your external storage device and it has two or more connections, you can use MPIO for that.
If you have multiple connections established, but don’t use MPIO, Disk Management will show the disk twice. One of them will report: “Offline (The disk is offline because it has a redundant path with another device)”. If the disk is formatted, Server Manager’s disk console will show it twice, with one of them having Unknown for Status and Partition. Get-Disk is much more terrifying, giving one of the two disks a status of Failed. Only DISKPART is happy with it, but it will show zero-bytes of capacity on one disk. Interestingly enough, I wrote about fixing this on my personal blog over two years ago, and it is continually among my most-visited articles.
MPIO is in all flavors of Windows Server and in Hyper-V Server. No additional charge. Isn’t that nice? But, you have to install it. If you’re using the Add Roles and Features wizard, it’s on the Features page:
You do not need to restart after installing. You will need to restart after the first time assigning it to a disk system.
I’m only going to talk about iSCSI disks here because that’s the only use I’ve ever had for MPIO. Be aware that many vendors have their own method of setting up MPIO, so read the manual! What follows is for a generic iSCSI target.
After you’ve installed MPIO, run MPIOCPL.EXE at any command or PowerShell or Run prompt. As with ISCSICPL.EXE, this will work whether you are on a GUI server or not. The first tab you’ll see is MPIO Devices. It’s pretty much useless at the beginning. It will show an item named “Vendor 8Product 16”. This is a meaningless placeholder for developers to see how it’s done. What you want to do is flip over to the Discover Multi-Paths tab. Since we’re talking about iSCSI, tick that checkbox. If you’re here for SAS, tick that one. Then, click the Add button:
MPIO Discover Paths
Now, you’re going to have to reboot. When it comes back, everything will be fine.
The other tabs in the MPIO dialog box aren’t really that useful unless your vendor documentation indicates that they are.
In case you’re curious, no, you can’t set up MPIO path discovery before you make the connections to storage, nor can you avoid the reboot. You could opt to make only one connection from storage, add MPIO support, then add another connection, if you wish. But, that doesn’t really do anything for you, so I’d skip it.
By default, MPIO works in load-balanced configuration with round robin. That means it will send sequential requests down the other path. Because, just like any other I/O request, this can get imbalanced, you may wish to use a different load-balancing algorithm. On a local system, Disk Management allows you to modify MPIO settings on the property sheet of an MPIO disk. Unfortunately, Disk Management cannot set MPIO policy on a remote server. For a GUI-less or remote system, you can use PowerShell. If you must, MPCLAIM.EXE still works. For iSCSI disks, you can find an MPIO button on the Properties sheet for a disk in ISCSICPL.EXE (read the iSCSI sections above if you don’t know how to find this).
MPIO Disk Policy
When you select an option from the drop-down, this will show a details pane that explains the setting you’ve picked. Making a selection here configures the policy for all connections to that disk, but does not affect other connections to other disks.
This will be easy: as long as the share and NTFS permissions are configured properly, all you have to do is point Hyper-V to it.
SMB 3 Share Permissions
The really nice thing is that this is, by default, shared storage. It works right away for standalone and clustered systems alike.
Storage for a Hyper-V Cluster
Aside from SMB 3 storage, there are two specific ways to use storage in your cluster. The first, the Cluster Disk, is the oldest, and the one that works for every single application that can be covered by Microsoft Failover Clustering. A cluster disk can only be owned by a single node at any given time, and only that node can access the disk’s data. If another node, or any other system, tries to access it, the data will be corrupted. So, when a virtual machine on a cluster disk is Live Migrated to another node, disk ownership moves with it. For this reason, if you use a standard cluster disk for Hyper-V, only one virtual machine can be stored on it. This is because you can’t Live Migrate two virtual machines in precise lockstep.
Standard Cluster Disks
The first step is to connect each node to storage. For direct-connected systems, this is probably just a matter of plugging it in. For iSCSI, read above. For Fibre Channel, read your manufacturer’s instructions. For bringing it online and formatting it (as described in the Prepare a Local Disk for Usage section), do this on only one node (it doesn’t matter which). Leave it Offline everywhere else.
Once the disk is connected to all nodes, access Failover Cluster Manager, expand the Storage node, and click Disks. In the right pane, click Add Disk:
FCM Add Disk
A progress bar will show while storage is scanned on the nodes for disks that can be used in clustering: anything that’s not local storage, is formatted as NTFS or ReFS, is visible from all nodes, and is online on one node.
Be aware that the disk # on node # header might be the easiest way to identify which disk you’re actually looking at. The Capacity might be of assistance, as well. Select the disks you want in your cluster and click OK. The disk(s) will now be in the list of disks in Failover Cluster Manager. If you’re not especially taken with the name Cluster Disk 1, you can double-click it and change the name in the Properties sheet. You might also want to flip through the other tabs of this dialog, although most people won’t want to change it. You can now set about using the disk. Cluster disks are optimal for the disk witness in a quorum. Personally, I never put VMs on plain cluster disks. Instead, I convert them to CSV.
Cluster Shared Volumes
A Cluster Shared Volume (CSV) is only owned by one node at a time, but all nodes in a cluster have direct read/write access to CSVs, provided they have their own connection. If a node can’t directly connect to the storage underlying the CSV, its I/O is redirected through the node that currently owns the CSV. This is called Redirected Access Mode; the owner node is called the Coordinator Node. Metadata operations (file open, close, rename, etc.) are always handled through the coordinator node and transmitted throughout the cluster over the cluster network(s). Redirected I/O also moves over the cluster network(s). Despite a common misperception (likely due to some confusing early documentation on the subject), there is no dedicated CSV network. The benefit of the CSV is that multiple VMs can be stored on the same CSV.
In order to create a CSV, first follow the steps to create a standard cluster disk. Then, still in the Disks node of Storage in Failover Cluster Manager, right-click on the disk and click Add to cluster shared volumes. All done. As with the cluster disk, you can rename it by double-clicking it. You’ll notice there are fewer options for a CSV than for a standard cluster disk. The cluster really handles the care and feeding of its CSVs.
Your CSV is accessible through a Symbolic Link (I think it’s technically a Junction if refers to a folder, but that’s probably just trivia). If you look inside the ClusterStorage folder on the system drive of any node, you’ll find a “folder” named Volume1. This folder can be renamed, and you can put data in it. Don’t let the familiar view fool you though, this is actually a direct connection to the root of the volume that was a cluster disk just a few moments ago. When you create new virtual machines, target them to this folder, they’ll automatically be on highly available storage (but that doesn’t mean the VM is highly available — a discussion for another topic).
This post was devoted to the various ways you can connect to disk storage in current iterations of Windows and Hyper-V Server. In order to qualify, the disk must be formatted.
Our next, and final, article in this series will perform some performance testing.
Bringing a physical operating environment into Hyper-V can be a challenging task. It is recommended that you use some application-level migration rather than trying to convert a physical operating system installation directly. However, some systems do survive the transition well. One tool that can be used in conversion operations is Sysinternal’s Disk2VHD.
What is Disk2VHD?
Disk2VHD is a software solution provided by Sysinternals. It reads the boot information, partition information, and data regions of a physical hard drive and produces a corresponding VHD or VHDX file. It is very important to understand that this is not a true physical-to-virtual conversion. The operating system is not prepared to run inside a virtual environment, nor is any cleanup work done on the source system to improve the odds of a successful migration.
Microsoft Virtual Machine Converter 3.0
Before using this Disk2VHD to attempt a P2V, you might try Microsoft’s Virtual Machine Converter. It has been expanded to include P2V capability. The tool is provided free-of-charge. Visit the TechNet article for more information. The first link on that page will take you to the download.
Downloading and Installing Disk2VHD for Hyper-V P2V
Sysinternals (now owned by Microsoft) offers this tool as a free download from TechNet. Please read their overview and warnings. There is no actual “installation” of this product. Simply extract the “disk2vhd.exe” file from the ZIP archive and place it on the system whose drives you wish to convert. You can also extract the “Disk2vhd.chm” file; this is the software’s help manual.
Jeffery Hicks has also written a PowerShell script that will retrieve and update the entire Sysinternals catalog.
Pre-Flight Checks and Procedures
It’s a hard fact that physical to virtual conversions are fraught with peril. The older the operating system, the greater the risk. A few things that you should do to prepare:
- Perform a complete backup to a safe location.
- Verify that there aren’t any problems with the system. Check event logs, etc. Run CHKDSK.
- Remove any unnecessary software and drivers, but do not make any changes to the system’s vital configuration that cannot be easily redone. If the P2V fails, you don’t want to be left with an unusable source system. RAID drivers are a common paint point but removing them will render your system unusable.
- Optional: Remove the system from the domain. This will avoid post-conversion issues.
- Stop as many services as possible, especially any that run databases. Disk2VHD essentially captures an application-consistent backup, but it will have superior results if all applications are closed when it runs. This also prevents anyone from using those services. Anything that happens to the system after the Disk2VHD process begins will be lost.
There are two ways to use the Disk2VHD tool for P2V in Hyper-V. The first is in interactive GUI mode. The second is via command-line, which is useful if you need to use scripting for such things as handling large batches.
Before starting, shut down all applications and stop any vital services especially if the machine to be converted runs any databases. I/O is paused for the conversion (if you choose the option for VSS) but many applications won’t know to flush active buffers.
Using Disk2VHD in GUI Mode
Copy the “disk2vhd.exe” file to the computer whose drive(s) is/are to be converted and double-click on it from that computer. You’ll be presented with a screen similar to the following:
Disk2Vhd Main Screen
That screenshot was taken from a Windows 2012 R2 computer; some older operating systems will not have the “System Reserved” partition. If the system that you’re creating a VHDX from has this partition, you’ll need to include it if you wish for the VHDX to be bootable.
You’ll first need to select a destination in the “VHD File Name” text box. The button at the end of that line with the triple dots serves as a browse button. If keeping the option for VSS snapshots, it’s perfectly acceptable to create the VHDX(s) on the same drive that you’re copying as long as there is enough room.
Unless you know that you’ll be using the generated file on Windows/Hyper-V Server 2008 R2 or earlier, check the box to Use Vhdx. It does not matter if the guest operating system is older than 2012.
Once you’re satisfied with all of your selections, click the “Create” button. The software will first indicate that it is creating the VSS snapshot. After that, it will begin the process of copying out the data. If you make any changes to the drive’s data after that, they will not be saved. You can continue to use the system during the copy process.
The primary purpose of using the command-line version would be if you need to script this. For instance, if you have purchased Software Assurance for all of your Windows 7 desktops and want to move them into a VDI environment, you could craft a specialized computer login script that relied on “Disk2vhd” to capture the physical Windows 7 systems in your organization.
The help file shows the usage syntax: disk2vhd [-h] <[drive: [drive:]…]|[*]> <vhdfile>
There is one optional setting and two required ones.
|If the drive to be converted is attached to a Windows XP or Windows Server 2003 system, this prepares its hardware abstraction layer
Drive: or *
|You can enter drive letter designators separated by spaces OR you can use an asterisk to indicate all drives. You must use the asterisk for the “System Reserved” partition to be included.
|Enter the full path and file name of the VHD[X] to be created. UNCs are acceptable.
Both methods will result in the same type of output: one VHD file will be made for each disk in the source. If there is only one VHD to be created, it will be exactly the file name you specified. If there is more than one, each VHD will be the filename you specified with a hyphen and the disk number appended.
Sample drive layout:
With all drives checked, running the above system through Disk2VHD produces:
Demo-0.VHD contains the System Reserved and C: volumes, Demo-1.VHD contains the D: volume, and Demo-2.VHD contains the E: and F: volumes.
Test the VHDX
On a system with Hyper-V 2012 or later installed, run the following, substituting in your actual path:
Test-VHD -Path C:\LocalVMs\Virtual Hard Disks\disk2vhded.vhdx
This testing isn’t overly comprehensive, but it can identify any immediate red flags that would prevent it from being usable with Hyper-V.
Create the Virtual Machine
All that the Disk2VHD Hyper-V P2V tool does is prepare the VHDX files. It does not create a virtual machine. You’ll need another tool for that. The following steps will demonstrate using Hyper-V Manager to create the VM using the VHDXs that were output by Disk2VHD in the steps above.
- Copy the created VHD files to a location that your Hyper-V host can access them. If you need some guidance on this, open Hyper-V Manager and ensure that the host you want to work with is selected. In the right pane, click on “Hyper-V Settings…” In the dialog box, click on “Virtual Hard Disks” in the “Server” section. The right pane of the dialog will show the folder that the host is currently set to access VHDs from. It isn’t required that you place them in that folder, but you can be certain that the Hyper-V host will be able to access VHDs there.
- Once you have placed the files where you want them, go to the Actions pane at the right. Click New, then Virtual Machine to start the wizard.
New VM: Start the Wizard
- The first page of the New Virtual Machine Wizard is purely informational. Click Next when ready.
- The first data screen asks for the name of the virtual machine and the initial location. The name that you choose will be used as a label in all virtual machine tools, including Hyper-V Manager, Failover Cluster Manager, and PowerShell. The name of the guest operating system that you’ll eventually install will not be automatically set to this, and does not need to be the same.
If you do not check the option to Store the virtual machine in a different location, then all of the virtual machine’s metadata (XML, BIN, VSV, and second-level paging files) will all be placed in the host’s default location. This path will be visible in the grayed out text box. Checking the box allows you to override this location. If it’s never held a virtual machine before, a subfolder named “Virtual Machines” will be created in the path that you specified. Whether that folder is new or existing, another subfolder will be created underneath it that will have the name of the automatically-generated GUID that represents the virtual machine. Security will automatically be configured for you.
New VM: Name and Location
- On the next screen (2012 and later), you’ll be asked to choose the Generation of the virtual machine. What you choose here will depend on the source physical machine. If the source ran in the traditional BIOS mode, you must choose Generation 1. If the source was operating in the newer UEFI mode, you must choose Generation 2.
New VM: Generation
- On the next screen, you set the preliminary memory settings for the virtual machine. You only have two options: the amount of Startup memory and whether or not you wish to Use Dynamic Memory for this virtual machine. If you do not wish to use Dynamic Memory, leave the box unchecked and set the startup memory to the amount of memory that you wish the virtual machine to have. If you do wish to use Dynamic Memory, you’ll have to configure the minimums and maximums later. They will be automatically set to 512 and 1TB, respectively.
New VM: Memory Settings
- Next, you’ll need to determine how to connect the virtual network adapter. You can leave it at Not Connected, which means the virtual machine will begin its life without any network connectivity. Otherwise, you can connect it to a virtual switch right away. Be aware that you can’t set any advanced options such as the VLAN here. Whatever you choose, this adapter is always the newer synthetic adapter, which cannot be used to PXE boot a Generation 1 virtual machine. If you need that functionality, it is granted on a later screen.
New VM: Network Connection
- After networking, you’ll be asked to configure the first virtual hard disk for the virtual machine. The second option allows you to connect an existing virtual hard disk. Browse to the VHDX file that you created with Disk2VHD:
- The final screen is simply a confirmation and your last opportunity to go back and make any necessary changes.
New VM Confirmation
- Once you click Finish, the wizard undertakes all of the options that you selected and creates the virtual machine. The wizard does not turn the virtual machine on, so you have an opportunity to make any desired modifications beforehand, such as to vCPU count or memory allocation.
- Start the VM in Hyper-V Manager. The operating system will go through a hardware detection phase; how problematic this is will depend on the operating system and the hardware involved. Also, the VHDs have been left in a “crash-consistent” state, meaning that they will act as though the machine they are attached to was shut down improperly. This might not mean anything more than answering the “Shutdown Event Tracker” dialog and perhaps restarting once or twice to finalize hardware changes, but you’ll want to validate any application data just to be certain.
(note: the comment was added for the screenshot, not by the Disk2VHD software)
Test your new VM thoroughly. Some software that requires hardware-based activation may need to be reactivated. Some software may be hardware-dependent and won’t run in a virtual environment. If you didn’t connect the network adapter during VM creation, you’ll need to do so before attempting to utilize it. If you are working with a machine that is a domain member, be warned that there will be a name collision; the typical approach is to remove the new VM from the domain and then rename it prior to attaching it to the network. Once it is able to communicate with a domain controller, re-add it to the domain.