How to Perform Hyper-V Storage Migration

How to Perform Hyper-V Storage Migration

New servers? New SAN? Trying out hyper-convergence? Upgrading to Hyper-V 2016? Any number of conditions might prompt you to move your Hyper-V virtual machine’s storage to another location. Let’s look at the technologies that enable such moves.

An Overview of Hyper-V Migration Options

Hyper-V offers numerous migration options. Each has its own distinctive features. Unfortunately, we in the community often muck things up by using incorrect and confusing terminology. So, let’s briefly walk through the migration types that Hyper-V offers:

  • Quick migration: Cluster-based virtual machine migration that involves placing a virtual machine into a saved state, transferring ownership to another node in the same cluster, and resuming the virtual machine. A quick migration does not involve moving anything that most of us consider storage.
  • Live migration: Cluster-based virtual machine migration that involves transferring the active state of a running virtual machine to another node in the same cluster. A Live Migration does not involve moving anything that most of us consider storage.
  • Storage migration: Any technique that utilizes the Hyper-V management service to relocate any file-based component that belongs to a virtual machine. This article focuses on this migration type, so I won’t expand any of those thoughts in this list.
  • Shared Nothing Live Migration: Hyper-V migration technique between two hosts that does not involve clustering. It may or may not include a storage migration. The virtual machine might or might not be running. However, this migration type always includes ownership transfer from one host to another.

It Isn’t Called Storage Live Migration

I have always called this operation “Storage Live Migration”. I know lots of other authors call it “Storage Live Migration”. But, Microsoft does not call it “Storage Live Migration”. They just call it “Storage Migration”. The closest thing that I can find to “Storage Live Migration” in anything from Microsoft is a 2012 TechEd recording by Benjamin Armstrong. The title of that presentation includes the phrase “Live Storage Migration”, but I can’t determine if the “Live” just modifies “Storage Migration” or if Ben uses it as part of the technology name. I suppose I could listen to the entire hour and a half presentation, but I’m lazy. I’m sure that it’s a great presentation, if anyone wants to listen and report back.

Anyway, does it matter? I don’t really think so. I’m certainly not going to correct anyone that uses that phrase. However, the virtual machine does not necessarily need to be live. We use the same tools and commands to move a virtual machine’s storage whether it’s online or offline. So, “Storage Migration” will always be a correct term. “Storage Live Migration”, not so much. However, we use the term “Shared Nothing Live Migration” for virtual machines that are turned off, so we can’t claim any consistency.

What Can Be Moved with Hyper-V Storage Migration?

When we talk about virtual machine storage, most people think of the places where the guest operating system stores its data. That certainly comprises the physical bulk of virtual machine storage. However, it’s also only one bullet point on a list of multiple components that form a virtual machine.

Independently, you can move any of these virtual machine items:

  • The virtual machine’s core files (configuration in xml or .vmcx, .bin, .vsv, etc.)
  • The virtual machine’s checkpoints (essentially the same items as the preceding bullet point, but for the checkpoint(s) instead of the active virtual machine)
  • The virtual machine’s second-level paging file location. I have not tested to see if it will move a VM with active second-level paging files, but I have no reason to believe that it wouldn’t
  • Virtual hard disks attached to a virtual machine
  • ISO images attached to a virtual machine

We most commonly move all of these things together. Hyper-V doesn’t require that, though. Also, we can move all of these things in the same operation but distribute them to different destinations.

What Can’t Be Moved with Hyper-V Storage Migration?

In terms of storage, we can move everything related to a virtual machine. But, we can’t move the VM’s active, running state with Storage Migration. Storage Migration is commonly partnered with a Live Migration in the operation that we call “Shared Nothing Live Migration”. To avoid getting bogged down in implementation details that are more academic than practical, just understand one thing: when you pick the option to move the virtual machine’s storage, you are not changing which Hyper-V host owns and runs the virtual machine.

More importantly, you can’t use any Microsoft tool-based technique to separate a differencing disk from its parent. So, if you have an AVHDX (differencing disk created by the checkpointing mechanism) and you want to move it away from its source VHDX, Storage Migration will not do it. If you instruct Storage Migration to move the AVHDX, the entire disk chain goes along for the ride.

Uses for Hyper-V Storage Migration

Out of all the migration types, storage migration has the most applications and special conditions. For instance, Storage Migration is the only Hyper-V migration type that does not always require domain membership. Granted, the one exception to the domain membership rule won’t be very satisfying for people that insist on leaving their Hyper-V hosts in insecure workgroup mode, but I’m not here to please those people. I’m here to talk about the nuances of Storage Migration.

Local Relocation

Let’s start with the simplest usage: relocation of local VM storage. Some situations in this category:

  • You left VMs in the default “C:\ProgramData\Microsoft\Windows\Hyper-V” and/or “C:\Users\Public\Documents\Hyper-V\Virtual Hard Disks” locations and you don’t like it
  • You added new internal storage as a separate volume and want to re-distribute your VMs
  • You have storage speed tiers but no active management layer
  • You don’t like the way your VMs’ files are laid out
  • You want to defragment VM storage space. It’s a waste of time, but it works.

Network Relocation

With so many ways to do network storage, it’s nearly a given that we’ll all need to move a VHDX across ours at some point. Some situations:

  • You’re migrating from local storage to network storage
  • You’re replacing a SAN or NAS and need to relocate your VMs
  • You’ve expanded your network storage and want to redistribute your VMs

Most of the reasons listed under “Local Relocation” can also apply to network relocation.

Cluster Relocation

We can’t always build our clusters perfectly from the beginning. For the most part, a cluster’s relocation needs list will look like the local and network lists above. A few others:

  • Your cluster has new Cluster Shared Volumes that you want to expand into
  • Existing Cluster Shared Volumes do not have a data distribution that does not balance well. Remember that data access from a CSV owner node is slightly faster than from a non-owner node

The reasons matter less than the tools when you’re talking about clusters. You can’t use the same tools and techniques to move virtual machines that are protected by Failover Clustering under Hyper-V as you use for non-clustered VMs.

Turning the VM Off Makes a Difference for Storage Migration

You can perform a very simple experiment: perform a Storage Migration for a virtual machine while it’s on, then turn it off and migrate it back. The virtual machine will move much more quickly while it’s off. This behavior can be explained in one word: synchronization.

When the virtual machine is off, a Storage Migration is essentially a monitored file copy. The ability of the constituent parts to move bits from source to destination sets the pace of the move. When the virtual machine is on, all of the rules change. The migration is subjected to these constraints:

  • The virtual machine’s operating system must remain responsive
  • Writes must be properly captured
  • Reads must occur from the most appropriate source

Even if the guest operating does not experience much activity during the move, that condition cannot be taken as a constant. In other words, Hyper-V needs to be ready for it to start demanding lots of I/O at any time.

So, the Storage Migration of a running virtual machine will always take longer than the Storage Migration of a virtual machine in an off or saved state. You can choose the convenience of an online migration or the speed of an offline migration.

Note: You can usually change a virtual machine’s power state during a Storage Migration. It’s less likely to work if you are moving across hosts.

How to Perform Hyper-V Storage Migration with PowerShell

The nice thing about using PowerShell for Storage Migration: it works for all Storage Migration types. The bad thing about using PowerShell for Storage Migration: it can be difficult to get all of the pieces right.

The primary cmdlet to use is Move-VMStorage. If you will be performing a Shared Nothing Live Migration, you can also use Move-VM. The parts of Move-VM that pertain to storage match Move-VMStorage. Move-VM has uses, requirements, and limitations that don’t pertain to the topic of this article, so I won’t cover Move-VM here.

A Basic Storage Migration in PowerShell

Let’s start with an easy one. Use this when you just want all of a VM’s files to be in one place:

This will move the virtual machine named testvm so that all of its components reside under the C:\LocalVMs folder. That means:

  • The configuration files will be placed in C:\LocalVMs\Virtual Machines
  • The checkpoint files will be placed in C:\LocalVMs\Snapshots
  • The VHDXs will be placed in C:\LocalVMs\Virtual Hard Disks
  • Depending on your version, an UndoLog Configuration folder will be created if it doesn’t already exist. The folder is meant to contain Hyper-V Replica files. It may be created even for virtual machines that aren’t being replicated.

Complex Storage Migrations in PowerShell

For more complicated move scenarios, you won’t use the DestinationStoragePath parameter. You’ll use one or more of the individual component parameters. Choose from the following:

  • VirtualMachinePath: Where to place the VM’s configuration files.
  • SnapshotFilePath: Where to place the VM’s checkpoint files (again, NOT the AVHDXs!)
  • SmartPagingFilePath: Where to place the VM’s smart paging files
  • Vhds: An array of hash tables that indicate where to place individual VHD/X files.

Some notes on these items:

  • You are not required to use all of these parameters. If you do not specify a parameter, then its related component is left alone. Meaning, it doesn’t get moved at all.
  • If you’re trying to use this to get away from those auto-created Virtual Machines and Snapshots folders, it doesn’t work. They’ll always be created as sub-folders of whatever you type in.
  • It doesn’t auto-create a Virtual Hard Disks folder.
  • If you were curious whether or not you needed to specify those auto-created subfolders, the answer is: no. Move-VMStorage will always create them for you (unless they already exist).
  • The VHDs hash table is the hardest part of this whole thing. I’m usually a PowerShell-first kind of guy, but even I tend to go to the GUI for Storage Migrations.

The following will move all components except VHDs, which I’ll tackle in the next section:

Move-VMStorage’s Array of Hash Tables for VHDs

The three …FilePath parameters are easy: just specify the path. The Vhds parameter is tougher. It is one or more hash tables inside an array.

First, the hash tables. A hash table is a custom object that looks like an array, but each entry has a unique name. The hash tables that Vhds expects have a SourceFilePath entry and a DestinationFilePath entry. Each must be fully-qualified for a file. A hash table is contained like this: @{ }. The name of an entry and its value are joined with an =. Entries are separated by a ; So, if you want to move the VHDX named svtest.vhdx from \\svstore\VMs to C:\LocalVMs\testvm, you’d use this hash table:

Reading that, you might ask (quite logically): “Can I change the name of the VHDX file when I move it?” The answer: No, you cannot. So, why then do you need to enter the full name of the destination file? I don’t know!

Next, the arrays. An array is bounded by @( ). Its entries are separated by commas. So, to move two VHDXs, you would do something like this:

I broke that onto multiple lines for legibility. You can enter it all on one line. Note where I used parenthesis and where I used curly braces.

Tip: To move a single VHDX file, you don’t need to do the entire array notation. You can use the first example with Vhds.

A Practical Move-VMStorage Example with Vhds

If you’re looking at all that and wondering why you’d ever use PowerShell for such a thing, I have the perfect answer: scripting. Don’t do this by hand. Use it to move lots of VMs in one fell swoop. If you want to see a plain example of the Vhds parameter in action, the Get-Help examples show one. I’ve got a more practical script in mind.

The following would move all VMs on the host. All of their config, checkpoint, and second-level paging files will be placed on a share named “\\vmstore\slowstorage”. All of their VHDXs will be placed on a share named “\\vmstore\faststorage”. We will have PowerShell deal with the source paths and file names.

I used splatting for the parameters for two reasons: 1, legibility. 2, to handle VMs without any virtual hard disks.

How to Perform Hyper-V Storage Migration with Hyper-V Manager

Hyper-V Manager can only be used for non-clustered virtual machines. It utilizes a wizard format. To use it to move a virtual machine’s storage:

  1. Right-click on the virtual machine and click Move.
  2. Click Next on the introductory page.
  3. Change the selection to Move the virtual machine’s storage (the same storage options would be available if you moved the VM’s ownership, but that’s not part of this article)
  4. Choose how to perform the move. You can move everything to the same location, you can move everything to different locations, or you can move only the virtual hard disks.
  5. What screens you see next will depend on what you chose. We’ll cover each branch.

If you opt to move everything to one location, the wizard will show you this simple page:


If you choose the option to Move the virtual machine’s data to different locations, you will first see this screen:


For every item that you check, you will be given a separate screen where you indicate the desired location for that item. The wizard uses the same screen for these items as it does for the hard-disks only option. I’ll show its screen shot next.

If you choose Move only the virtual machine’s virtual hard disks, then you will be given a sequence of screens where you instruct it where to move the files. These are the same screens used for the individual components from the previous selection:


After you make your selections, you’ll be shown a summary screen where you can click Finish to perform the move:


How to Perform Hyper-V Storage Migration with Failover Cluster Manager

Failover Cluster Manager uses a slick single-screen interface to move storage for cluster virtual machines. To access it, simply right-click a virtual machine, hover over Move, and click Virtual Machine Storage. You’ll see the following screen:


If you just want to move the whole thing to one of the display Cluster Shared Volumes, just drag and drop it down to that CSV in the Cluster Storage heading at the lower left. You can drag and drop individual items or the entire VM. The Destination Folder Path will be populated accordingly.

As you can see in mine, I have all of the components except the VHD on an SMB share. I want to move the VHD to be with the rest. To get a share to show up, click the Add Share button. You’ll get this dialog:


The share will populate underneath the CSVs in the lower left. Now, I can drag and drop that file to the share. View the differences:


Once you have the dialog the way that you like it, click Start.

Confusing Terms and Concepts in Hyper-V

Confusing Terms and Concepts in Hyper-V



If I ever got a job at Microsoft, I’d want my title to be “He Who Fixes Stupid Names and Labels”. Depending upon my mood, I envision that working out in multiple ways. Sometimes, I see myself in a product meeting with someone locked in a condescending glare, and asking, “Really? With nearly one million unique words in the English language, we’re going with ‘Core’? Again?” Other times, I see myself stomping around like Gordon Ramsay, bellowing, “This wording is so unintelligible that it could be a plot for a Zero Wing sequel!” So, now you know one of the many reasons that I don’t work for Microsoft. But, the degree of my fitness to work in a team aside, the abundance of perplexing aspects of the Hyper-V product generates endless confusion for newcomers. I’ve compiled a shortlist to help cut through a few of them.


This particular item doesn’t have a great deal of relevance to Hyper-V for most of us. On the back end, there is a great deal of intersection in the technologies. Site Recovery allows you to replicate your on-premises virtual machines into Azure. But, there’s not a lot of confusion about the technology that I’m aware of. It’s listed here, and first, as an example of what we’re up against. Think about what the word “azure” means. It is the color of a clear, cloudless sky. You think one thing when a salesman walks in and says, “Hi, we’d like to introduce you to our cloud product called ‘Azure’.” That sounds nice, right? What if, instead, he said, “Hi, we’d like to introduce you to our cloud product called ‘Cloudless’.” What?

"Microsoft Drab" Just Doesn't have the Same Ring

“Microsoft Drab” Just Doesn’t have the Same Ring

Azure’s misnomer appears to be benign, as it works very well and sells very well. I just want you to be aware that, if you’re confused when reading a product label or a dialog box, it’s probably not your fault. Microsoft doesn’t appear to invest many resources in the “Thoroughly Think Through Labelling” department.

What Should I Call Hyper-V, Anyway?

Some of the confusion kicks in right at the beginning. Most people know that Hyper-V is Microsoft’s hypervisor, which is good. But, then they try to explain what they’re using, and everything immediately goes off the rails.

First, there’s Hyper-V. That part, we all understand. Or, at least we think that we understand. When you just use the word “Hyper-V”, that’s just the hypervisor. It’s completely independent of how you acquired or installed or use the hypervisor. It applies equally to Hyper-V Server and Windows Server with Hyper-V.

Second, there’s Client Hyper-V. It’s mostly Hyper-V, but with different bells and whistles. You only find Client Hyper-V in the client editions of Windows, conveniently enough. So, if you’ve installed some product whose name includes the word “Server”, then you are not using Client Hyper-V. Simple enough, right?

Third, there’s the fictitious “Hyper-V Core”. I’ve been trying to get people to stop saying this for years, but I’m giving up now. Part of it is that it’s just not working. Another part of it:


With Microsoft actively working against me, I don’t like my odds. Sure, they’ve cleaned up a lot of these references, but I suspect that they’ll never completely go away.

What I don’t like about the label/name “Hyper-V Core” is that it implies the existence of “Hyper-V not Core”. Therefore, people download Hyper-V Server and want to know why it’s all command-line based. People will also go to the forums and ask for help with “Hyper-V Core”, so then there’s at least one round of, “What product are you really using?”

What Does it Mean to “Allow management operating system to share this network adapter”?

The setting in question appears on the Virtual Switch Manager’s dialog when you create a virtual switch in Hyper-V Manager:


The corresponding PowerShell parameter for New-VMSwitch is AllowManagementOs.

If I had that job that we were talking about a bit ago, that Hyper-V Manager line would say, “Connect the management operating system to this virtual switch.” The PowerShell parameter would be ConnectManagementOs. Then the labels would be true, explainable, and comprehensible.

Whether you choose the Hyper-V Manager path or the PowerShell route, this function creates a virtual network adapter for the management operating system and attaches it to the virtual switch that you’re creating. It does not “share” anything, at least not in any sense that this phrasing evokes. For more information, we have an article that explains the Hyper-V virtual switch.

I Downloaded and Installed Hyper-V. Where Did My Windows 7/8/10 Go?

I see this question often enough to know that there are a significant number of people that encounter this problem. The trainer in me must impart a vital life lesson: If the steps to install a product include anything like “boot from a DVD or DVD image”, then it is making substantial and potentially irreversible changes.

If you installed Hyper-V Server, your original operating environment is gone. You may not be out of luck, though. If you didn’t delete the volume, then your previous operating system is in a folder called “Windows.old”. Don’t ask me or take this to the Hyper-V forums, though, because this is not a Hyper-V problem. Find a forum for the operating system that you lost and ask how to recover it from the Windows.old folder. There are no guarantees.

Many of the people that find themselves in this position claim that Microsoft didn’t warn them, which is absolutely not true.

The first warning occurs if you attempt to upgrade. It prevents you from doing so and explicitly says what the only other option, “Custom”, will do:


If you never saw that because you selected Custom first, then you saw this warning:


That warning might be a bit too subtle, but you had another chance. After choosing Custom, you then decided to either install over the top of what you had or delete a partition. Assuming that you opted to use what was there, you saw this dialog:


The dialog could use some cleanup to cover the fact that it might have detected something other than a previous installation of Hyper-V Server, but there’s a clear warning that something new is pushing out something old. If you chose to delete the volume so that you could install Hyper-V Server on it, that warning is inescapably blatant:


If this has happened to you, then I’m sorry, but you were warned. You were warned multiple times.

How Many Hyper-V Virtual Switches Should I Use?

I often see questions in this category from administrators that have VMware experience. Hyper-V’s virtual switch is markedly different from what VMware does, so you should not expect a direct knowledge transfer.

The default answer to this question is always “one”. If you’re going to be putting your Hyper-V hosts into a cluster, that strengthens the case for only one. A single Hyper-V virtual switch performs VLAN isolation and identifies local MAC addresses to prevent superfluous trips to the physical network for intra-VM communications. So, you rarely gain anything from using two or more virtual switches. We have a more thorough article on the subject of multiple Hyper-V switches.

Checkpoint? Snapshot? Well, Which Is it?

To save time, I’m going to skip definitions here. This is just to sort out the terms. A Hyper-V checkpoint is a Hyper-V snapshot. They are not different. The original term in Hyper-V was “snapshot”. That caused confusion with the Volume Shadow Copy Service (VSS) snapshot. Hyper-V’s daddy, “Virtual Server”, used the term “checkpoint”. System Center Virtual Machine Manager has always used the term “checkpoint”. The “official” terms have been consolidated into “checkpoint”. You’ll still find many references to snapshots, such as:


But We Officially Don’t Say “Snapshot”

We writers are looking forward to many more years of saying “checkpoint (or snapshot)”.

Do I Delete a Checkpoint? Or Merge It? Or Apply It? Or Something Else? What is Going on Here?

If you’re the person that developed the checkpoint actions, all of these terms make a lot of sense. If you’re anyone else, they’re an unsavory word soup.

  • Delete: “Delete” is confusing because deleting a checkpoint keeps your changes. Coming into this cold, you might think that deleting a checkpoint would delete changes. Just look under the hood, though. When you create a checkpoint, it makes copies of the virtual machine’s configuration files and starts using new ones. When you delete that checkpoint, that tells Hyper-V to delete the copies of the old configuration. That makes more sense, right? Hyper-V also merges the data in post-checkpoint differencing disks back into the originals, then deletes the differencing disks.
  • Merge (checkpoint): When you delete a checkpoint (see previous bullet point), the differencing disks that were created for its attached virtual hard disks are automatically merged back into the original. You can’t merge a checkpoint, though. That’s not a thing. That can’t be a thing. How would you merge a current VM with 2 vCPUs and its previous setting with 4 vCPUs? Split the difference? Visitation of 2 vCPUs every other weekend?
  • Merge (virtual hard disk): First, make sure that you understand the previous bullet point. If there’s a checkpoint, you want to delete it and allow that process to handle the virtual hard disk merging on your behalf. Otherwise, you’ll bring death and pestilence. If the virtual hard disk in question is not related to a checkpoint but still has a differencing disk, then you can manually merge them.
  • Apply: The thought process behind this term is just like the thinking behind Delete. Remember those copies that your checkpoint made? When you apply the checkpoint, the settings in those old files are applied to the current virtual machine. That means that applying a checkpoint discards your changes. As for the virtual hard disks, Hyper-V stops using the differencing disk that was created when the virtual machine was checkpointed and starts using a new differencing disk that is a child of the original virtual hard disk. Whew! Get all of that?
  • Revert: This verb makes sense to everyone, I think. It reverts the current state of the virtual machine to the checkpoint state. Technologically, Hyper-V applies the settings from the old files and discards the differencing disk. It creates a new, empty differencing disk and starts the virtual machine from it. In fact, the only difference between Revert and Apply is the opportunity to create another checkpoint to hold the changes that you’re about to lose. If I had that job, there would be no Apply. There would only be Revert (keep changes in a new checkpoint) and Revert (discard changes).

If this is tough to keep straight, it might make you feel better to know that my generation was expected to remember that Windows boots from the system disk to run its system from the boot disk. No one has ever explained that one to me. When you’re trying to keep this checkpoint stuff straight, just try to think of it from the perspective of the files that constitute a checkpoint.

If you want more information on checkpoints, I happen to like one of my earlier checkpoint articles. I would also recommend searching the blog on the “checkpoint” keyword, as we have many articles written by myself and others.

Dynamic Disks and Dynamically Expanding Virtual Hard Disks

“Dynamically expanding virtual hard disk” is a great big jumble of words that nobody likes to say. So, almost all of us shorten it to “dynamic disk”. Then, someone sees that the prerequisites list for the product that they want to use says, “does not support dynamic disks”. Panic ensues.

Despite common usage, these terms are not synonymous.

With proper planning and monitoring, dynamically expanding hard disks are perfectly safe to use.

Conversely, Dynamic disks are mostly useless. A handful of products require them, but hopefully they’ll all die soon (or undergo a redesign, that could work too). In the absence of an absolute, defined need, you should never use Dynamic disks. The article linked in the previous paragraph explains the Dynamic disk, if you’re interested. For a quicker explanation, just like at this picture from Disk Management:

Basic and Dynamic Disks

Basic and Dynamic Disks

Dynamic disks, in the truest sense of the term, are not a Hyper-V technology.

Which Live Migration Do I Want?

I was attempting to answer a forum question in which the asker was configuring Constrained Delegation so that he could Live Migrate a virtual machine from one physical cluster node to another physical node in the same cluster. I rightly pointed out that nodes in the same cluster do not require delegation. It took a while for me to understand that he was attempting to perform a Shared Nothing Live Migration of an unclustered guest between the two nodes. That does require delegation in some cases.

To keep things straight, understand that Hyper-V offers multiple virtual machine migration technologies. Despite all of them including the word “migration” and most of them including the word “live”, they are different. They are related because they all move something Hyper-V, but they are not interchangeable terms.

This is the full list:

  • Quick Migration: Quick Migration moves a virtual machine from one host to another within a cluster. So it’s said, the virtual machine must be clustered, not simply on a cluster node. It is usually the fastest of the migration techniques because nothing is transmitted across the network. If the virtual machine is on, it is first saved. Ownership is transferred to the target node. If the virtual machine was placed in a saved state for the move, it is resumed.
  • Live Migration: A Live Migration has the same requirement as a Quick Migration: it is only applicable to clustered virtual machines. Additionally, the virtual machine must be turned on (otherwise, it wouldn’t be “live”). Live Migration is slower than Quick Migration because CPU threads, memory, and pending I/O must be transferred to the target host, but it does not involve an interruption in service. The virtual machine experiences no outage except for the propagation of its network adapters’ MAC address change throughout the network.
  • Storage Live Migration: A Storage Live Migration involves the movement of any files related to a virtual machine. It could be all of them, or it could be any subset. “Storage Live Migration” is just a technology name; the phrase never appears anywhere in any of the tools. You select one of the options to “Move” and then you choose to only move storage. You can choose a new target location on the same host or remote storage, but a Storage Live Migration by itself cannot change a virtual machine’s owner to a new physical host. Unlike a “Live Migration”, the “Live” in “Storage Live Migration” is optional.
  • Shared Nothing Live Migration: The “Shared Nothing” part of this term can cause confusion because it isn’t true. The “live” bit doesn’t help, because the VM can be off or saved, if you want. The idea is that the source and destination hosts don’t need to be in the same cluster, so they don’t need to share a common storage pool. Their hosts do need to share a domain and at least one network, though. I’m not sure what I would have called this one, so maybe I’m glad that I don’t have that job. Anyway, as with Storage Live Migration, you’ll never see this phrase in any of the tools. It’s simply one of the “move” options.

If you’re seeking help from others, it’s important to use the proper term. Otherwise, your confusion will become their confusion and you might never find any help.

What Else?

I’ve been doing this long enough that I might be missing other things that just don’t make sense. Let us know what’s boggled you about Hyper-V and we’ll add it to the list.

Hyper-V Differencing Disks Explained

Hyper-V Differencing Disks Explained


Usually when we talk about Hyper-V’s virtual disk types, we focus on fixed and dynamically expanding. There’s another type that enjoys significantly less press: differencing disks. Administrators don’t deal directly with differencing disks as often as they work with the other two types, but they are hardly rare. Your Hyper-V knowledge cannot be complete without an understanding of the form and function of differencing disks, so let’s take a look.

What are Hyper-V Differencing Disks?

A differencing disk contains block data that represents changes to a parent virtual hard disk. The salient properties of differencing disks are:

  • A differencing disk must have exactly one parent. No more, no less.
  • The parent of a differencing disk must be another virtual hard disk. You cannot attach them to pass-through disks, a file system, a LUN, a remote share, or anything else.
  • The parent of a differencing disk can be any of the three types (fixed, dynamically expanding, or differencing)
  • Any modification to the data of the parent of a differencing disk effectively orphans the differencing disk, rendering it useless
  • Hyper-V can merge the change data back into the parent, destroying the differencing disk in the process. For Hyper-V versions past 2008 R2, this operation can take place while the disk is in use

Typically, differencing disks are small. They can grow, however. They can grow to be quite large. The maximum size of a differencing disk is equal to the maximum size of the root parent. I say “root” because, even though a differencing disk can be the parent of another differencing disk, there must be a non-differencing disk at the very top for any of them to be useful. Be aware that a differencing disk attached to a dynamically expanding disk does have the potential to outgrow its parent, if that disk isn’t fully expanded.

How Do Differencing Disks Work?

The concept behind the functioning of a differencing disk is very simple. When Hyper-V needs to write to a virtual disk that has a differencing child, the virtual disk driver redirects the write into a differencing disk. It tracks which block(s) in the original file were targeted and what their new contents would have been.

Differencing Disk Write

Differencing Disk Write


The most important thing to understand is that the virtual disk driver makes a choice to write to the differencing disk. The file itself is not marked read-only. You cannot scan the file and discover that it has a child. The child knows who its parent is, but that knowledge is not reciprocated.

Writes are the hard part to understand. If you’ve got that down, then reads are easy to understand. When the virtual machine requests data from its disk, the virtual disk driver first checks to see if the child has a record of the requested block(s). If it does, then the child provides the data for the read. If the child does not have a record of any changes to the block(s), the virtual disk driver retrieves them from the parent.

This is a Hyper-V blog, so I mostly only talk about Hyper-V. However, the virtual disk driver is part of the Windows operating system. The normal tools that you have access to in Windows without Hyper-V cannot create a differencing disk, but you can mount one as long as its parent is present.

How are Differencing Disks Created?

Unlike fixed and dynamically expanding virtual hard disks, you don’t simply kick off a wizard and create a differencing disk from scratch. In fact, most Hyper-V administrators will never directly create a differencing disk at all. There are four generic methods by which differencing disks are created.

Backup Software

For most of us, backup software is the most likely source of differencing disks. When a Hyper-V aware backup application targets a virtual machine, Hyper-V will take a special checkpoint. While the disk and the state of the virtual machine are frozen in the checkpoint, the backup application can copy the contents without fear that they’ll change. When the backup is complete, Hyper-V deletes the checkpoint and merges the differencing disk that it created back into its parent. If it doesn’t, you have a problem and will need to talk to your backup vendor.

Note: backup software operations will always create differencing disks in the same location as the parent. You cannot override this behavior!

Standard and Production Checkpoints

Standard and Production Checkpoints are created by administrators, either manually or via scripts and other automated processes. As far as the disks are concerned, there isn’t much difference between any of the checkpoint types. Unlike backup checkpoints, Hyper-V will not automatically attempt to clean up standard or production checkpoints. That’s something that an administrator must do, also manually or via scripts and other automated processes.

Note: checkpoint operations will always create differencing disks in the same location as the parent. You cannot override this behavior!

Pooled Remote Desktop Services

For the rest of this article, I’m going to pretend that this method doesn’t exist. If you’re operating a full-blown Remote Desktop Services (RDS) operation for your virtual desktop infrastructure (VDI), then it’s using differencing disks. Your gold master is the source, and all of the virtual machines that users connect to are built on differencing disks. When a user’s session ends, the differencing disk is destroyed.

Manual Creation

Of the four techniques to create a differencing virtual hard disk, manual creation is the rarest. There aren’t a great many uses for this ability, but you might need to perform an operation similar to the gold master with many variants technique employed by VDI. It is possible to create many differencing disks from a single source and connect separate virtual machines to them. It can be tough to manage, though, and there aren’t any tools to aid you.

You can create a differencing disk based on any parent virtual hard disk using PowerShell or Hyper-V Manager.

Creating a Hyper-V Differencing Virtual Hard Disk with PowerShell

The New-VHD cmdlet is the tool for this job:

Don’t forget to use tab completion, especially with ParentPath.

Creating a Hyper-V Differencing Virtual Hard Disk with Hyper-V Manager

Use the new virtual hard disk wizard in Hyper-V Manager to create a differencing disk:

  1. In Hyper-V Manager, right-click on the host to create the disk on, or use the Action pane in the far right. Click New, then Hard Disk.
  2. Click Next on the informational screen.
  3. Choose VHD or VHDX. The differencing disk’s type must match its parent’s type. You cannot make a differencing disk of a VHDS file.
  4. Choose Differencing.
  5. Enter the file name and path of the differencing disk that you want to create.
  6. Select the source virtual hard disk.
  7. Check your work and click Finish if you’re satisfied, or go Back and fix things.

How Manual Differencing Disk Creation is Different

A differencing disk is a differencing disk; no matter how you create them, they are technologically identical. There are environmental differences, however. Keep these things in mind:

  • Hyper-V will automatically use the differencing disks created by backup, standard, and production checkpoints. It will retarget the connected virtual machine as necessary. No such automatic redirection occurs when you manually create a differencing disk. Remember that modifying a virtual hard disk that has differencing disks will render the children useless.
  • During manual creation of a differencing, you can specify a different target path for the differencing disk. While convenient, it’s tougher to identify that a virtual hard disk has children when they’re not all together.
  • Hyper-V Manager maintains a convenient tree view of standard and production checkpoints. Manually created differencing disks have no visual tools.
  • The checkpointing system will conveniently prepend an “A” (for “automatic”) to the extensions of the differencing disks it creates (and give them bizarre base file names). Both Hyper-V Manager and PowerShell will get upset if you attempt to use AVHD or AVHDX as an extension for a manually-created differencing disk. That makes sense, since “A” is for automatic, and automatic is antonym of “manual”. Unfortunately, these tools are not supportive of “MVHD” or “MHVDX” extensions, either. If you do not give it an obvious base name, you could cause yourself some trouble.

You can use PowerShell to detect the differencing disk type and its parent:


The Inspect function in Hyper-V Manager does the same thing. You can find this function in the same Action menu that you used to start the disk creation wizard.


I also wrote a PowerShell script that can plumb a VHD/X file for parent information. It’s useful when you don’t have the Hyper-V role enabled, because none of the above utilities can function without it. Head over to my orphaned Hyper-V file locator script and jump down to the Bonus Script heading. It’s a PowerShell veneer over .Net code, so it will also be of use if you’re looking to do something like that programmatically.

Merging Manually-Created Differencing Disks

Now that you know how to create differencing disks, it’s important to teach you how to merge them. Ordinarily, you’ll merge them back into their parents. You also have the option to create an entirely new disk that is a combination of the parent and child, but does not modify either. The merge process can also be done in PowerShell or Hyper-V Manager, but these tools have a different feature set.

Warning: Never use these techniques to merge a differencing disk that is part of a checkpointed VM back into its parent! Delete the checkpoint to merge the differencing disk instead. It is safe to merge a checkpointed VM’s disk into a different disk.

Merging a Hyper-V Differencing Virtual Hard Disk with PowerShell

Use the aptly-named Merge-VHD cmdlet to transfer the contents of the differencing disk into its parent:

The differencing disk is destroyed at the end of this operation.

PowerShell cannot be used to create a completely new target disk, for some reason. It does include a DestinationPath parameter, but that can only be used to skip levels in the differencing chain. For instance, let’s say that you have a root.vhdx with child diff1.vhdx that has its own child diff2.vhdx that also has its own child diff3.vhdx. You can use Merge-VHD -Path .\diff3.vhdx -DestinationPath .\diff1.vhdx to combine diff3.vhdx and diff2.vhdx into diff1.vhdx in a single pass. Without the DestinationPath parameter, diff3.vhdx would only merge into diff2.vhdx. You’d need to run Merge-VHD several times to merge the entire chain. Hyper-V Manager has no such capability.

Merging a Hyper-V Differencing Virtual Hard Disk with Hyper-V Manager

Hyper-V Manager has a disk editing wizard for this task.

  1. In Hyper-V Manager, right-click on the host to create the disk on, or use the Action pane in the far right. Click
  2. Click Next on the informational screen.
  3. Browse to the differencing disk that you wish to merge.
  4. Choose Merge.
  5. Choose to merge into the parent or into a new disk.
  6. Check your work and click Finish if you’re satisfied, or go Back and fix things.

If you chose to merge the disk into its parent, the differencing disk is destroyed at the end of the operation. If you chose to merge into a new disk, both the source and differencing disk are left intact.

Hyper-V Manager cannot merge multiple layers of a differencing chain the way that PowerShell can.

The Dangers of Differencing Disks

There are two risks with using differencing disks: performance and space.

Differencing Disk Performance Hits

When a differencing disk is in use, Hyper-V will need to jump back and forth from the child to the parent to find the data that it wants for reads. Writes are smoother as they all go to the differencing disk. On paper, this looks like a very scary operation. In practice, you are unlikely to detect any performance problems with a single differencing child. However, if you continue chaining differencing disk upon differencing disk, there will eventually be enough extraneous read operations that you’ll start having problems.

Also, merge operations require every single bit in the differencing disk to be transferred to the parent. That operation can cause an I/O storm. The larger the differencing disk is, the greater the impact of a merge operation.

Differencing Disk Space Issues

As mentioned earlier, a differencing disk can expand to the maximum size of its parent. If you have a root disk with a maximum size of 50 gigabytes, then any and all of its differencing disks can also grow to 50 gigabytes. If the root is dynamically expanding, then it is possible for its differencing disk(s) to exceed its size. For example, in this article I have used a completely empty root VHDX. It’s 4 megabytes in size. If I were to install an operating system into the differencing disk that I created for it, root.vhdx would remain at 4 megabytes in size while its differencing disk ballooned to whatever was necessary to hold that operating system.

A merge operation might require extra space, as well. If I were to merge that differencing disk with the OS back into the empty 4 megabyte root disk, then it would need to expand the root disk to accommodate all of those changed bits. It can’t destroy the differencing disk until the merge is complete, so I’m going to need enough space to hold that differencing disk twice. Once the merge is completed, the space used by the differencing disk will be reclaimed.

If the root disk is fixed instead of dynamically expanding, then the merges will be written into space that’s already allocated. There will always be a space growth concern when merging trees, however, because differencing disks are also dynamically expanding and they merge from the bottom up.

Transplanting a Differencing Disk

Did a forgotten differencing disk run you out of space? Did a differencing disk get much larger than anticipated and now you can’t merge it back into its parent? No problem. Well… potentially no problem. All that you need is some alternate location to hold one or more of the disks. Follow these steps:

  1. Shut down any connected virtual machine. You cannot perform this operation live.
  2. Move the disk(s) in question. Which you move, and where you move them, is up to you. However, both locations must be simultaneously visible from a system that has the Merge-VHD cmdlet and the Hyper-V role installed. Remember that it is the disk that you are merging into that will grow (unless it’s fixed).
  3. If you moved the differencing disk and you’re still on the system that it came from, then you can probably just try the merge. It’s still pointed to the same source disk. Use Get-VHD or Inspect to verify.
  4. If you moved the root disk, run the following cmdlet:

    Your usage will obviously be different from mine, but what you’re attempting to do is set the ParentPath to reflect the true location of the parent’s VHDX. You can now attempt to merge the disk.

If a differencing disk somehow becomes disjoined from its parent, you can use Set-VHD to correct it. What you cannot do is use it to rejoin a differencing disk to a parent that has been changed. Even though the Set-VHD cmdlet may work, any merge operation will likely wreck the data in the root disk and render both unusable.

How to Delete Hyper-V Checkpoints

How to Delete Hyper-V Checkpoints

Checkpoints (formerly known as snapshots) are something like an “undo” button for a virtual machine. As the name suggests, you set a marker at a particular point in time for the virtual machine. If you decide that you don’t like what’s happened to the virtual machine at any point after that, you can go back (revert) to that checkpoint. As we, and a great many others, have gone to great lengths to emphasize, checkpoints are in no way a replacement for backup. Like an actual “undo” button, all of the changes made after the checkpoint are lost if you revert. There is something like a “redo” button; you are given the opportunity to take another checkpoint when you revert to an earlier one. However, no copies of the virtual machine are ever made. Only changes are tracked. Do not attempt to use this feature as a backup tool.

The purpose of this post is to explain how to delete hyper-v checkpoints and what happens when you do so in various conditions. The reasoning behind deleting a checkpoint is straightforward: once you’ve decided that you are happy with the state that a virtual machine is in, or if you have multiple checkpoints and are certain that at least some of them are no longer useful, you can get rid of the unnecessary ones. This post will use a virtual machine with the following checkpoint states as an example:

Checkpoint Example

Checkpoint Example

I’ve renamed them to give them some meaning. We’ll treat this as though we are software developers testing potential patch builds to determine which has the optimal outcome. This configuration is not nearly as complicated as it might at first seem, but it does warranty some explanation. To decipher the tree, start at the root. The checkpoint named “Greenfield” represents the base virtual machine. It is using the original VHDX which is now frozen in time. Some data might be being read from it (particularly the operating system files) but nothing is being written to it. Both the “A” and the “C” branches are completely dormant; none of their files are being used at all. The “B” branch has more activity. Any unique data in checkpoints titled “Patch Branch B”, “Patch B1”, “Patch B2”, and “Patch B3 Variant 1” will be read as necessary. Checkpoint “Patch B3” is completely dormant. All writes are being directed to “Patch B3 Variant 1”.

Simple Checkpoint Deletion

For the sake of this discussion, we’ll say that each checkpoint named “Patch Branch” is nothing but a checkpoint right off of “Greenfield”, and that each of the other checkpoints was taken immediately after the application of the entity it is named after. To illustrate a simple deletion, imagine that it’s been determine that Patch A3 had a flaw and needed to be rebuilt. Rather than correct it in Patch A4, your development team has decided to re-release Patch A3. In that case, it makes sense to simply delete the checkpoint titled “Patch A3” and start over from there. To perform the delete, right-click on the checkpoint and click Delete Checkpoint:

Checkpoint: Simple Delete

Checkpoint: Simple Delete

Because “Patch A3” is in a dormant branch, all files related to that checkpoint will be permanently deleted and that will be the end of it. No other checkpoints or the running state of the virtual machine will be affected. To continue with the scenario outlined in the previous section, you would likely Apply the “Patch A2” checkpoint and then upgrade it with the re-released Patch A3 and then take a new checkpoint.

Branch Checkpoint Deletion

Let’s move on to something a bit more complex. Let’s say that the development team decided that the “C” patch branch was an unmitigated disaster and they just want to forget the whole thing to focus on “A” and “B” branches. To effect that, right-click on the “Patch Branch C” checkpoint and choose “Delete Checkpoint Subtree”:

Checkpoint: Subtree Delete

Checkpoint: Subtree Delete

This action will delete all of the checkpoints and all of their related files from the checkpoint that you chose and all of its child checkpoints. Like the “Patch A3” object, this is in a dormant tree so the running state of the virtual machine is not affected in any way.

Mid-Tree Deletion

Next, you get a call from the software engineering team that says that they are perfectly happy with the testing of Patch B1 and will be adding it to the permanent repository. Technically, you don’t have to do anything. But, this is the actively running branch and each layer of checkpoints impacts performance. However, Patches B2 and B3 are still undecided, so you can’t just delete the entire subtree like you did with the former Patch C branch. Fortunately, it’s not a problem. Just delete “Patch B1” the same way that you deleted “Patch A3”:

Checkpoint: Mid-tree Delete

Checkpoint: Mid-tree Delete

Once this completes, you’ll see that “Patch B2” and both of the “Patch B3” items are still present and unchanged… or are they? The merge operation combined all of its changes back into its parent of “Patch Branch B”, as you likely expected. However, those files were the parents of “Patch B2”. If you check, you’ll find that the parent objects of “Patch B2” are now the files for “Patch Branch B”. The parent chain for the items below “Patch B2” are unchanged. The action that you took has the effect of reducing the overall length of the I/O chain for data reads. However, any difference tracking between the former “Patch Branch B” and “Patch B1” are now lost; “Patch B1” is permanently applied to “Patch Branch B”.

Active Tree Deletion

After some time, the development team contacts you again, saying that they are thrilled with the entire B patch series that you’re working with. Patch B3 Variant 1, which you are currently working with, is better than the original Patch B3, so they just want to get rid of that and end testing of the Patch B branch entirely. To effect this, you could, as before, leave everything alone. But, to improve your performance, you could clean up all of those checkpoints. So, you choose to right-click “Patch Branch B” and click Delete Checkpoint Subtree:

Checkpoint: Delete Active Tree

Checkpoint: Delete Active Tree

This is the outcome:

Checkpoint: Active Tree Deleted

Checkpoint: Active Tree Deleted

Was that a mistake, or was it what you really wanted to do? That depends. Let’s go over what all has happened here. First, anything that happened in the “Patch B3” item is completely lost. Second, all of the changes made in “Patch Branch B”, “Patch Branch B2”, and “Patch Branch B3 Variant 1” are all collapsed into the singular “Now” object. What did not happen is any combining of the “A” branch with the “B” branch, even though there are no longer any objects with the “B” name. The “Now” object and the “Patch Branch A” object exist in the tree at exactly the same level, which means that they are separate, unrelated items. Since you may need to do some more testing with the “A” branch, you might find it wise to create another checkpoint from the level that you’re at now just to keep the “B” branch alive. If you jump to another checkpoint without creating a new checkpoint at this level, the entire branch formerly known as “B” will be lost.

Downstream Checkpoint Deletion

With the successful testing of the “B” branch, the development would like you to return to testing the “A” branch. They’re not sure about A1, so they wanted you to start there again. This is how it looks:

Checkpoint: Downstream Setup

Checkpoint: Downstream Setup

As you test, they discover that A1 had a critical flaw that was then propagated into A2. They don’t want either of these patches to see the light of day. What should you do? This?:

Checkpoint: Delete Downstream Quiz

Checkpoint: Delete Downstream Quiz

If this is what you do, you will not have met the requirements of the development team. This action will discard “Patch A2”. However, it will permanently roll all of the changes from “Patch A1” back into “Patch Branch A” and set that as the running state of the virtual machine. Since the development team wanted you to get rid of Patch A1, that means that the “Patch Branch A” object is useless. In this scenario, you fortunately have the “Greenfield” object to apply, but not everyone remembers to take a checkpoint for that purpose. What you actually want to do is first apply “Patch Branch A” (since you don’t care about keeping anything underneath it, there’s no point in taking another checkpoint when prompted). That will leave you with both “Patch A1” and “Patch A2” as dormant downstream objects. You can now safely delete the subtree from the “Patch A1” level to achieve the stated outcome:

Checkpoint: Delete Downstream Tree

Checkpoint: Delete Downstream Tree

Accepting the Current State — Deleting All Checkpoints

After a strange silence, the head of development calls you. Their department has run out of money. They’ve decided to release the “Patch B” tree. What they were working on for the “Patch A” branch is going to be rebranded as “version 2.0” so they can make money on it, but that will all be done at some point in the future. So, they want you to make the “Patch B” branch permanent and get rid of everything else. As you learned from the previous section, this means that you first need to apply the saved “Patch B” branch so that it is active. Then, just start at the root and Delete Checkpoint Subtree:

Checkpoint: Delete All

Checkpoint: Delete All

A couple of things happen when you delete the root. All checkpoints in the active tree are merged upward into the root. All checkpoint data in all other branches are irrevocably lost. So, before you do this, take care to start at the “Now” object and trace your way upward back to the root to ensure that you really are in the tree that you think that you’re in. If you want to discard all of the changes, apply the root object first and then delete the entire subtree.

Once this operation completes, there will be no checkpoints at all. All read and write activity will occur on the primary VHDX and VM files.


PowerShell & Hyper-V: Finding the Parents of a Differencing VHD

PowerShell & Hyper-V: Finding the Parents of a Differencing VHD

We’ve had quite a few posts about Hyper-V checkpoints lately (formerly snapshots). We also spend a fair bit of time warning people not to tinker with them manually. There are still those people that are going to tinker despite any warnings, and there will always be those people who don’t even find the warnings until they’re too late to be of any value. Worse, there will always be those unfortunate few that do everything right and still find themselves in a mess. The least I can do is provide a tool that can be of use to anyone that’s stuck working on a complicated tree of differencing disks.

As a refresher, a differencing virtual hard disk contains all the changes that would have been made to a fixed or dynamically expanding virtual hard disk. These disks have a number of uses. The most common is with checkpoints, which automatically create differencing disks (and prepend an A to the extension so that it becomes an AVHD or an AVHDX). They are also used by Remote Desktop Services (RDS) to deploy an individual machine in a pool from a single parent. We can create them manually with New-VHD to attach one or more dependent child virtual machines to a single master.

Once a differencing disk has been created, any virtual machine that is assigned to that differencing disk will only write changes into the differencing disk. When it reads from the differencing disk, any blocks that it does not contain are retrieved from the parent. It is possible for a differencing disk to be a parent of a differencing disk, but they must eventually trace back to a root fixed or dynamically expanding disk.

To put it mildly, problems arise when changes are made to a VHD that is the parent of a differencing disk. So that there is no misunderstanding, these problems are catastrophic to the differencing disk. When a parent-child relationship exists between a virtual disk and a differencing virtual disk, all changes must occur in the child. If any change, however minor, is made to the parent, the data in the child is invalidated. The parent is still fully usable.

So, if you’re going to tinker with a disk in a differencing relationship, you must have a clear idea of what that relationship is. Unfortunately, there’s really no way to look at a VHD and determine if it has children. It’s easiest with checkpoints, because Hyper-V will always create the AVHD/X files in the same folder as the parent. Even if you can’t visually confirm which is the newest or oldest, at least you know that none have wandered off into a neighboring pasture. For RDS, you should only be using the provided modification tools because they have their own methods of protecting the differencing children. If you’re out creating differencing disks on your own, hopefully you paid attention to where you placed them.

How to see all disks upward of a VHD file

While I can’t be of much help in determining what the newest disk in a differencing chain is, I can show you a very quick way to see all the disks in a chain upward of a VHD file that you provide. Just use the following script:

The above script is one of those fun ones where the introductory comments dramatically outnumber the functional parts of the script. The only problem is that it runs the risk of being a bit too clever. I like my scripts to be easy to read. If this were any longer, or was part of a bigger script, I wouldn’t do it this way. This is because one line of script does three things, and some of the functionality is obscured. Look specifically at this line:

The first thing this line does is run Get-VHD against $Path.

The second thing that it does is assign one property of the output of Get-VHD to $Path. There are two confusion points in that piece alone. First, remember that in PowerShell, the = sign is the assignment operator; -eq is the equality operator. We are not checking if $Path is equal to its own ParentPath. The second confusion point is using $Path as a parameter to a cmdlet whose output is being assigned back into $Path. It works because as soon as Get-VHD completes, PowerShell moves on to evaluating the value of its ParenPath, then performs the assignment. The original value of $Path does not collide in PowerShell because it stops caring about the value of $Path before its even done with Get-VHD. It’s not a problem for us to reuse $Path because we don’t need its original value anymore.

The third thing that this line does is verify the outcome of that assignment for the while. while is a deceptively simple language construct that expands to: if the condition is not zero or empty, loop until the condition is zero or empty. Pay attention to the “condition is zero or empty” part, as this is not intuitive for newcomers. It would be logical to expect it to expand to: if the condition is true, loop until the condition is not true, because that is how humans use if in natural language. The only computer language that I am familiar with that implements If in the natural language fashion is Visual Basic. If this same line of script were converted to VB, it would refuse to compile because $Path = (Get-VHD -Path $Path).ParentPath cannot be evaluated as a simple true/false condition. Because PowerShell will continue operating the while loop as long as the condition inside parentheses produces something, it will loop until it encounters a VHD that doesn’t have a parent.

There is only one remaining line of script, and that is the  $Path item on a line all by itself inside the while loop. This cause the value of $Path to be placed into the pipeline. If you run this script as-is, it will just emit the text to the screen, line by line. If you pipe into something that accepts a String object or an array of Strings, it will handle them appropriately.

Most of the programming and scripting best practices recommendations that I’ve seen will tell you to avoid writing code that doesn’t follow natural language conventions because of the potential confusion. That means that a friendlier script would look like this:

While friendlier, it duplicates script and, in my opinion, has other issues that make it harder to read than the method that I chose. Of course, there are other ways to script this to wind up with the same output, but this is one of those few cases where increased readability introduces increased complexity.

Bonus: Re-using a Get-VHD Script

I wrote a script that looks for orphaned virtual machine files a while back, and that same post I included a script that could use only built-in PowerShell components to check a VHD/X file for parents. The purpose of doing so is that you need the Hyper-V PowerShell module loaded in order to use Get-VHD, even though virtual disk files can be used on systems without Hyper-V, such as Windows 7. If you wanted to use that script with this one, then making two simple changes to produce the following script will do the trick (assuming that you loaded Get-VHDDifferencingParent as a dot-sourced function):

The referenced script wasn’t included in the PowerShell and Hyper-V series because it’s far more complex than what I had in mind for this teaching series. You shouldn’t let that stop you from tearing into it to see what you can learn (or if you can find something I did that could be fixed).


How To Take a Checkpoint in Hyper-V

How To Take a Checkpoint in Hyper-V

Checkpoints (known as “Snapshots” in previous versions), provide something similar to an “Undo” capability to Hyper-V virtual machines. In fact, Hyper-V’s ancestor Virtual Server/Virtual PC employed a technology called undo disks that served a similar purpose. With a Hyper-V checkpoint, everything about a virtual machine is captured in a checkpoint; the disk contents to be sure, but also the state of memory and active CPU threads, the hardware configuration, the condition of Integration Services, etc. Essentially, anything captured by any of the virtual machine’s files is perfectly preserved at the aptly named “checkpoint”.

Setting a checkpoint is very simple using Hyper-V Manager. Simply right-click on the virtual machine in question and click Checkpoint:

Capture Checkpoint

Capture Checkpoint

Since the process is non-destructive, there’s no confirmation. You’ll see the progress of the checkpoint creation in the Status column.

Checkpoint Status

Checkpoint Status

The checkpoint’s virtual hard disks are stored next to the originals, with a new automatically generated name and an AVHD[X] extension:

Checkpoint Virtual Hard Disks

Checkpoint Virtual Hard Disks

The checkpoint is given its own XML file that describes the virtual machine (such as the state and connectivity status of hardware, etc.) in the same format as the original but with a unique ID. It is kept in a separate Snapshots folder.

Checkpoint XML

Checkpoint XML

As with a normal virtual machine, a folder with the same ID as the XML file is created alongside the XML file. Inside it are the BIN and VSV files for the checkpoint.

Checkpoint BIN and VSV

Checkpoint BIN and VSV

Checkpoint Creation Notes

  • Checkpoints are not backups. No data is duplicated. The checkpoint’s virtual disk files are useless without the base virtual machine files.
  • An AVHDX file contains the changes made to the blocks of its parent. It will start very small but can theoretically grow to be as large as its parent disk’s maximum size. This means that for dynamically expanding disks, it is possible for the child to be larger than the parent.
  • The XML, BIN, and VSV files that are created in the Snapshots folder contain the state of the virtual machine as it was when the checkpoint was taken. Any modifications made to the virtual machine’s configuration while it has an active checkpoint are applied directly to the XML, BIN, and VSV files in the virtual machine’s usual storage location. This process is the opposite of how the VHD[X] files for the virtual machine are treated, as the original VHD[X] files are placed in a read-only state and all changes are made to AVHD[X] files.
  • Pass-through disks are not captured in a checkpoint. This can cause data consistency problems if its virtual machine is reverted. The same is true for anything attached to a virtual fibre channel port (virtual SAN).
  • The connection state of a pass-through disk is captured in a checkpoint. What this means is that if a pass-through disk is removed or added while a checkpoint is active, its connection state will be reverted to the prior state if the checkpoint is reverted.  The data on it is unaffected. The same is true for anything attached to a virtual fibre channel port (virtual SAN).
  • Never make any changes to any files that constitute a checkpoint or a virtual machine that has an active checkpoint. It is very difficult, sometimes impossible, to correct these changes.
  • By default, files for a checkpoint are kept in the host’s default location for virtual machine files in an automatically-created Snapshots subfolder. The virtual machine location can be overriden for any given virtual machine. From that point onward, any new checkpoints are created in a Snapshots subfolder of that location. The Snapshots location itself can also be overriden for any given virtual machine, which will also cause a Snapshots subfolder to be generated.
  • Only the BIN, VSV, and XML files are affected by relocating the Snapshots folder. The AVHD[X] files always exist with their parents.
  • When performing a Storage Live Migration using Hyper-V Manager on a virtual machine with active snapshots, it appears that only the base VHD[X] will be moved. In actuality, the base disk and all of its AVHD[X] files are moved together.
  • When performing a Storage Live Migration using Failover Cluster Manager on a virtual machine with active snapshots, it appears that only the current AVHD[X] will be moved. In actuality, the base disk and all of its AVHD[X] files are moved together.


The Hyper-V Hub 2014 Year in Review

The Hyper-V Hub 2014 Year in Review

Like any creative work, a blog post is never really done; it’s just abandoned. Unlike many other mediums, blogs do allow us to easily refresh those older articles, but we so rarely ever do it. To close out this year, a few of us on the editorial team got together and selected a few highlights from the past year.

Our 14 selections from 2014 (in no particular order):

Hyper-V Guest Licensing

This was our first licensing article directly related to guest licensing. We followed it up with a downloadable eBook that was expanded to include a number of examples, and Andy Syrewicze and Thomas Maurer gave a fantastic webinar on the topic. We’ve received quite a few questions and some great feedback. Keep an eye out for a follow-up post that takes on some of those questions and incorporates some of the suggestions. If you’ve got questions or suggestions of your own, feel free to send them in (by leaving a comment)!

Demystifying Virtual Domain Controllers (series)

This two part series started with a look at all the myths surrounding the virtualization of domain controllers and exposed the truths. In part 2, we explained how to successfully virtualize your domain controllers without headaches. I felt that both of these articles explained the situation and remedies very well, except that it seems that some people have been unable to make time synchronization work correctly when the Hyper-V Time Synchronization service is partially disabled for virtualized domain controllers. Microsoft has changed their published policy to include a recommendation that time sync be fully disabled for DC guests, although personally, I think the jury is still out. Completely disabling it is certainly an expedient solution to a frustrating situation, but that doesn’t automatically make it the best option. I’ve been able to make partial disablement work every time I’ve tried, and it’s the only way to guarantee that a DC that was saved (even accidentally) will be able to properly recover.

7 Reasons not to Make Hyper-V a Domain Controller

I wrote this post as I saw a lot of people really trying hard to justify using their Hyper-V hosts as domain controllers. It’s a really bad idea, so I collected all the reasoning I could think of not to do it. In retrospect, I probably should have ordered them a little better to address the topmost concerns first. Those are:

  1. Licensing costs. People want to save on licensing by just installing the domain controller in the Hyper-V host so that they don’t have to pay for a Windows license just to run domain services. As explained in #4, this doesn’t work.
  2. The chicken-and-egg myth. People believe that if they join the Hyper-V host to the domain and the only DC is a guest, then the Hyper-V host won’t boot or will have other problems. That wasn’t even mentioned in this article, at least not outright, although it was a major point of the articles that it links to.
  3. The myth that Hyper-V hosts should be left in workgroup mode to increase security. This was included as point #1, which isn’t a bad placement for it. People get too close to a situation and sometimes make irrational decisions. They don’t want to put their systems at unnecessary risk and a compromised Hyper-V host can potentially put a lot of guests at risk all at once, so they do take steps to distance the host from the guests. While there are solutions, workgroup mode isn’t one of them. I mean, just try to say this out loud without laughing: “Workgroup mode is more secure than domain mode”. Or, write this on your resume: “I once put a computer in workgroup mode to increase security over domain mode”. I’ll bet that won’t get you many callbacks.

Common Hyper-V Deployment Mistakes

“There are three kinds of men. The one that learns by reading. The few who learn by observation. The rest of them have to pee on the electric fence for themselves.” – Will Rogers

One really good way to learn is from your own mistakes. Most people that do something really wrong are much less likely to do that same thing twice. But, as far as I’m concerned, it’s a whole lot better if you can learn from other people’s mistakes. This article compiles a list of the ones I’ve seen most frequently in the hopes that at least a few people can avoid them.

Storage Formatting and File Systems (part 4 of a series)

We had an unusually long (7 parts!) series on storage and Hyper-V. Even though this particular piece looked at parts of storage that some people might find very basic, we often forget that there are always newcomers, and sometimes even experienced administrators missed some of the basics during their career. A highlight of this article is that it puts to bed an old recommendation about spending a lot of time on sector alignment. It also takes a generalized look at architecting disk layout for Hyper-V.

Connecting to Storage in Hyper-V (part 6 of a series)

Everyone likes a good how-to. Even if you can figure something out on your own, it makes little sense to do so if someone else has already done the work. For many administrators, moving to a virtualization platform is the first time they’ll connect to external storage, which is why I took the time to lay out exactly how it’s done. I noticed that we had intended to publish a Powershell-equivalent article to this one, but that never came to fruition. We’ll rectify that in 2015.

Creating a Virtual Machine from a Hyper-V Snapshot (series)

Another outstanding submission from Jeff Hicks, this article shows a clever way to create a snapshot (now checkpoint) of a virtual machine and turn it into its own virtual machine. This gives you some cloning powers without needing to incur the expense of something like Virtual Machine Manager. Be sure to keep reading into part 2, where he shows you how to do it all much more efficiently with PowerShell.

10 Awesome Hyper-V Cmdlets

Jeff Hicks compiled a list of his 10 favorite Hyper-V cmdlets and took us all for a quick tour. If you’re thinking about integrating PowerShell with Hyper-V into your toolkit as a New Year’s resolution, this is a great place to start with one of the topmost experts.

A PowerShell-Based Hyper-V Health Report

This fantastic article was written by Jeff Hicks, and is one of my personal favorites. This is a wonderful little script that quickly runs against the target hosts that you specify and returns a snappy-looking HTML health report. Even better, Jeff shows you how to set it up to run automatically against all your hosts, so you can easily have a daily report.

Best Practices to Maximize Hyper-V Performance

In this article, Nirmal discusses the best approaches to ensure your Hyper-V guests are operating at their peak performance. Since Nirmal focused on peak performance and we had more than a few comments about that, we’re planning a follow-up article that contains more generalized best practices.

Resizing Online VM Disks

Another great how-to guide from Nirmal shows you how to use the new feature of 2012 R2 that allows you to resize a Hyper-V VM’s virtual SCSI disk without shutting the guest down.

Hyper-V Virtual CPUs Explained

Out of all my articles, this is one of my personal favorites. I feel really bad for people who spend a lot of time wringing their hands over how many cores should be in their Hyper-V hosts, especially when they wind up spending too much money on CPUs that are just going to sit idle. One thing I probably should have mentioned in that article is that one of the first things many of us do to address certain Hyper-V performance issues is disable many of the power-saving features of our processors (C-states, especially). If you’ve got a lot of cores sitting idle, that’s a lot of wasted energy. And, if Aidan Finn’s prediction about per-core licensing comes true in 2015 (I really hope it doesn’t), then it’s going to translate into lots of wasted licensing dollars as well.

The Hyper-V and PowerShell Series

PowerShell still hasn’t become of serious importance to far too many administrators, and that’s really a shame. Since I clearly remember the days before I learned to embrace it and all the reasons I invoked to avoid it, I can certainly understand why. It’s just one of those things that can be tough to see the value in until you have your own personal “A ha!” moment. This series is meant to serve a dual role: one is to provide useful scripts to the Hyper-V community. The other is to simply to teach by example. If you’ve just come for the scripts, great! If you happen to learn something while you’re here, that’s even better!

Finding Orphaned Hyper-V VM Files

I’ve written a number of PowerShell scripts for you now, but far and away this one is both the most complicated and my favorite. For a very long time prior, I had been thinking, “Someone should do that.” I was eventually forced to accept that I qualify as “someone”. A few days into this, I realized just why no one had done it. I learned a lot, though, and there are certainly a great many uncommon techniques included in this script. So, it has its surface value of being the only tool I know of that can track down orphaned Hyper-V files, but it would also be something that intermediate PowerShell scripters can tear apart for tidbits to add to their own scripting toolkit.

The Future

Time keeps on slippin’, slippin’, slippin’, into the future. – The Steve Miller Band, “Fly Like an Eagle”

As we close out 2014 by examining our successes, we acknowledge that all of it has been made possible by you, our readers. If you want to have a say in how in how we approach 2015, now is your chance! Let us know in the comments what you’d like to see more of, what you’d like to see less of, and what big things we’ve missed entirely.

On behalf of the Altaro blogging crew, we’ve had a wonderful time writing for you and interacting with you in the comments section. We wish you and yours a wonderful holiday season and look forward to another wonderful year!

Free Script: Find Orphaned Hyper-V VM Files

Free Script: Find Orphaned Hyper-V VM Files

If you’ve had very much virtual machine churn in your environment, it’s almost inevitable that you’ve wound up with a few disconnected virtual machine files here and there. This free script will help you to locate orphaned Hyper-V VM files. For fellow infrastructure scripters, there is a special bonus script included as well.

The worst thing about my testing environment is that I often wind up with files that look legitimate, but are no longer part of any virtual machine. Even in live environments, some virtual machine moves and operations leave a trail of unwanted files. Errors and failures can produce more.

Finding virtual machine files isn’t hard, but ensuring that they’re not really attached to something can be much more difficult than it might at first seem. I tried very hard to include a great many features to cover every scenario I could think of.

Last Updated: April 11th, 2015

Feature List

  • Aware of snapshots/checkpoints
  • Aware of differencing disk chains
  • Cluster-aware
    • Will only scan Cluster Shared Volumes from a single node, and will not generate false positives while scanning other nodes
    • Will differentiate between shared and not shared storage, so you can scan local drives on nodes
    • Will detect all nodes in a cluster and scan each
    • Can be set to ignore cluster membership if you only want to scan non-shared locations
    • I didn’t do extensive testing with standard cluster disks as they’re not something I use, but it seems to work well enough. However, if a cluster disk were to move between the portion of the script where it scans for files to exclude and when it scans again for orphans, the results will be invalid.
    • The script understands raw volume identifiers, although I didn’t do a great deal of testing with those either.
  • Can be run from any system with PowerShell 4.0; the Hyper-V module is not required. I didn’t test it with PowerShell 3.0 systems because I haven’t got any left, but I think it will work
  • Can target 2012 and 2012 R2 systems simultaneously
  • Options to scan one or all of the following locations: the host’s default VM and VHD paths, all paths currently in use by VMs, and paths that you specify
  • All scans occur directly on the target host(s) except in the case of UNC paths, and each separate host is scanned in parallel
  • Detailed built-in help. Use Get-Help -Full to see it

Files that the Script Scans For

  • XML files with a base name in the format of a GUID. It does not parse the XML to see if they really are Hyper-V files, so watch out for false positives.
  • BIN files with a base name in the format of a GUID.
  • VSV files with a base name in the format of a GUID
  • SLP files with a base name in the format of a GUID and an intermediate extension
  • VHD, VHDX, AVHD, AVHDX, and VFD files; if you look at the script, it will match on anything that has VHD or VFD anywhere in the extension; please report false positives as I can update the script to be more selective if necessary.


  • All target hosts must have the Hyper-V PowerShell module installed. Any that don’t have it will throw errors that they cannot be scanned. The computer that you’re using to run the script does not need anything other than the default PowerShell modules installed. However, scanning SMB shares will be considerably faster if the Hyper-V module is installed locally.
  • The script will only exclude differencing disk chains from the point that a VM references it and upward to the root. For example, let’s say you have a “root.vhdx” with children “diff1.vhdx”, “diff2.vhdx”, and “diff3.vhdx”, with each one being a child of the previous disk. Let’s say you have a virtual machine that references “diff2.vhdx”. The scan will find the VM using “diff2.vhdx” and trace its chain upward through “diff1.vhdx” and “root.vhdx” and ignore them. If it doesn’t find any VM using “diff3.vhdx”, that file will be marked as orphaned. However, the script is snapshot/checkpoint aware so even complicated trees should not trigger a false positive.
  • The script uses a mix of implicit and explicit PowerShell Remoting with a very heavy dependence upon explicit. Even if you initiate directly from a system to be scanned, it will use PSRemoting. The drawback is that you can’t initiate this script inside a remote session unless you have CredSSP enabled and even then it might fail. To compensate, I ensured that you don’t need to have any special modules installed on the system that you run the script from.
  • You can’t specify connection credentials for scanning SMB 3 storage. I tried. It didn’t work. I didn’t get any errors to troubleshoot. It just didn’t work. If you use the -Credential parameter, the credentials will be used for retrieving host details, determining existing VM information and paths, and scanning storage local to the target hosts, but SMB 3 shares will be scanned using the credentials of the local session being used to invoke this function.
  • This script uses fully-qualified domain names to connect to domain members and you can’t get around it. The purpose of this is to ensure that cluster scans work correctly. For instance, if you pass in the short name of a cluster (“clhv1” in my case), it will use WSMan to attach to that address, query the system for its FQDN using WMI, and use that for all future connections. While it’s there, it will also determine the names of any other systems in the cluster and retrieve those as well. The side effect is that if you supply a DNS alias, it will be replaced with the true computer’s name for all subsequent connections. If you’re having trouble connecting to a system, this is probably why. Use the -Verbose switch to find out what system name it’s trying to connect to. If the target isn’t domain-joined, then the script should connect using only its NetBIOS name.
  • I had to install a lot of wiring to avoid returning false positives on Cluster Shared Volumes. The solution I settled on is that all scans of the \ClusterStorage path will be handled only through the primary node in all cases. You won’t be able to bypass this behavior by any means, even by manually specifying a path of C:\ or C:\ClusterStorage. I don’t think this will be a problem, but the explanation is here just in case you see behavior that you didn’t expect, such as by setting -IgnoreClusterMembership and scanning the C: drive of a node other than the primary.
  • The way I use PowerShell Remoting in parallel might cause some scans to take longer than you might think they would when you are only scanning the local system. That’s because of the behavior of the Wait-Job cmdlet. Since disk scans tend to be slow anyway, I don’t think that this adds enough time to the scan to be harmful, but if you think that something is taking a few seconds longer than it should, this is probably why.
  • The script will scan UNC paths, but might have some unexpected behavior in a few cases. For instance, let’s say you have a Windows Server system serving an SMB 3 share to hold VMs for other hosts. You also decide to stand up a local VM on that host and put it in the same location. From that host’s perspective, the VM might be stored in, let’s say D:\VMs. D:\VMs is also shared as \\storage-server\VMs. If the scan is set to look at that share location, then it’s going to return all of that virtual machine’s files as orphans because it only knows them as being registered in D:\VMs on that host.

There is no built-in option to remove the files that it finds. This would be a fairly trivial modification, but I’m not going to make it. I worked very hard to do almost everything I could to eliminate false positives, but they can and will happen and I won’t be responsible for someone accidentally wiping out all their templates or a completely unrelated file that happened to be named in the pattern of a Hyper-V file.

Script Usage

There are two parameter sets. The first is the default. It first finds all the virtual machines on a host for their files. It then scans all the folders that hold those files and the folders marked as the host’s defaults for orphans.

Default Parameter Set

  • ComputerName accepts a string array of computer names. If you don’t include any, the local system is scanned.
  • ExcludeDefaultPath prevents the host’s default paths from being scanned for orphaned files.
  • ExcludeExistingVMPaths prevents the paths of existing VMs from being scanned for orphaned files.
  • IgnoreClusterMembership treats the target system like a standalone machine even if it’s in a cluster. CSV folders will be ignored.
  • Credential allows you to specify a set of credentials that will be used to scan remote locations. Any SMB 3 shares will always be scanned from the perspective of the local system using the credentials of the local session.

Alternate Parameter Set

  • ComputerName in this set is the same as the default.
  • Path accepts a string array of paths to check. Each host in the ComputerName array will be scanned for this folder. If it’s not there, you’ll just get a warning about it, so there’s no harm in including it even for systems where you know it doesn’t exist. When you manually specify paths, only those locations are scanned by default.
  • IncludeDefaultPath instructs the script to scan the host’s default paths for orphans when -Path has been specified.
  • IncludeExistingVMPaths instructs the script to scan the folders of existing VMs for orphans when -Path has been specified.
  • IgnoreClusterMembership works in this set the same way as in the default.
  • Credential in this set is same as the default.

Sample Output

Get-OrphanedVMFiles ScreenshostThe output is essentially that of Get-ChildItem. Take a look at the PSComputerName column. That will tell you which computer the file was found on. If it reports “localhost”, that means the containing folder is on a share. All others will be local or on a CSV on the indicated computer.

Script Listing

Copy and paste the included script into a file. It’s written to expect Get-OrphanedVMFiles.ps1, but you’re welcome to use anything you like. Don’t forget that copying and pasting from this site includes a tag line at the very end that you’ll need to remove.

I did not write the script to be dot-sourced. If you want to do that, encase the contents as shown:

And here’s the script:

Bonus Script

The important parts of this script are contained in the body of the main script above, but I thought its contents might be useful to anyone trying to use PowerShell or .Net to work with the metadata or contents of VHD and/or VHDX files. I needed it in the above script because I promised that you could run it from any remote system, even one without the Hyper-V PowerShell module installed. Because of that, I needed some way to check VHD files to see if they are differencing disks, and if so, to retrieve their parent disk files to exclude them as orphans. To do that with a Microsoft-developed PowerShell script, you need Get-VHD. Get-VHD doesn’t only need the PowerShell module installed, it also requires the Hyper-V role to be enabled. I didn’t think it was fair to expect you to do all that just to see if a VHD on a share is a differencing disk or not, so I built a function that only needs the default PowerShell modules.

I only needed to know enough about the file to see what its parent VHD(X)s are, but there is more than enough example here to help someone go so far as to build a complete VHD(X) parser, if desired. I/O operations have always been one of my weakest programming skills, so I pretty much abandoned any notions of designing an efficient script and went for one that more or less maximizes readability. It works more than quickly enough for my purposes, but I would highly recommend that if you intend to build this into something intended for a heavier role that you spend some time optimizing what I’m leaving you.

This script illustrates how to safely open a VHD(X) file for reading, even if it’s in use by Hyper-V. It shows how to step through the components of the file to look for the information that you need. It also includes a couple of methods for deciphering the bizarre information encoding patterns of VHD files. Look in the .NOTES section of the comment-based help for links to download the specifications for both VHD and VHDX files which can help you to design a script for whatever you’re looking for.

Note: Even though this script is always safe to run, I’ve gotten semaphore timeouts while reading very large files at the other end of an SMB share.


  • Path is the script’s only input. This is a string field that will take the name of the disk file to scan. I didn’t test it, but based on my understanding of PowerShell, piping a string array to it that contains multiple file names should cause the script to run against each of them.

If the targeted file name has a parent, its complete path name will be returned as a string. If it doesn’t, the script will return nothing. The entire script is encapsulated in a single try block whose main purpose is to ensure that the targeted file is closed under any circumstances. Errors from earlier sections will be rethrown. If you call this script from one of your own, consider encapsulating it in a try block with -ErrorAction Stop.

Because this script was intended for use with the above script that does its own extensive validation, I didn’t bother with any validation of -Path. The script will fail predictably even without such checks so it’s not strictly necessary.


How to Create a New Virtual Machine from a Hyper-V Snapshot Part 2

How to Create a New Virtual Machine from a Hyper-V Snapshot Part 2

In the first part of this series I walked through using the Hyper-V Manager to create a new virtual machine from a snapshot. If this is something you only need to do occasionally, there is no reason not to use a graphical tool. But if you find yourself doing this often, or would like a way to automate the process for the sake of efficiency, then you’ll need to turn to Windows PowerShell. (more…)

Page 1 of 212