How To Copy or Backup a VHD File While the VM is Running

How To Copy or Backup a VHD File While the VM is Running

I think that we can all agree that backup software exists for a reason. Well, lots of reasons. Very good reasons. So, if you ask me in the abstract how to make a copy or backup of a virtual machine’s virtual hard disk file while the virtual machine is running, I’m probably going to refer you to your backup vendor.

If you don’t have one, or don’t have one that you can trust, then I am naturally going to recommend that you download Altaro VM Backup. Backup is their wheelhouse and they’re going to have a lot more experience in it than any of us. The outcomes will be better than anything that we administrators can do on our own.

But, I also understand that sometimes you have one-off needs and you need to get something done right now.

Or, you need to script something.

Or your backup software isn’t quite granular enough.

Or you have some other need that’s unique to your situation.

If you need to get a copy or backup of one or more of a virtual machine’s hard disks without shutting down the virtual machine, you have three options, shown in their preferred order:

  1. Use your backup application, as we discussed.
  2. Export the virtual machine with Hyper-V Manager or Export-VM. This only works for Hyper-V versions 2012 R2 and later.
  3. Copy the file manually.

I’m not going to change my mind that a backup application is the best way to get that copy. But, I’m done beating that horse in this article.

Export is the second-best solution. The biggest problem with that is that it exports the entire virtual machine, which might be undesirable for some reason or another. It also locks the virtual machine. That won’t necessarily be a bad thing, especially if all you’re doing is getting a copy, but maybe it’s a showstopper for whatever you’re trying to accomplish.

That leaves us with option 3, which I will illustrate in this article. But first, I’m going to try to talk you out of it.

You Really Shouldn’t Manually Copy the Disks of a Live Virtual Machine

Manually copying a live VHD/X file isn’t the greatest idea. The best that you can hope for is that your copy will be “crash consistent”. The copy will only contain whatever data was within the VHD/X file at the moment of the copy. Any in-flight I/Os will be completely lost if you’re lucky or partially completed if you’re not. Databases will probably be in very bad shape. I’m sure that whatever reason that you have for wanting to do this is very good, but the old adage, “Because you can do a thing, it does not follow that you must do a thing,” is applicable. Please, think of the data.

OK, the guilt trip is over.

Just remember that if you attach the copied disk to a virtual machine and start it up, the volume will be marked as dirty and your operating system is going to want to run a repair pass on it.

Manually Copying a Hyper-V Disk the Dangerous Way

That header is a bit scarier than the reality. Most importantly, you’re not going to hurt your virtual machine doing this. I tested this several times and did not have any data loss or corruption issues. I was very careful not to try this process with a disk that housed a database because I was fairly certain that would break my perfect streak that way.

Use robocopy in restartable mode:

The format is:

It is important that you do not use a trailing slash on the folder names! If you want to copy multiple files, just enter them with a space between each.

Pros of the robocopy method:

  • It’s easy to remember
  • It works on anything that your account can reach — local storage, CSVs, SMB shares, whatever
  • Is “good enough”

Cons of the robocopy method:

  • Restartable mode (specified by the /z switch) is sssssllllllllooooooooow especially over networks
  • There is no guarantee of data integrity. But, there’s no real guarantee of data integrity when manually copying a live VHD/X anyway, so that probably doesn’t matter.
  • I doubt that anyone will ever give you support if you use this method

For basic file storage VHD/X files, this is probably the best of these bad methods to use. I would avoid it for frequently written VHD/X files.

Manually Copying a Hyper-V Disk the Safer Way

A somewhat safer method is to invoke VSS. It’s more involved, though. The following is a sampledo not copy/paste!

This is going to need somewhat more explanation than the robocopy method. We’ll take it line-by-line.

The first line tells VSSADMIN to create a shadow copy for the C: volume. The VHD/X file that I’m targeting lives on C:. Substitute your own drive here. The shadow copy becomes a standard Windows volume.

The second line creates a symbolic link to the newly created volume so that we can access its contents with the usual tools. You can discover what that lines contents should be via the output from the previous command.

VSS Admin Create Shadow

VSS Admin Create Shadow

We use “mklink.exe” to create the symbolic link. The /D switch lets it know that we’re going to make a directory link, not a file link. After that, we only need to tell it what to call the link (I used C:vssvolume) and then the target of our link. It is vitally important that you place a trailing slash on the target or your symbolic link will not work.

Next, we copy the file out of the shadow copy to our destination. I used XCOPY because I like XCOPY (and because it allows for tab completion of the file name, which robocopy does not). You can use any tool that you like.

That’s all for the file. You can copy anything else out of the shadow copy that you want.

We need to clean up after ourselves. Do not leave shadow copies lying around. It’s bad for your system’s health. The first step is to destroy the symbolic link:

The last thing is to delete the shadow. There are multiple ways to do this, but my preferred way is to delete the exact shadow copy that we made. If you look at the output of your vssadmin create shadow command, it has all of the information that you need. Just look at the Shadow Copy ID line (it’s directly above the red box in my screen shot). Since VSSADMIN was nice enough to place all of that on one line, you can copy/paste it into the deletion command.

You’ll be prompted to confirm that you want to delete the shadow. Press [Y] and you’re all finished! If you want to see other ways to remove VSS shadows, type vssadmin delete shadows without any other parameters. It will show you all of the ways to use that command.

Yes, this works for Cluster Shared Volumes. Get a shadow of C: as shown above and copy from the shadow’s ClusterStorage folder. I would take caution to only perform this from the node that owns the CSV.

Pros of the VSSADMIN method:

  • It’s completely safe to use, if you do it right.
  • It’s not entirely perfect, but some quiescing of data is done. The copied volume is still dirty, though.
  • Faster when copying to a network destination than robocopy in restartable mode.
  • Works for local disks and CSVs. Won’t work for SMB 3 from the Hyper-V host side.

Cons of the VSSADMIN method:

  • Tough to remember (but this blog article should live for a long time, and that’s what Favorites and Bookmarks are for)
  • If you don’t clean up, you could cause your system to have problems. For instance, you might prevent your actual backup software from running later on in the same evening.
  • May not work at all if another VSS snapshot exists
  • May have problems when third-party and hardware VSS providers are in use

If the integrity of the copied data is important and/or changing frequently, you’ll likely get better results from the VSSADMIN method than the robocopy method. It’s still not as good as the other techniques that I promised not to harp on you about.

A Guide to Implement Hyper-V Checkpoints

A Guide to Implement Hyper-V Checkpoints

Hyper-V checkpoints are a double-edged technology. Used appropriately, they can be a quick and simple solution to any number of problems. Used inappropriately, they can cause an entire deployment to be brought to a catastrophic halt. To make it worse, many people are not entirely clear on just what a checkpoint truly is. This guide is meant to help you correctly Implement Hyper-V Checkpoints.

Snapshots or Checkpoints?

In the earliest versions of Hyper-V, as in many other hypervisors, this technology went by the name of “snapshots”. System Center Virtual Machine Manager has always used the name checkpoint. Starting in 2012, the term “checkpoint” was introduced to the Hyper-V tools. PowerShell has used both terms, even within the same version. With Windows 10/Windows Server 2016, you should see more uniform usage of the term “checkpoint”. The Hyper-V community seems to still prefer using “snapshot”. Whatever term you use, both refer to the same thing. The feature set has changed, but the general principle has not. I will use “checkpoint” throughout the remainder this post as it is official terminology.

What is a Checkpoint?

Checkpoints are often spoken of as if they represent a virtual machine past a certain point in time. In fact, the opposite is true. A checkpoint represents the entire state of a virtual machine exactly as it was at the time the checkpoint was captured. When I was a young adult, platform games evolved to include checkpoints. Your character would pass a checkpoint on the screen and if s/he died, s/he would start over from that point. Everything you had done after that checkpoint was lost. So it is with Hyper-V checkpoints.

Checkpoint Visualization

Checkpoint Visualization

Why Checkpoint?

The purpose of a checkpoint is to provide you with a very quick method to revert to a previous point in time. A scenario that I commonly use is patching; you capture a checkpoint, apply the patch, and then delete or revert the checkpoint depending upon the output. I do some development work that modifies the state of virtual machines, so I use complicated checkpoint trees to try out different iterations of my programs and to compare their effects.

What are the Dangers of Checkpoints?

Checkpoints present a few problems.

  • They rely on differencing disks. Every change that would be written to the virtual hard disk is instead written to this differencing disk, which resides alongside its parent. The differencing disk can potentially grow to the maximum size of its parent, which can be an issue in situations with limited drive space.
  • People sometimes try to tinker with the files, causing broken parent-child relationships that cannot be rectified without resorting to backup.
  • People erroneously expect checkpoints to serve as a backup, which often leads to heartbreak. I’ll come back to this in a bit, but the very short message is: checkpoints are not backups.

There is no “Active” Checkpoint

I commonly see people asking questions about “the active checkpoint”. I might have even used some form of that term myself. To be plain: there is no such thing as an “active checkpoint”. By definition, a checkpoint cannot be active. Only a virtual machine can be active. What people are really asking about here is the checkpoint that is the parent of the virtual machine. If people didn’t get so terribly perplexed by all of this, I would embrace “active checkpoint” as a term because it certainly distinct in the checkpoint tree. “Most recent checkpoint” doesn’t roll off the tongue quite as easily but it is much more accurate and less likely to cloud the situation.

The Anatomy of a Checkpoint

When a checkpoint is created, nothing inside the virtual machine is impacted in any way. The operating system continues to run as though nothing changed. No service is impacted. All changes occur in the files that support the virtual machine.

Virtual Hard Disks

For every virtual hard disk file that comprises the current virtual machine state, a corresponding differencing disk is created with the following attributes:

  • The newly created file belongs to the virtual machine. It does not belong to the checkpoint. If this is the first checkpoint taken, the checkpoint keeps the original VHD/X file.
  • The newly created file stays with its parent. Changing the virtual machine’s checkpoint location has no effect on AVHD/X file placement.
  • Performing a Storage Live Migration of the virtual hard disks of a virtual machine with a checkpoint causes all of the virtual hard disk files to move, original and checkpoint alike.
  • The extension of the new virtual disk file will be prepended with an “A”, so AVHD to match a VHD and AVHDX to match VHDX.
  • The file name is automatically modified so that a procedurally-generated GUID is between the original file name and the new extension.

These are differencing disks, so they will only contain any changes that would have been made to their parent disk file.

XML, BIN, VSV, etc.

Copies of the original files that make up the virtual machine’s configuration are copied to the specified Checkpoint File Location folder for the virtual machine. We showed you how to modify that location in an earlier blog post. Keep these facts in mind about these files:

  • In the past, some have attempted to “guide” you into breaking the checkpoint functionality by setting the default file location to a point that does not exist. In those days, it would cripple the technology and prevent checkpoints from being taken. Doing so is now an exercise in futility. If the specified location does not exist when a checkpoint is created, the files are created in the ProgramDataMicrosoftWindowsHyper-VSnapshots directory tree on the host’s system drive. That can be quite problematic if your system drive is low on space and can be catastrophic for highly-available virtual machines, so take care that the location is configured correctly.
  • The BIN file is the same size as the virtual machine’s memory usage, so it might require a lot of space. Plan accordingly when taking snapshots.
  • The XML and VSV files in the Snapshots folder contain the virtual machine configuration. That includes which drives are attached, the amount of memory and CPU assigned, and anything else you can find on the settings dialog. Keep in mind that reverting to checkpoints will reverse those changes as well as anything that happens to the virtual hard drive files.

Pass-through Objects

If you’re using pass-through disks, virtual SAN, or connecting iSCSI inside the guest, their contents are not affected. A pass-through device’s connection state is tracked. What that means is that if a pass-through device was connected at the time that a checkpoint was taken but then removed later, reverting to the checkpoint will cause Hyper-V to attempt to reconnect that pass-through device. If it is no longer available, the virtual machine will need to be reconfigured before it can start.

Because the contents of pass-through disks operate independently of checkpoints, your data will be in an inconsistent state.

Checkpoints are Not Backups, But…

The checkpointing system does not replace a backup system. We have a solid post on this topic already that I do not intend to rehash. However, you likely have a problem that you want it to solve, namely that you need a quick backup copy of your virtual machine, or at least a VHDX file. What you can use for that is the export system. If you want to export a checkpoint, you can do that as well. Just remember that the checkpoint is everything up until that point in time. I’ve seen a few people upset that an exported checkpoint didn’t contain everything after the checkpoint. If you want the active state of a virtual machine, Hyper-V now allows you to export a virtual machine while it’s running. In keeping with the tradition of the earlier article, exports are a poor substitute for backup for many reasons, but, unlike checkpoints, they actually duplicate data. If you’re trying to make a copy, checkpoints are not the correct solution.

Further Reading

We have written fairly extensively on the topic of checkpoints. Answers to all your questions should be contained in one of these documents. If not, let us know in the comments.

Live Backup Changes in Hyper-V Server 2012 R2

Live Backup Changes in Hyper-V Server 2012 R2

Quite some time ago, we wrote a post about taking live backups in Hyper-V. Hyper-V Server 2012 R2 really changed the mechanics of backup. This post examines how those changes have affected live, or hot, backups.

Until 2012 R2, backup was strictly based on VSS (Volume Shadow Copy Service) operations. Backup applications trigger VSS in the host. For standard backup operations of the file system, VSS responds by flushing buffers and pausing I/O. It also notifies any applications that had registered with VSS that a backup was about to occur, granting those applications the ability to perform any additional preparations necessary. One such application is Hyper-V.

Hyper-V in these earlier versions would simply use the Integration Services, specifically the backup service, to notify the guest’s VSS of an impending backup operation. It would perform the same operations as VSS in the host by preparing its own operating system and registered applications for a backup.

Once everything is prepared, the earlier versions would just use the standard VSS approach, which was to take a VSS snapshot. This redirects new I/O to a temporary location so that backup operations are free to read from the source without disturbing ongoing operations. Once the backup is complete, everything returns to normal.

What’s new in 2012 R2 is that instead of using the VSS snapshot system, Hyper-V relies on its on snapshot system, which is gradually being renamed to “checkpoints”, ostensibly to avoid confusion with the VSS system. Instead of placing the new I/O in the System Volume Information tree the way that a standard VSS backup does, Hyper-V creates AVHD(X) files for the backup to use. For standard Windows guests, you won’t see any effective changes. Guests that don’t use backup services will have a great deal of change, assuming your backup program knows how to work with the new system. Sadly, a great many still don’t, even though this has been publicly available for over a year and was in public beta for a time before that. If you’re having trouble with getting live backups of your guests, it might be time to look into getting a new backup application.

For this posting, we’re going to take a look at some of the operations that would have been problematic for a live backup in 2012 and earlier. We’re going to use Altaro Backup for Hyper-V, since it works just fine with the new checkpoint system and since this is an Altaro blog.

Windows Guests with Dynamic Disks

In 2012 and earlier versions, it was impossible for a live backup to be taken of a guest with Dynamic disks. By Dynamic, I don’t mean dynamically expanding VHD(X) files; dynamically expanding disks are, and always have been, backed up using the same rules as fixed VHD(X) files. Rather than use a lot of words to explain what I’m talking about, here’s a screenshot of Disk Management inside my test system:

Dynamic DiskBefore, this virtual machine would be saved while it was backed up because the E: drive is Dynamic. As a 2012 R2 guest, it continues running. During my test, the virtual machine did stop accepting input through the virtual machine console for a few moments, but Hyper-V Manager never changed the status of the VM from “Running”. Keep in mind that my test system is quite low-powered, so I’m not sure how big of a deal to make of the brief interruption. Importantly, the uptime counter of the machine was not reset, so this is considered a “live” backup.

I was not able to definitively answer my second question, which is, is VSS in the guest triggered for the contents of that Dynamic volume? Documentation on previous versions said that it would not be, but that was because the virtual machine would have been in an offline state during the backup and unable to receive messages from anything. Since the virtual machine stays online now, it should be receiving the VSS request through the Integration Services. The Dynamic disk is listed as eligible for VSS operations:

Dynamic VSS

 

Furthermore, an AVHD(X) file is created for the Dynamic disk just like the Basic disk:

Dynamic Disk During BackupI don’t have any method to really test this, but the evidence at hand seems to indicate that virtual machines with Dynamic disks will now be backed up online, as long as your backup application supports the new features of 2012 R2.

Windows Guests With Disabled Integration Services

The next thing I did was disable the Integration Service for backup and try another backup. It worked just the same as it did in the previous test. Of course, its contents would be backed up in crash-consistent rather than application-consistent fashion as it has no way of knowing that a backup is occurring.

Linux Guests

There’s really good news if you’ve got Linux guests… or, at least as good as it’s likely to get. Backup for Linux guests in 2012 R2 works the same way that it does for Windows guests with the backup Integration Service disabled. I tested this with the current version of Altaro Hyper-V Backup (4.1.40.0). The backup worked without any interruption at all, even without selecting the “Enable Hot Backups for this Non-VSS Aware Guest VM”. When I did a restore, the data was in a crash-consistent state. Linux doesn’t have anything like VSS at this time, so this is the best that can be expected. The big thing here is that, as long as your backup application is fully compliant with the new backup system of 2012 R2, backups of Linux guests will automatically be live.

Application-Consistent vs. Crash-Consistent

If you’re not familiar with these terms, some of the first articles I ever wrote for Altaro described the difference in detail. The shorter version is that “crash consistent” means that the backup will be in the exact same state that it would be if the backed up machine had simply had its power removed at the moment that the backup was taken. Anything that was in-memory at the time would be lost. This is especially problematic for database systems that don’t have protections against that sort of thing, and still means data loss for those that do. Under Windows, these applications have the ability to register with VSS. When a VSS operation occurs, it notifies these applications so that they can prepare for the event. This usually involves clearing all pending operations. This way, when the backup occurs, there will be no activities in memory that can be lost. This style of backup is called “application consistent”.

 

Hyper-V Replica Requirements and Facts

Hyper-V Replica Requirements and Facts

For an introduction check out the article What is Hyper-V Replica and how does Replication Work. It explains what the benefits are of using it in a production environment as a disaster recovery technology.

Hyper-V Replica Requirements

To take advantage of the Hyper-V Replica, which is included as part of the Hyper-V server role, the following pre-requisites must be met:

  • A Windows Server 2012 computer with Hyper-V Role enabled/installed.
  • The hardware that supports the Hyper-V Role.
  • Sufficient storage on both the Primary and Replica servers to host the files used by virtualized workloads.
  • Network connectivity between the locations hosting the Primary and Replica servers
  • Properly configured firewall rules to permit replication between the Primary and Replica sites
  • For data to be transferred in encrypted formatted over the network, you are required to use HTTS which requires an X.509v3 certificate which supports mutual authentication.

Hyper-V Replica Facts

  • Change Tracking Module

Hyper-V Replica works on a module named “Change Tracking”. This is a function, in a programming language or a routine which is implemented to look for any write operations on the Virtual Machine files.

  • Two Components

There are at least two Hyper-V Servers involved when configuring Hyper-V Replica feature.  One server acts as Primary Server and other acts as a Replica Server as shown in the below screenshot:

  • Windows Server 2012

Hyper-V Replica requires Primary and Replica server to be running on Windows Server 2012.

  • Workgroup or Domain Models

There is no need to have Active Directory domain Services configured for Hyper-V Replica to work. Hyper-V Replica can be implemented in both Workgroup and Domain security models. For Workgroup security model, authentication must be configured using a certificate.

  • Disaster Recovery Solution

Many people confuse with Hyper-V Replica feature as to see how it is beneficial in achieving the high availability of Virtual Machines running on the Hyper-V. Hyper-V Replica is not a failover solution. It is a disaster recovery solution meaning it does not provide automatic failover capability which is provided by the Microsoft Failover Clustering.

  • Use of Storage

Hyper-V Replica is designed in such a way that it makes the scenario work irrespective of where the virtual machine VHD file(s) resides (VHD files can be hosted on Direct Attached Storage (DAS), a SAN LUN, an SMB share on a File Server, or a Cluster Shared Volume (CSV). Hyper-V Replica can be configured to transfer Virtual Machines across Hyper-V Hosts.

  • Replication Interval

Replication Interval provided by the Hyper-V Replica feature is 5 minutes and interval cannot be changed. There are several reasons for imposing this limitation but explaining those in this article is out of scope.

  • HTTP or HTTPs Replication

The HTTP or HTTPS authentications are provided by the Network Module implemented in the Hypervisor.

  • Different Networks

Hyper-V Replica can be configured for different networks also. It allows you to configure Hyper-V Replica for Replica Server running in the different network subnet.

  • PowerShell Interaction

PowerShell is just robust as Unix shell today. Microsoft offers one-liner PowerShell commands to configure Hyper-V Replica quickly without spending too much of time in GUI.

  • Firewall Exception

A firewall exception is mandatory for Hyper-V Replica. Please note default port 80 for HTTP and 443 for HTTPS are already enabled in the Windows Firewall. The firewall rules just need to be enabled.

  • Use of VSS

Hyper-V Replica uses Volume Shadow Copy Service (VSS) on servers to take a point-in time snapshots of the Virtual Machines. You can configure Snapshots to be taken when you enable Replication for a Virtual Machine.

  • By default No Virtual Machines configured to replicate

By default, Virtual Machines are not configured to replicate. You must configure Virtual Machines to be replicated on the Primary Server which are replicated over the network to Replica Server located on the destination site.

Please stay tuned for Part 3 of this article series, in which we’ll cover how to enable and configure Hyper-V Replica.

In Part IV, we’re going to learn about Hyper-V Replica Vs Hyper-V Virtual Machine Backup and why Hyper-V Replica should not be considered a replacement for Virtual Machine backup products!

 

Backing Up and Restoring Virtual Machines in Hyper-V

Backing Up and Restoring Virtual Machines in Hyper-V

The Volume Shadow Copy Service (VSS) is tightly integrated with Windows XP and later versions of the Windows Operating Systems. Microsoft developers put together the efforts to design a centralized component which allows Windows applications (Windows Server Backup) and third party applications to initiate backup or restore requests for Hyper-V Virtual Machines with the help of Volume Shadow Copy Service. (more…)

Hyper-V Snapshots vs. VSS Snapshots Explained

Hyper-V Snapshots vs. VSS Snapshots Explained

Ah, the joys of encountering distinct technologies that share a name. There’s nothing at all confusing about that, is there? Today, we’ll look at the differences between Hyper-V’s snapshots and VSS’s snapshots. They do have more in common than just a name, but they have far more differences.

(more…)

Live Backup of a Hyper-V Guest VM with Hyper-V VSS Writer

Live Backup of a Hyper-V Guest VM with Hyper-V VSS Writer

An earlier posting, “Backing Up Hyper-V Guests – Host-Based vs Guest-Based Methods”, discussed the differences between backing up virtual machines as though they were physical computers against utilizing software installed within Hyper-V’s parent partition to back up virtual machines as complete resources. When using the latter method, some backup software has the ability to coordinate with Hyper-v’s VSS writer to back up virtual machines without taking them offline. The considerations for enabling this to occur are examined in this article. (more…)

VSS Crash-Consistent vs. Application-Consistent VSS Backups (post 2 of 2)

VSS Crash-Consistent vs. Application-Consistent VSS Backups (post 2 of 2)

When is Application-Consistent Backup Vital? Not all situations require an application-consistent backup. Things such as file and print servers will be fine with crash-consistent and possibly inconsistent backups. If your application doesn’t provide a VSS writer, there might not even be a way to get an application-consistent backup of it while its containing machine is live. The most common need for application-consistent backups is the usage of database-backed applications.

NOTE: This is the second blog post in a 2-series post.  You can read the first post in this series here.

When is Application-Consistent Backup Vital?

Not all situations require an application-consistent backup. Things such as file and print servers will be fine with crash-consistent and possibly inconsistent backups. If your application doesn’t provide a VSS writer, there might not even be a way to get an application-consistent backup of it while its containing machine is live. The most common need for application-consistent backups is the usage of database-backed applications. In particular, Microsoft Exchange and SQL Server are fairly dependent upon application-consistent backups. In their default configurations (non-circular logging and full recovery modes, respectively), they do not normally write directly to their database files but instead rely upon log files. When their VSS writer is triggered, they flush all pending operations from memory and commit the contents of their log files into the main database files. This behavior ensures that a crash during normal operations will pose a risk to the active log file, not the primary database. If application-consistent backups are not taken on a regular basis and these applications are left in their default configurations, the log files will eventually consume all available space. Even if transaction quantity is low enough and disk space ample enough to prevent consumption from being a major concern, restoring either system from a crash-consistent state is much more painful and risky than restoring from an application-consistent state.  Also, if these applications are left in their default configuration and backed up using a non-application-consistent method, even if it is a full image-level backup, they will not commit their logs.

Backup Operations in Hyper-V

Looking at the simplistic file-based layout of a virtual machine, it is tempting to assume you can use any available file-based backup or even copying software. Unless you are shutting down or saving the virtual machines, avoid falling into this trap. If you are saving the VMs, ensure you are including the VSV and BIN files in your copies. Even if you follow all these steps, Hyper-V takes a particular ownership of its virtual machines and won’t like it if you try to simply copy in the files of a virtual machine that were running on a different host. If you intend to back up the virtual machines without saving them, ensure that whatever method you utilize triggers VSS. There are a handful of Internet posts suggesting that you can simply run copying tools like Robocopy against your VHD files and everything will be fine. Without calling on VSS to pause I/O first, this technique can easily result in a VHD that is unusable. If you intend to use nothing but freely available tools and scripts, research things like Diskshadow and implement your scripts properly.

Hyper-V has its own VSS writer. This deceptively simple sentence means that by deploying on Hyper-V, you automatically have the ability to get application-consistent backups of virtual machines from the host level. There are some conditions to be met in order to ensure this works as expected, but the basic operation is that when Hyper-V’s VSS is triggered, it notifies VSS within the virtual machine, which in turn notifies all of its registered VSS writers that a backup is taking place.

No VSS Writer Available?

In some cases, you need an application-consistent backup but there is no VSS writer available. One example of this is MySQL. Hyper-V backups of virtual machines containing MySQL will always result in either a crash-consistent or an image-level backup. For MySQL, the latter is probably acceptable as MySQL doesn’t perpetually expand the log file. However, if you’re using MySQL within a VSS-aware VM, then a Hyper-V-based backup tool is going to take a crash-consistent backup. MySQL (like any other database system) isn’t always recoverable from a crash-consistent backup; even when recovery is possible, it may be painful. MySQL is just one example; any number of line-of-business applications could tell a similar tale. In the case of MySQL, one solution is to find a guest-level backup application that is MySQL-aware and can back it up properly. For applications for which no backup application has a plug-in, you may need to have pre- and post-backup scripts that stop services or close applications. If brief downtime is acceptable, you can disable the Backup item in Hyper-V Integration Services, thereby forcing Hyper-V to save the state of the VM during backup. This technique results in an image-level backup and can be used on any application that doesn’t have a VSS writer.

Hyper-V-Backup-Volume-Snapshot

General Requirements for Application-Consistent Backups in Hyper-V

Note that some backup software contains its own methods to ensure application consistency and may be able to circumvent some or all of these rules. Please consult with your application vendor for further details.

  • The VM must have a current version of Hyper-V Integration Services installed and the Backup component must be enabled.
  • The VM must have a compliant implementation of VSS (Vista Enterprise or later for desktop operations and Windows Server 2003 SP2 or later for servers)
  • All of a VM’s VHDs must reside on the same volume (whether it is local or CSV)
  • All of the VHDs must be formatted with NTFS. This is a limitation even in a physical environment as VSS does not interact with FAT volumes.
  • None of the VHDs can contain Dynamic volumes; they must all be Simple (not to be confused with Dynamic vs. Fixed VHD, either of which can be used)
  • The volume holding the VHDs must have enough free space for the VSS snapshot; volumes contained by the VHDs must also have enough space to hold their VSS snapshots. The exact need will vary based on several conditions but the safe line is at 20%.

Special Considerations for Cluster Shared Volumes

With VSS operating at the volume level, it might seem that a Cluster Shared Volume is automatically covered, but this isn’t precisely true. VSS does operate on volumes, but it is constrained to the host it is running on. A CSV is simultaneously connected to multiple hosts. In normal mode, only the metadata for a CSV is handled by a single host; all other virtual machine data can be directly accessed by the host that owns it. Therefore, when VSS quiesces a CSV and takes a snapshot, no other host in the cluster is aware of it. In this instance, the volume-level nature of VSS can be a threat. Even if you’re specifically calling VSS on a host to back up only a VM owned by that host, VSS is snapping the entire CSV. No other host is aware of the VSS operation and will carry on with its I/O operations, so the entire status of the VSS snapshot is questionable. Furthermore, and much more dangerous, there is nothing preventing another host from initiating a VSS snapshot while the first is still being used. Partly to address this situation, Microsoft implemented “Redirected Access” mode. In this state, all I/O for a given CSV is handled by the coordinator node. If any other host in the cluster wishes to access that CSV, its I/O is encapsulated in SMB blocks and funneled through the coordinator node. This ensures that the VSS snapshot is not disturbed and that it is not trampled on by another VSS operation.

If you are using CSVs, your backup solution absolutely must be able to turn on redirected access mode. There is no guaranteed safe way to copy virtual machine data off of a CSV without turning on redirect access unless the VSS snapshot is not used and the VM is saved or turned off.

Hyper-V-VSS-Cluster-Shared-Volumes-CSV

NOTE: This is the second blog post in a 2-series post.  You can read the first post in this series here.

Have any questions?

Leave a comment below!

 

Backing up Hyper-V

If you’d like to make backing up your Hyper-V VMs easy, fast and reliable, check out Altaro Hyper-V Backup v4. It’s free for up to 2 VMs and supports Hyper-V Server 2012 R2! Need more? Download a 30-day trial of our Unlimited Edition here: https://www.altaro.com/hyper-v-backup/.

 

(Don’t worry, we hate spam too!)

VSS Crash-Consistent vs Application-Consistent VSS Backups (post 1 of 2)

VSS Crash-Consistent vs Application-Consistent VSS Backups (post 1 of 2)

When designing any IT solution, many administrators often consider “Backup” to be little more than another box on a long list of items to check off. They verify that the software and hardware they’re using will handle the load, configure it to back up on a reasonable schedule, and forget about it. Some will take the extra step of restoring some data to an alternate location as a test. Hardly any go through the full exercise of simulating an actual catastrophe. Most of the time, this practice is completely harmless. Unfortunately, if disaster does strike, there are often more questions than answers. Planning ahead is critical, and that involves knowing what sort of backup you need and if your backup application can provide it.

NOTE: This is the first blog post in a 2-series post.  You can read the second post in this series here.

Consistency Definitions

To determine what sort of backups you need, you must first understand the terminology in use. The backup types applicable to this discussion are inconsistent, crash consistent, and application consistent.

Inconsistent
Crash-Consistent
Application-Consistent
Image
Live backup clip_image001[16] clip_image001[17] clip_image001[18] clip_image002
Interdependent files guaranteed to be the same version clip_image002[1] clip_image001[19] clip_image001[20] clip_image001[21]
Protection against transactional data loss (application-aware) clip_image002[2] clip_image002[3] clip_image001[22] clip_image003

 

Inconsistent Backup

An inconsistent backup is the oldest type. Backup software starts at the beginning of the file structure and copies out data until it gets to the end. If any file changed after it was backed up but before the job completed, then the result is an inconsistent backup. It means, simply, that the files contained in the backup job are mismatched. In some instances, this isn’t a problem. Inconsistent backups are often acceptable for things like simple file servers and data locations that rarely change. These can be dangerous to the point of uselessness for database systems. Databases are typically composed of multiple interdependent files, so they must all be synchronized in order for a backup to be considered successful. Also, an inconsistent backup method only grabs what is on disk at the time; if a database has a pending operation in memory, it is completely lost.

A software program can take an inconsistent backup at any time without any real interaction with anything except the operating system. It may struggle if it isn’t running with sufficient permissions to grab a file or encounters a file that’s locked for reading, but otherwise it’s just another process running on the system.

image

Crash-Consistent Backup

The most important distinction between a crash-consistent backup and an inconsistent backup is that all data within a crash-consistent backup set is captured at exactly the same time. This is the most common method in use by commercial backup software today and is almost always sufficient for applications that do not rely on a database. One thing it shares with the inconsistent method is that it does not capture the contents of memory or any pending I/O operations. If a crash-consistent backup is restored in its entirety, then the data will be in the same state it would have been if the system had crashed at the exact moment that the backup was taken. This is reason it is considered “crash-consistent”. If a database system is restored to a crash consistent state, then it is necessary to follow any procedures that would be followed if the system had actually crashed at that point. Many applications, like Active Directory, have an automated recovery mechanism and will attempt to handle the problem without administrator intervention. If these automated systems aren’t successful, ensure that you know what the application vendor’s process is for crash recovery and be prepared to follow those steps. For Microsoft Exchange, you may need to know how to set up a recovery group and use its utilities to integrate log files. For Microsoft SQL, you may need to know how to replay logs into a database file.

A crash-consistent backup requires more effort on the part of backup software than inconsistent backups. Techniques vary, but in the Windows world, the Volume Shadow Copy Service (VSS) was designed as a component of the operating system specifically to assist backup software. A backup application coordinates its efforts with VSS. When triggered, VSS pauses I/O on the volume and takes a block-level snapshot of it, and then the backup software pulls its backup from that snapshot at its leisure.

image

Application-Consistent Backup

An application-consistent backup is the most involved but also the most desirable. Again, techniques vary, but the general process for Windows applications is for the application manufacturer to provide a VSS writer. When the VSS service is triggered, it will notify these writers that a backup is occurring. It’s then up to the VSS writer how to handle it. In general terms, a database application will respond to its VSS writer being triggered by flushing all of its memory and I/O operations so that the database is completely consistent. In doing so, there is nothing in memory and no pending I/O to be lost. A proper VSS writer should effectively place all the data for an application in the same state it would be if the application were properly closed. When the VSS snapshot is complete, it signals the VSS writers, which are then to resume normal operation of the attendant application while the backup software safely copies out of the snapshot.

If an application does not provide or properly register a VSS provider but its data resides on a volume with VSS enabled, the data is backed up in a crash-consistent state.

Image-Level Backup

An image-level backup is quite different from the other types. All previously mentioned methods operate in some fashion within a machine that is actively running. Since no backup software can properly capture the active state of memory and I/O, the most reliable way to completely replicate all aspects of a machine is to shut it down. Then, the storage medium can be copied out block-by-block and there is absolutely no concern of anything being lost. Virtualization does add another option; Hyper-V manifests this as “Saved State”. This is similar to hibernation on a physical machine, except that the virtual machine itself is unaware of the state change. I/O in the virtual machine is paused and the contents of memory are copied to a file on disk. As long as every single file related to the virtual machine is copied, a restore of that backup will result in a virtual machine that is in exactly the same state it was when it was saved.

NOTE: This is the first blog post in a 2-series post.  You can read the second post in this series here.

 

Page 1 of 212