How to Back Up and Restore Domain Controllers Virtualized on Hyper-V

Microsoft’s Active Directory technology enables system administrators to group large numbers of computers together inside security boundaries. The directory holds a potentially vast amount of information on users, computers, and all other types of objects. Any object with proper security rights can query the directory and retrieve information. This directory allows for single-sign-on, easily controlled access to resources, software deployment, group policy management, and many other benefits over the older workgroup model. The directory’s layout, called the “schema”, can even be extended to hold other data types so that third party software can utilize the repository as well. Management of Active Directory requires a special type of server called a domain controller. This server is responsible for handling queries and keeping the directory up-to-date. Microsoft had domain controllers in earlier versions of Windows Server, but with the introduction of Active Directory in Windows 2000, domain controllers ceased working under the single-master/multiple-slave model and began operating under a multi-master model. This means that in a domain with multiple domain controllers, objects can be changed on any domain controller. Changes are periodically synchronized between domain controllers using active directory replication technology.

Identifying the Challenge

Most of the intricacies and details of the multi-master Active Directory operation are well beyond the scope of this article, but some explanation is necessary to understand the special challenges domain controllers pose for both backup and restore operations. Every domain controller is considered “authoritative”; that means it can make changes to the directory without gaining consent from the others or having a centralized domain controller handle update operations. To do this, each domain controller stores a local copy of the directory and controls updates to it. To keep track of changes, Active Directory assigns an Update Sequence Number (USN) to every attribute for every object that it is tracking. These aren’t easily viewable, but some USN-related attributes can be seen in the ADSI Edit screenshot below:


Figure 1: ADSI Edit showing object attributes

When a domain controller updates the attributes for an object, it modifies them and a local copy of the attribute’s USN which is based on the domain controller’s own USN. When it replicates with another domain controller, they determine whether or not they need to update objects from remote servers based on USNs. If an object attribute on a remote domain controller has a higher USN that it did during the last replication cycle, then the local domain controller knows it needs to retrieve that data. In the event of a collision, other mechanisms are in play to determine which attribute update is kept (usually, most recent wins).

Challenges of Virtualization

With the multi-master model, all domain controllers are considered “authoritative”. With the exception of a handful of operations, no single domain controller is ever “in charge”. So, if problems occur in the directory, there isn’t a simplistic way for domain controllers to “work it out amongst themselves”. In most cases, there won’t be a problem, but virtualization adds some potential issues. When Active Directory was first introduced, no one was virtualizing anything. The biggest uncontrolled threat to Active Directory was a system crash. Microsoft built in a mechanism so that if a domain controller crashes, when it comes back up, it essentially reaches out to all the other domain controllers and says, “I was out for a bit, what did I miss?” Replication is not hindered. However, a virtualized domain controller can be paused or suspended for an indefinite length of time. When it comes back, the only thing it’s going to know is that its clock has changed. This has the potential to behave exactly like restoring an old copy of Active Directory into a functioning forest. This condition is discussed in the “Challenges of Domain Controller Restores” section.

Challenges of Virtualized Domain Controller Backup

Running a traditional-style backup that specifically triggers the VSS writer to operate on the System State of a domain controller ensures that Active Directory knows it’s been backed up and therefore the consistency of the Active Directory database is guaranteed.  Backing up the virtual machine from the host level using a method that does not trigger the guest’s VSS writer does not result in a state that’s guaranteed to be accurate. More details are given in the “Challenges of Domain Controller Restores” section.

Challenges of Domain Controller Restores

There are multiple issues around restoring a domain controller, many of which can be problematic even in a non-virtualized environment. Most of them revolve around restoring an old copy of the directory database without properly preparing Active Directory.  This happens when you use backup software that doesn’t properly trigger the VSS writer or when a paused/suspended virtualized domain controller is started. The serious part of the problem is that there’s not a simplistic way to know that a problem has occurred. It’s highly probable that if you attempt to restore or bring an old domain controller out of pause state that it will tell you that everything is fine. If you’re lucky, the other domain controllers will notice that something is amiss and the newly restored domain controller will mark itself as being in a USN Rollback state and will stop participating in replication. That will generally require a rebuild of the domain controller; bad, not as bad as the alternative. If the other domain controllers don’t realize anything happened, then objects may exist on the domain with different attributes that will never be properly reconciled because the various domain controllers may not realize that they don’t agree with each other. A more serious possibility is that objects that the revived domain controller might have active copies of objects that other domain controllers had deleted. These objects (called “lingering objects”) will be returned to active status and replicated to the other domain controllers.

How to Approach Virtualized Domain Controller Backup and Restore

The most important thing is to select a backup application that is Active Directory-aware. Not all programs are marketed this way; if your application is VSS-aware and can specifically back up the System State of a Windows machine, it should work. Because this article specifically has Hyper-V guest machines in mind, then the application must be able to work with Hyper-V’s VSS writer so that it can communicate with the guest’s VSS writer. Also, in the virtual machine’s property dialog, on the Integration Services tab, the “Backup” integration component must be enabled. If all of these requirements are not met, then Hyper-V will pause the domain controller while it is being backed up. If that domain controller is ever restored, you run the risk of being in a USN rollback condition.

As mentioned earlier, restores are the dangerous part and something you want to avoid if at all possible. If you are recovering from a catastrophe and any domain controller survived with a suitably recent copy of the directory, the preferred recovery method is to have it seize all FSMO roles, make it a global catalog server if it wasn’t already, and delete the missing domain controllers from Active Directory Sites and Services. After that is complete, just deploy new domain controllers. If no domain controllers survived at all, then the recommended approach would be to restore only one domain controller and proceed as though it was the sole survivor mentioned earlier. Even if its copy of the database has some problems, it won’t cause a USN rollback if it’s the only domain controller still standing and it won’t be as problematic as potentially introducing multiple inconsistent domain controllers.

Of course, sometimes, you just have to restore a domain controller into an existing forest amid other functioning domain controllers. In that case, you need to ensure that your software is Active Directory aware (hopefully you did this before needing to restore). The only thing required to bring back the directory itself is a restore of System State data. If you are going to be restoring a virtualized domain controller using some sort of “bare metal recovery” method, then you need to double-check with your software vendor that it is designed specifically to handle virtualized domain controllers in this scenario. If a domain controller is not notified that a restore took place, then it will operate as though it is current and this will always trigger a USN rollback condition.

How to Verify Your Software is Active Directory-Aware

After a backup, check the domain controller’s event logs. Look under “Applications and Services”->”Directory Services” for Event ID 1917. If you find it, this is an absolute guarantee that your software is properly triggering the VSS writer. However, some applications still take a consistent backup of Active Directory without generating this event, as long as they trigger the VSS writer. Ensure that your application is set to at least take a System State backup and use the VSS writer, then look in the “Application” log for several Event 2001 and Event 2003 entries generated by ESENT. If there are no associated errors, then your directory is being safely backed up.

After you restore a domain controller into a domain with other active domain controllers, immediately check its “Directory Services” event log for ID 1109. If it is not there, disconnect that domain controller from the network as soon as possible or it could lead to a USN rollback condition.

I Only Have One Domain Controller. How Does this Affect Me?

If you only have one domain controller, then there’s not as much to worry about. It could crash without a commit and lose some data, but that is a potential problem regardless of how many domain controllers you have. Since a lone domain controller doesn’t participate in replication, a USN rollback state will not corrupt the directory or force you to into needing to demote/promote any domain controllers. It just means that later changes are lost. USN rollbacks are very rare in practice, especially if administrators are sufficiently educated on the subject, so the benefit of never having a USN rollback is completely eclipsed by the benefits of having multiple domain controllers. Only the smallest domains should ever operate with only one domain controller.

Can’t I Just Never Back Up and Instead Rely on Multiple Domain Controllers?

If your domain controllers are split across multiple physically separate sites and replication is reliable and consistent, there is a temptation to just not back them up. You should be protected from most environmental and physical disasters. However, without a backup, you aren’t protected from human disasters or malice. Always take backups.

I Can Rebuild the Domain with One Restore, so Should I Only Backup One Domain Controller?

You should back up several domain controllers, the more the better. Even though you only need to restore one, you cannot predict the circumstances that will require a restore or if any given backup will be successful and remain uncorrupted until needed. The more backups you have, the more likely you are to have the backup that you want. It is always better to have too many backups than not enough.

Acknowledgements

The source for much of the research in this article is a piece by Sander Berkouwer at “The things that are better left unspoken”. You can read the original article as well and download a PDF with more detailed explanation of USNs and how they are impacted by backups and restores.

 

Altaro Hyper-V Backup
Share this post

Not a DOJO Member yet?

Join thousands of other IT pros and receive a weekly roundup email with the latest content & updates!

9 thoughts on "How to Back Up and Restore Domain Controllers Virtualized on Hyper-V"

  • Craig says:

    Hi There,

    Good article but my question is about AD restore in a non-virtualised environment.

    We have 3 DCs in 3 sites. One 2k8 DC (main office) holds all the FSMO roles (yeah I know that’s a bad idea but the 2 other DCS are ancient 2k3 boxes). We currently backup the 2k8 DC using Acronis. We have set Acronis to use VSS so the NTDS VSS writer should be backing up AD properly. In the event of the Wk8 DC going down what should we do to restore?

    My thinking is that we would:

    1. Seize the FSMO roles to another DC.
    2. Do a bare metal restore of the 2k8 DC
    3. Wait and cross fingers.

    Is this correct?

  • Eric Siron says:

    Hi Craig,
    Caveat emptor: this is sort of off-topic for this particular blog, but I’ll try to help.

    No, do not do those steps. If you ever seize the FSMO roles from a domain controller, you must never bring that domain controller online again. Ever. Either seize the roles and rebuild your 2k8 DC from scratch *or* restore it. Never both. For more information, search for “Directory Services Restore Mode”.

  • Dirk says:

    Hi Eric,

    a really good article. Thanks for that.

    I am just missing the statement “Yes, our Altaro Hyper V Backup is AD aware and its safe to restore an backed up VM besides other existing domain controllers”.
    Because this is exactly what I need to do.
    There are two DCs a 2008 R2 and a second one with HyperV 2012 R2.
    The hardware machine with hyper v 2012 now is broken.
    After the repair I would like to restore the DC from my last backup but of course I want to avoid an USN rollback.
    Maybe this informations are provided but I dont find anything about that.
    Regards

    • Eric Siron says:

      “Active Directory aware backups” is something that is only an issue when you are backing up from within the ADDS’s operating system directly, and even that literally only means “System State aware backup”. When backing up a virtual machine that contains Active Directory, there really is no such thing as “Active Directory aware” because they all capture the System State. Altaro does trigger VSS in the guest, so it will know that it has been backed up, but that isn’t the point either.
      If you restore a domain controller virtual machine using Altaro VM Backup and do nothing but turn it on, that is a non-authoritative restore. Any other systems with newer USNs will overwrite whatever is in the restored VM’s AD database. If it’s a really old backup, then tombstoned items might be reanimated, but that’s not a USN rollback.
      If you want to perform an authoritative restore, then you are looking for an entirely different article: https://www.altaro.com/hyper-v/use-altaro-vmbackup-perform-authoritative-restore/.
      But, if you have two good DCs, then decomm and replace the sick one. Much easier and safer than a restore.

      • Dirk says:

        Thank you for your response.

        I do know your mentioned article.
        But I am exactly looking for the possibility to do a non-authoritative restore.
        So thanks again for your confirmation.

        Small Server in small businesses are often high integrated DC with further functions and centralised applications. So it isn`t the easy way to decomm and replace because there are further applications to be considered.
        Best regards

  • Dmitry Kudlaenko says:

    Hi, Eric! Thank you for the interesting post. Could you advice what will be the best practice in the following situation:
    I’m planning to deploy my Hyper-V environment like this: 1 hyper-v host with Hyper-V server 2016 installed; this host has 2 VM, one of them is DC and the 2nd machine is used for SQL-based 3rd part software. These 2 VMs are covered by single win2016Std license, as I remember. I also have QNAP NAS joined to the domain, which I used for backup storage. I suppose the ideal decision for backup here is to use Altaro software which allows to backup 2 VMs for free. My question is: where should I install Altaro software? Is it better to install on the machine with AD services or the 2nd machine? Will it have any impact to other services running on the same machine? Thank you a lot.

    • Eric Siron says:

      Install the primary Altaro software on the Hyper-V Server host directly. The management console can be installed anywhere you want.

Leave a comment or ask a question

Your email address will not be published. Required fields are marked *

Your email address will not be published.

Notify me of follow-up replies via email

Yes, I would like to receive new blog posts by email

What is the color of grass?

Please note: If you’re not already a member on the Dojo Forums you will create a new account and receive an activation email.

cyber security roundtable