Virtualized Domain Controllers: 4 Myths and 12 Best Practices30 Aug 2016 by 19
Domain controllers don’t mix with virtual environments — or do they? A mountain of conflicting information exists on this topic, and few of us have time to make the expedition over all of that territory. What you’ll find here is a guide map that will take you safely past the traps of myth straight to the pinnacle of best practices for your virtualized domain controllers.
Virtualized Domain Controllers: Myths
You’ll have to get through the fiction before you can get to the facts. Many outdated truths, fear, and blind guesses have led to the creation of abundance of misinformation on the subject of virtualized domain controllers.
Myth 1: Domain Controllers Should Not be Virtualized
We might as well start with this catch-all myth. Whatever arguments, whatever anecdotes are supplied as support, they are insufficient. We’ll look at many of those in the next sections, but there is simple proof that this is a myth: many, many institutions are virtualizing their domain controllers with success.
Myth 2: You Must Keep at Least One Physical Domain Controller
While this might have had some truth years ago, it’s not a big issue today. The reasons that people used to maintain physical domain controllers had more to do with the comparatively primitive state of virtualization. Failures and issues were more common. Even when failures didn’t occur, virtualization was still a young and somewhat unknown quantity. It only made sense to be distrustful.
Times have changed. If you have a physical domain controller, I wouldn’t get in a hurry to get rid of it. If you don’t, or don’t want to keep one, move ahead. Any fears that you have should be safely alleviated by the regular backups that you’re going to take.
Myth 3: A Domain Controller on Hyper-V is Vulnerable to the “Chicken and Egg” Problem
Of all the myths around domain controllers and Hyper-V, the most tenacious is the notion of the “chicken and egg”. I’ve even talked to MVPs that believe this one.
Here’s the format of the myth: a domain controller is a guest on a Hyper-V system that is part of its domain. When the host starts, it can’t talk to a domain controller. So, the universe implodes and everyone becomes Justin Bieber fans. Or something. I’m not really sure what they think is supposed to happen, but the faithful of this myth are certain to their core that Something Bad(TM) will occur.
The problem for them is, this myth is demonstrably false, and ridiculously simply so. Anyone with a Hyper-V-capable physical machine and access to a trial copy of Windows Server can disprove this one in under an hour. What irks me is that anyone who has much experience with Active Directory shouldn’t need to have this proven to them. These are the reasons that there is no “chicken and egg” problem:
- Hyper-V is a Microsoft kernel. No Microsoft kernel requires access to a domain controller in order to start. Anyone that has taken their laptop home from work can demonstrate this. It is possible that you won’t be able to log on as a domain user, but that doesn’t mean you can’t log on as a local user. Here’s the bigger implication that myth-believers seem to ignore: if you see a logon screen, Hyper-V is already started. That’s how type 1 hypervisors work. They start, then they start the management operating system.
- The service that you care most about, VMMS.EXE (Hyper-V Virtual Machine Management Service) runs under the Local System account. It doesn’t care one bit about a domain. If you configured it to run as a domain user, you’re going to have other problems anyway.
SMB 3 and the “Chicken and Egg”
There is one condition in which you could encounter a true chicken and egg scenario with Hyper-V and domain controllers. SMB 3 requires active authentication. A chicken and egg scenario could occur if both of the following are true:
- A domain controller guest is stored on SMB 3 storage
- No other domain controller is reachable by the Hyper-V host
This issue has a very simple solution: don’t put your domain controllers on SMB 3 storage.
Clusters and the “Chicken and Egg”
In 2008 R2 and prior, a cluster wouldn’t start at all if it couldn’t contact a domain controller. This is no longer true in 2012 R2. Even if the cluster service won’t start, both Hyper-V and VMMS.EXE will. With basic cluster troubleshooting techniques, you can bring a clustered virtual machine online without the cluster running. These are techniques that you should know anyway if you’re operating a Hyper-V cluster.
Hyper-V Hosting its Own Domain Controller
Simply put, there is no chicken and egg problem. This is the process:
Myth 4: Time Drift is Uncontrollable When Domain Controllers are Virtualized
Windows is not a real-time operating system, so time drift is inevitable. If a Hyper-V host’s CPUs are heavily burdened, time will drift more quickly. This is not an uncontrollable issue, though, unless the CPUs are really bogged down. If that’s the case, the problem isn’t that the domain controller is virtualized; the problem is that the host is overburdened. There are simple ways to deal with normal drift. Time drift is not a good argument against virtualizing domain controllers.
Virtualized Domain Controllers: Best Practices
With the myths out of the way, you’re clear to design your domain controller deployment. What follows are the some of the best practices around domain controllers with an emphasis on running them in a virtualized environment.
1: Always Start by Assessing Your Situation
Before you begin, determine what you want your final domain controller situation to look like, how small failures will be handled, and how you’ll recover from any catastrophic disasters. How many domain controllers do you need? Where should they be placed? Should domain controllers be highly available? What will the backup schedule look like? The answers to these questions will draw the most definitive picture of what your final deployment should look like. Do not start throwing up domain controllers with the notion that you’ll figure all of this out later, because later never comes until there is a failure to be addressed.
2: Determine How Many Domain Controllers are Necessary
This is the first question you should answer. If you’re a small shop with a single location, you can shortcut this question. A single domain controller can easily handle thousands of objects. Any extra DCs are for resiliency, not capacity. If you have multiple sites, try to place at least one domain controller in each site and make sure to configure those sites in Active Directory so that authentication and replication traffic is properly handled. If you need more in-depth help, refer to TechNet’s thorough article on the subject.
3. Determine Where to Place Domain Controllers
There are some very simple guidelines for domain controller placement.
- If possible, domain controllers should not reside on the same hardware. The primary purpose of multiple domain controllers is to provide 100% availability for domain services. If you have two domain controllers and they’re both on the same host, that’s not substantially better than just running with a single domain controller. Active Directory is resilient even without redundancy
- If you have multiple sites and they have any sort of network connection to each other, your preference should be to place one domain controller in each site. The need for domain controllers in any given remote site is tied to the number of users in that site and the quality of the intersite connection. If you have a lot of users, you want a domain controller. If the intersite link quality is poor, you want a domain controller. Make sure that the domain controller or some other server in the remote site is also running DNS and that all systems in that site refer to it.
4. Do Not Checkpoint Virtualized Domain Controllers
Prior to 2012, reverting a domain controller to a checkpoint (snapshots in those days) could cause irreversible damage to your domain. That’s because the other domain controllers thought that they’d already performed replication with that domain controller, but it would be oblivious to any changes that were discarded when the checkpoint was reverted. It would then resubmit object changes using Update Sequence Numbers that it had already used, causing inconsistencies in the directory. Sometimes these are detected, sometimes they’re not. The problem is fairly catastrophic and the fixes are painful, when they’re even possible. Microsoft has a TechNet article that explains this condition and can help you to find solutions if it happens to you.
In 2012, a new feature called VM Generation-ID was added. As long as the virtualized domain controller is running 2012 or later, USN rollbacks due to checkpoint reversion should be prevented. There are no guarantees. Your first, best choice is to never checkpoint a domain controller. A domain controller that runs no other services does not fit the envisioned use cases for checkpoints anyway, so you should be highly skeptical of any reasons that anyone submits to the contrary.
5. Disable Hyper-V Time Synchronization for Virtualized Domain Controllers
By default, Hyper-V is in charge of keeping the clocks updated in its guests. In days past, we recommended sharing responsibility for the clock in virtualized domain controllers. That’s because resuming a virtual machine from a saved state or reverting a checkpoint will ensure that its clock is incorrect and it might be too incorrect to allow for updates. With a time-sensitive application like domain services, that can be a very bad thing. Unfortunately, the shared responsibility setting is no longer possible; something has changed in the Hyper-V Time Synchronization Service that causes it to override any other source set for the Windows Time service. This means that any guest with the Hyper-V Time Synchronization Service enabled at all will always get its time from the host. Because domain controllers expect that they are at the top of the local time hierarchy, this could cause issues.
First, disable the synchronization service for virtual domain controllers:
Second, configure your domain and devices for authoritative time synchronization. This is a fairly lengthy procedure, but definitely worth it.
For further reading, we have a recent article on time and Hyper-V.
6. Do Not Place Domain Controllers in Saved State
Saving a domain controller is not as dangerous as checkpointing it, but it’s not a great thing, either. When a virtual machine resumes from a saved state or is reverted to a checkpoint, the only thing that is guaranteed to fix its clock is the Hyper-V Time Synchronization Service. But, as you know from above, you can’t enable that on virtualized domain controllers. If its clock skews too far, it might never fix itself automatically. That would be bad.
A worse situation is a long-saved domain controller. Consider the following scenario:
- A virtualized domain controller is placed into saved state.
- A user is terminated and his user account deleted.
- The tombstone lifetime for the user account object expires and the record of the account is deleted.
- The virtualized domain controller is resumed.
Of course, in order for this to work at all, there must be multiple domain controllers. You can’t delete objects if the only DC isn’t responding.
In the described scenario, that deleted account will be reanimated. This is because the object was active on that domain controller at the time that it was saved. When it resumed, it had an active record while no other domain controller in the environment is aware that it ever existed. It will resume its existence and be replicated as though nothing ever happened. If you haven’t got a password expiration policy or if its cycle is longer than your tombstone lifetime, this situation is a security risk.
If you have multiple domain controllers and you determine that one has been saved for a very long time, you can discard its saved state. Active Directory can handily deal with the data loss. There is some risk if the domain controller is your only global catalog server, but you’ll probably notice that something isn’t right if your only GCDC is offline long before your tombstone lifetime becomes an issue.
I tried to find a definitive answer on default tombstone lifetimes, but I could not find a straightforward answer that covers all versions. Defaults have been 60 days and 180 days, depending on the Windows Server version. However, knowing the default only goes so far; if a domain began its life in one version, that tombstone lifetime will persist through upgrades unless changed. The best thing to do is find out what yours is. This TechNet article explains how to determine the tombstone lifetime for a forest.
7. Set the Automatic Stop Action of Virtual Domain Controllers to Shut Down
By default, shutting down a post-2012 Hyper-V host will save all the guests. We want to avoid Saved State wherever possible for virtualized domain controllers.
8. Set the Automatic Start Action of Virtual Domain Controllers to Always Start without Delay
Active Directory Domain Services typically provides core functionality to most everything else in a Windows environment. Make sure that it’s available without manual intervention.
9. Do Not Make Domain Controller Virtual Machines Highly Available
Due to the vastly different natures of the technologies, Active Directory’s high availability features are dramatically superior to anything that Hyper-V and Failover Clustering can provide. There is generally no benefit to clustering the VM that contains a domain controller. I do make one small exception: consider clustering VMs that hold FSMO roles, but only if you have at least one non-HA domain controller and you have enough domain activity to justify it.
If you’ve got some notion that a single HA virtual machine saves on licensing as opposed to multiple non-HA VMs, I’m going to have to dispel it. A highly available virtual machine must have an available virtualization right on every host that it will ever run on:
10. Place Virtualized Domain Controllers on Local, Preferably Internal Storage
As mentioned above, the only true “chicken and egg” scenario is if a domain controller is placed on SMB 3 storage and there are no other domain controllers. That configuration must be avoided. Additionally, Microsoft does not support non-HA virtual machines running from Cluster Shared Volumes. It does function, but behavior can be strange. Because the fate of non-HA VMs is already inextricably linked to the fate of the host that they live on, the best thing to do is place them on internal storage. Use mirroring or some other technology to protect them against storage failures.
11. Do Not Perform Physical-to-Virtual Conversions on Domain Controllers
Active Directory has one of the smoothest migration paths that I have ever seen for any application. If you want to keep the name and IP address of your physical domain controller, then use a temporary domain controller to make the transition. If you search for “Active Directory Migration”, you’re going to get a lot of articles that talk about migrating objects from one domain to another with the Active Directory Migration Tool (ADMT). That process looks complicated, and it kind of is, but that’s not what I’m talking about.
The simple path:
- Build a new virtual machine.
- Install ADDS in the new virtual machine as a new domain controller in an existing domain. Ensure that it connects with your existing domain.
- If necessary, make the new DC a global catalog and/or transfer FSMO roles. Configure DNS, DHCP, and any other adjunct services performed by your DCs as necessary. If you have a source DHCP server on 2012 or later, you can use DHCP failover to quickly and easily replicate DHCP scopes and then break the relationship if necessary.
- Whenever you’re ready, uninstall ADDS and decommission the existing physical domain controller (as a Best Practice, you should follow step 7 below, but it’s less critical when the old DC name isn’t reused).
The complicated path, if you want to keep the name and IP of the existing DC:
- Build a new virtual machine.
- Install ADDS in the new virtual machine as a new domain controller in an existing domain. Ensure that it connects with your existing domain.
- Make the new DC a global catalog and transfer FSMO roles.
- Configure DNS, DHCP, and any other adjunct services performed by the original DC. If the original DC is running DHCP services and is Windows Server 2012 or later, you can use DHCP failover to quickly and easily replicate DHCP scopes and then break the relationship.
- Remove DNS, DHCP, etc. from the original domain controller.
- Remove ADDS from the original domain controller and decommission it.
- In Active Directory Sites and Services, Active Directory Users and Computers, and ADSIEdit, track down the remnants of the original domain controller and wipe them out. The most important place is ADSS.
- Follow steps 1-7 again, using a permanent domain controller that has the same name and IP as the source.
12. Always Join Your Hyper-V Host to Your Domain
People that don’t have domain controllers are free to leave their Hyper-V hosts in workgroup mode. I can see some theoretical reasons for not joining a hosts when all its guests are in a public-facing DMZ, although those reasons are usually flimsy. No one else has any excuse. I’m quite familiar with the counter-arguments, so I’ll just deal with them now:
- There is no chicken and egg problem as demonstrated above.
- If workgroup mode is more secure than your domain, then you need to fix your domain security.
- The headache of managing a non-domain-joined host qualifies as torture outlawed by the Geneva Convention.
A Hyper-V host is just another member server with a very long track record of stability. Stop treating it like it’s radioactive. I’ve hit the point where I feel that all of the myths around virtualized domain controllers that people use to justify workgroup-only hosts have been so thoroughly debunked by myself and others that responding to the same objections is no longer worth my time. If you haven’t got a new objection with concrete proof, don’t expect me to listen.
Frequently Asked Questions about Virtualized Domain Controllers
I’ve been writing about virtualized domain controllers for quite some time and have received and seen many questions on the subject. I’ve collated the most common questions into a FAQ format.
Q: Do I Need a Second Domain Controller or is One Enough?
A: Smaller Organizations Do Not Need a Second Domain Controller
The purists and the textbook admins always say that multiple domain controllers are a minimum requirement. Usually, the purists work on the provider side and have never had the pleasure of balancing a small business operating budget — other people’s money is a foreign, abstract concept. The textbook admins are so paralyzed by the fear of a 2% failure rate that the 98% success rate looks like a zero.
For most of my career, the vast majority of companies that I have worked for/with (we’re talking in the thousands here) used only a single domain controller. Everything was fine. There are actually some benefits to this:
- It saves on Windows licensing and hardware costs
- USN rollback is impossible; this is important in a small environment, as technical resources with the capacity to detect it and the expertise to correct it are in short supply
- If Bad Things should occur, restore of domain services is extremely simple in a single DC environment
In today’s world of ubiquitous virtualization, the single DC environment is quite low-risk. A small business could run Hyper-V Server on a host with two guests; one running domain services and the other running the company’s other services. This only requires a single Windows Server Standard Edition license and satisfies the best practice of separating domain services from other server applications. This could all then be backed up using any Hyper-V-aware solution (as a shameless plug, Altaro Backup Hyper-V would handle this scenario with its free edition). This is a solution that I wish I had access to many years ago, as it would have fundamentally changed the way I worked with many small business customers.
The single most important thing is backup. Backup provides data redundancy at a very low cost. Providing run-time redundancy more than doubles the cost. If that cost is too great and you understand the risks and you take the time to develop solid contingency strategies, then the single domain controller environment is just fine. I have seen it in common practice since 1998.
Q: If I have one Hyper-V host and it hosts the only domain controller, what happens if the domain controller doesn’t start?
A: Use local credentials.
Joining a domain does not affect the local credentials by default. Some domains rename the local administrator account. No domain should be disabling the local administrator, especially on servers. Doing so does not meaningfully improve security but exposes you to needless risks. You should already have a policy of maintaining local administrator credentials in a secure fashion. As long as you have those available, you can log on. As long as you have backup available, you can rebuild from scratch in the worst case scenario anyway.
Q: Isn’t it better to just make the Hyper-V host a domain controller?
A: Absolutely not. Never make a Hyper-V host into a domain controller.
I wrote an entire article explaining why. I have seen a number people going against this recommendation and I pity their users because of the problems they have needlessly introduced.
Q: What if I Can’t Trust All of My Hyper-V Admins Around Virtual Domain Controllers?
A: That is a Human Resources Problem.
Avoiding any technology because someone made a poor hiring decision results in an unmaintanable house of cards. Someone needs to have that uncomfortable, “Sorry but it’s not working out,” conversation so you can move on. I understand that there are politics involved that result in situations like this, but there is also a reason that good administrators tend to job hop a lot before they find their forever home (and some never do). In my experience, many of the untrustables are more of a training issue than anything else, so I always try education before ostracization or termination. Sometimes, “s/he goes or I go” is the best course of action.
Q: Windows Server Full GUI or Core for Domain Controllers?
There is no right answer.
Personally, I prefer Core. I set up domain controllers and then never log on to them directly again, so what do I care about a GUI? I have a workstation that I log in to with non-domain-administrator credentials. It has VirtualBox guests with various versions of Windows. Those versions of Windows have Remote Server Administration Tools installed. When I need to manage a server, I either do so via an administrative PowerShell session from my primary instance or I log in to the matching VirtualBox instance as a domain administrator and manage the server. The bonus for some of you is that when a questionable administrator connects to one of those virtual machines and sees that black box with the flashing cursor, they panic and go into a catatonic state that lasts at least a couple of hours. There is no better preventative against Click Next Admins than Server Core.
But, that’s me. I am one of a nearly infinite set of possibilities. If Core is just not where you are yet, that’s OK. Use the GUI. I have no patience for people that refuse to use the command-line/PowerShell or think that those tools are somehow lesser than the GUI, but I have absolutely no problem with those who are just on a different point in their career development. I can’t claim that I instantly abandoned GUI domain controllers when Core became available, so I can’t criticize anyone else. It’s true that the GUI uses a bit more resources and is a bit more susceptible to malware because of the larger surface area, but these are very manageable issues. I ran a pair of DCs, one GUI and one non-GUI, side-by-side for a while, and the memory difference was never more than about 100MB. They both had to reboot after each patch cycle. The GUI installation might have had 4 “needs restart” patches as opposed to the non-GUI DC’s 1, but a restart is a restart.
Q: Should I Abandon Physical Domain Controllers?
A: Not Necessarily.
If you have functional hardware and no need to free up physical space or transfer the license (or if it’s OEM and the license isn’t transferable), keep your existing physical domain controller. The additional resiliency is nice and a weaning period for the uncertain might be helpful.
But, going forward, I personally would not create any new physical domain controllers. I haven’t kept on top of scalability knowledge for domain controllers so I don’t know where they stand today, but in the past, a single domain controller could only reliably handle a certain amount of objects before adding hardware just didn’t help anymore. In those cases, adding domain controllers was the only solution. I spent some time researching for this article, and found that most of the official documentation that I used in those days has never been updated, even into the beginning of the 64-bit era. The best article that I found was this one, and I’m questioning the validity of more than a little of it. Today, I would be unlikely to build a domain controller with more than 4 cores and 8 GB of RAM… maybe up to 16 GB… and a few hundred gigabytes of disk space at most, and I would only ever expect it to handle about 5,000 users. With the pricing of modern server hardware, building a stand-alone unit of that size is nearly pointless because you can more than double those numbers for only a fraction of the base cost. Use a bigger Hyper-V host and virtualize domain services. If a domain controller starts acting sluggish and has hit those numbers, then you should scale out, not up.
Q: Should I Use Dynamic Memory for Domain Controllers?
ADDS is very respectful of memory, as server applications go. For small domains with only one or two domain controllers, find the size of your NTDS.DIT file and make that the Startup. You can make the Minimum a little bit smaller. For the maximum, don’t go more than 2 GB over the size of NTDS.DIT. You can watch Hyper-V Manager’s demand metric to see how it’s working, although Performance Monitor tracing is preferred.
Q: Should I Place the Active Directory Database on C: or Make a D: VHDX?
A: For Generation 1 VMs, create a D: drive on the virtual SCSI chain to hold the AD database. For Generation 2 VMs, it doesn’t matter.
I believe that all of the issues around caching for the virtual IDE drive have been resolved, but better safe than sorry.
Q: Should I Place the NTDS.DIT on a Pass-through Disk?
A: You Should Not Use Pass-through Disks
Stop using pass-through disks. If you don’t want to do that, just stop virtualizing altogether. You’re doing it wrong.
Q: Should I Use Virtual Domain Controllers with Fixed or Dynamically Expanding Virtual Hard Disks?
A: Domain Controllers are an Ideal Use for Dynamically Expanding Virtual Hard Disks
Unless you make your VHDX files ridiculously small, your virtualized domain controllers will never consume their allocated space. Domain controllers are relatively low on the IOPS consumption scale. Dynamically expanding is the way to go.
Q: Should I Use Hyper-V Replica with Domain Controllers?
A: Generally, No.
If your disaster recovery site has sufficient connectivity for Replica to function, then it has more than enough connectivity for Active Directory replication to function. As with clustered VMs, Replicated VMs require the same licensing as creating two distinct, active VMs (unless Software Assurance is in play). Active Directory replication is superior to anything that Hyper-V Replica can do for it. Also, if a change is successfully replicated to the master image of a domain controller but does not make it to the Replica, then a USN Rollback could occur if that Replica is brought online. However, Hyper-V Replica cycles much more frequently than inter-site Active Directory replication does. Despite that, I would always counsel to architect Active Directory replication and backup around any concerns and use Hyper-V Replica strictly for VMs whose contained applications do not have their own replication technology.
Have any questions or feedback?
Leave a comment below!