Hyper-V Quick Tip: How to Choose a Live Migration Performance Solution

Q: Which Hyper-V Live Migration Performance Option should I choose?

A: If you have RDMA-capable physical adapters, choose SMB. Otherwise, choose Compression.

You’ve probably seen this page of your Hyper-V hosts’ Settings dialog box at some point:

The field descriptions go into some detail, but they don’t tell the entire story.

Live Migration Transport Option: TCP/IP

In this case, “TCP/IP” basically means: “don’t use compression or SMB”. Prior to 2012, this was the only mode available. A host opens up a channel to the target system on TCP port 6600 and shoots the data over as quickly as possible.

Live Migration Transport Option: Compression

Introduced in 2012, the Compression method mostly explains itself. The hosts still use port 6600, but the sender compresses the data prior to transmission. This technique has three things going for it:

• The vast bulk of a Live Migration involves moving the virtual machine’s memory contents. Memory contents tend to compress quite readily, resulting in a substantially reduced payload size over the TCP/IP method
• For most computing systems, the CPU cycles involved in compression are faster and cheaper than the computations needed to break down, transmit, and re-assemble multi-channel TCP/IP traffic
• Works for any environment
• Each virtual machine that you move simultaneously (limited by host settings) will get its own unique TCP channel. That gives the Dynamic and Hash load balancing algorithms an opportunity to use different physical pathways for simultaneous migrations

Live Migration Transport Option: SMB

Also new with 2012, the SMB transport method leverages the new capabilities of version 3 (and higher) of the SMB protocol. Two things matter for Live Migration:

• SMB Direct: Leveraging RDMA-capable hardware, packets transmitted by SMB Direct move so quickly you’d almost think they arrived before they left. If you haven’t had a chance to see RDMA in action, you’re missing out. Unfortunately, you can’t get RDMA on the cheap.
• SMB Multichannel: When multiple logical paths are available (as in, a single host with different IP addresses, preferably on different networks), SMB can break up traffic into multiple streams and utilize all available routes

Why Should I Favor Compression over SMB?

The SMB method sounds really good, right? Even if you can’t use SMB Direct, you get something from SMB multichannel, right? Well… no… not much. Processing of TCP/IP packets and Ethernet frames has always been intensive at scale. Ordinarily, our server computers don’t move much data so we don’t see it. However, a Live Migration pushes lots of data. Even keeping a single Ethernet stream intact and in order can cause a burden on your networking hardware. Breaking it up into multiple pieces and re-assembling everything in the correct order across multiple channels can pose a nightmare scenario. However, SMB Direct can offload enough of the basic network processing to nearly trivialize the effort. Without that aid, Compression will be faster for most people.

Should I Ever Prefer the Plain TCP/IP Method?

I have not personally encountered a scenario in which I would prefer TCP/IP over the other choices. However, it does cause the least amount of host load. If your hosts have very high normal CPU usage and you want Live Migrations to occur as discreetly as possible, choose TCP/IP. You may add in a QoS layer to tone it down further.

Your Comments

Don’t agree with my assessment or encountered situations which don’t line up with my advice? I’m happy to hear your thoughts on Live Migrations Performance Options and which are the correct solutions to choose in various circumstances. Write to me in the comments below.

If you like this Hyper-V Quick tip, check out another in the Hyper-V Quick Tip Series.