Hyper-V Key-Value Pair Data Exchange Part 3: Linux

Hyper-V Key-Value Pair Data Exchange Part 3: Linux

Some time ago, I discovered uses for Hyper-V Key-Value Pair Data Exchange services and began exploiting them on my Windows guests. Now that I’ve started building Linux guests, I need similar functionality. This article covers the differences in the Linux implementation and includes version 1.0 of a program that allows you to receive, send, and delete KVPs.

For a primer on Hyper-V KVP Exchange, start with this article: Hyper-V Key-Value Pair Data Exchange Part 1: Explanation.

The second part of that series presented PowerShell scripts for interacting with Hyper-V KVP Exchange from both the host and the guest sides. The guest script won’t be as useful in the context of Linux. Even if you install PowerShell on Linux, the script won’t work because it reads and writes registry keys. It might still spark some implementation ideas, I suppose.

What is Hyper-V Key-Value Pair Data Exchange?

To save you a few clicks and other reading, I’ll give a quick summary of Hyper-V KVP Exchange.

Virtual machines are intended to be “walled gardens”. The host and guest should have limited ability to interact with each other. That distance sometimes causes inconvenience, but the stronger the separation, the stronger the security. Hyper-V’s KVP Exchange provides one method for moving data across the wall without introducing a crippling security hazard. Either “side” (host or guest) can “send” a message at any time. The other side can receive it — or ignore it. Essentially, they pass notes by leaving them stuck in slots in the “wall” of the “walled garden”.

KVP stands for “key-value pair”. Each of these messages consists of one text key and one text value. The value can be completely empty.

How is Hyper-V KVP Exchange Different on Linux?

On Windows guests, a service runs (Hyper-V Data Exchange Service) that monitors the “wall”. When the host leaves a message, this service copies the information into the guest’s Windows registry. To send a message to the host, you (or an application) create or modify a KVP within a different key in the Windows registry. The service then places that “note” in the “wall” where the host can pick it up. More details can be found in the first article in this series.

Linux runs a daemon that is the analog to the Windows service. It has slightly different names on different platforms, but I’ve been able to locate it on all of my distributions with sudo service --status-all | grep kvp. It may not always be running; more on that in a bit.

Linux doesn’t have a native analog to the Windows registry. Instead, the daemon maintains a set of files. It receives inbound messages from the host and places them in particular files that you can read (or ignore). You can write to one of the files. The daemon will transfer those messages up to the host.

On Windows, I’m not entirely certain of any special limits on KVP sizes. A registry key can be 16,384 characters and there is no hard-coded limit on value size. I have not tested how KVP Exchange handles these extents on Windows. However, the Linux daemon has much tighter constraints. A key can be no longer than 512 bytes. A value can be no longer than 2,048 bytes.

The keys are case sensitive on the host and on Linux guests. So, key “LinuxKey” is not the same as key “linuxkey”. Windows guests just get confused by that, but Linux handles it easily.

How does Hyper-V KVP Exchange Function on Linux?

As with Windows guests, Data Exchange must be enabled on the virtual machine’s properties:

Hyper-V KVP Exchange on Linux

The daemon must also be installed and running within the guest. Currently-supported versions of the Linux kernel contain the Hyper-V KVP framework natively, so several distributions ship with it enabled. As mentioned in the previous section, the exact name of the daemon varies. You should be able to find it with: sudo service --status-all | grep kvp. If it’s not installed, check your distribution’s instruction page on TechNet.

All of the files that the daemon uses for Hyper-V KVP exchange can be found in the /var/lib/hyperv folder. They are hidden, but you can view them with ls‘s -a parameter:

Hyper-V KVP exchange

Anyone can read any of these files. Only the root account has write permissions, but that can be misleading. Writing to any of the files that are intended to carry data from the host to the guest has no real effect. The daemon is always monitoring them and only it can carry information from the host side.

What is the Purpose of Each Hyper-V KVP Exchange File?

Each of the files is used for a different purpose.

  • .kvp_pool_0: When an administrative user or an application in the host sends data to the guest, the daemon writes the message to this file. It is the equivalent of HKLMSOFTWAREMicrosoftVirtual MachineExternal on Windows guests. From the host side, the related commands are ModifyKvpItems, AddKvpItems, and RemoveKvpItems. The guest can read this file. Changing it has no useful effect.
  • .kvp_pool_1: The root account can write to this file from within the guest. It is the equivalent of HKLMSOFTWAREMicrosoftVirtual MachineGuest on Windows guests. The daemon will transfer messages up to the host. From the host side, its messages can be retrieved from the GuestExchangeItems field of the WMI object.
  • .kvp_pool_2: The daemon will automatically write information about the Linux guest into this file. However, you never see any of the information from the guest side. The host can retrieve it through the GuestIntrinsicExchangeItems field of the WMI object. It is the equivalent of the HKLMSOFTWAREMicrosoftVirtual MachineAuto key on Windows guests. You can’t do anything useful with the file on Linux.
  • .kvp_pool_3: The host will automatically send information about itself and the virtual machine through this file. You can read the contents of this file, but changing it does nothing useful. It is the equivalent of the HKLMSOFTWAREMicrosoftVirtual MachineGuestParameter key on Windows guests.
  • .kvp_pool_4: I have no idea what this file does or what it is for.

What is the Format of the Hyper-V KVP Exchange File on Linux?

Each file uses the same format.

One KVP entry is built like this:

  • 512 bytes for the key. The key is a sequence of non-null bytes, typically interpreted as char. According to the documentation, it will be processed as using UTF8 encoding. After the characters for the key, the remainder of the 512 bytes is padded with null characters.
  • 2,048 bytes for the value. As with the key, these are non-null bytes typically interpreted as char. After the characters for the value, the remainder of the 2,048 bytes is padded with null characters.

KVP entries are written end-to-end in the file with no spacing, headers, or footers.

For the most part, you’ll treat these as text strings, but that’s not strictly necessary. I’ve been on this rant before, but the difference between “text” data and “binary” data is 100% semantics, no matter how much code we write to enforce artificial distinctions. From now until the point when computers can process something other than low voltage/high voltage (0s and 1s), there will never be anything but binary data and binary files. On the Linux side, you have 512 bytes for the key and 2,048 bytes for the value. Do with them as you see fit. However, on the host side, you’ll still need to get through the WMI processing. I haven’t pushed that very far.

How Do I Use Hyper-V KVP Exchange for Linux?

This is the part where it gets fun. Microsoft only goes so far as to supply the daemon. If you want to push or pull data, that’s all up to you. Or third parties.

But really, all you need to do is read to and/or write from files. The trick is, you need to be able to do it using the binary format that I mentioned above. If you just use a tool that writes simple strings, it will improperly pad the fields, resulting in mangled transfers. So, you’ll need a bit of proficiency in whatever tool you use. The tool itself doesn’t matter, though. Perl, Python, bash scripts,… anything will do. Just remember these guidelines:

  • Writing to files _0, _2, _3, and _4 just wastes time. The host will never see it, it will break KVP clients, and the files’ contents will be reset when the daemon restarts.
  • You do not need special permission to read from any of the files.
  • _1 is the only file that it’s useful to write to. You can, of course, read from it.
    • Deleting the existing contents deletes those KVPs. You probably want to update existing or append new.
    • The host only receives the LAST time that a KVP is set. Meaning that if you write a KVP with key “NewKey” twice in the _1 file, the host will only receive the second one.
    • Delete a KVP by zeroing its fields.
  • If the byte lengths are not honored properly, you will damage that KVP and every KVP following.

Source Code for a Hyper-V KVP Exchange Utility on Linux

I’ve built a small utility that can be used to read, write, and delete Hyper-V KVPs on Linux. I wrote it in C++ so that it can be compiled into a single, neat executable.

Long-term, I will only be maintaining this project on my GitHub site. The listing on this article will be forever locked in a 1.0 state.

Compile Instructions

Each file is set so that they all live in the same directory. Use make to build the sources and sudo make install to put the executable into the /bin folder.

Install Instructions

Paste the contents of all of these files into accordingly-named files. File names are in the matching section header and in the code preview bar.

Transfer all of the files to your Linux system. It doesn’t really matter where. They just need to be in the same folder.

Run:

Usage Instructions

Get help with:

  • hvkvp –help
  • hvkvp read –help
  • hvkvp write –help
  • hvkvp delete –help

Each includes the related keys for that command and some examples.

Code Listing

The file list:

  • makefile
  • main.cpp
  • hvkvp.h
  • hvkvp.cpp
  • hvkvpfile.h
  • hvkvpfile.cpp
  • hvkvpreader.h
  • hvkvpreader.cpp
  • hvkvpremover.h
  • hvkvpremover.cpp
  • hvkvpwriter.h
  • hvkvpwriter.cpp

makefile

main.cpp

 

hvkvp.h

 

hvkvp.cpp

 

hvkvpfile.h

 

hvkvpfile.cpp

 

hvkvpreader.h

 

hvkvpreader.cpp

 

hvkvpremover.h

 

hvkvpremover.cpp

 

hvkvpwriter.h

 

hvkvpwriter.cpp

More in this series:

Part 1: Explanation

Part 2: Implementation

Hyper-V Key-Value Pair Data Exchange Part 1: Explanation

Hyper-V Key-Value Pair Data Exchange Part 1: Explanation

You’d be hard-pressed to find any feature of Hyper-V that’s been around as long as Data Exchange yet received so little attention. That’s not surprising, since it’s fairly difficult to use at all, much less effectively. My goal with this post is to introduce you to this feature along with a few methods that make using it easier. At the worst, you’ll get a decent understanding of what it does. If you’re lucky, you’ll come up with a use for it.

This blog is part of a series on Hyper-V Key-Value Pair (KVP) Data Exchange

Skip to part 2: Implementation

Skip to part 3: Linux

Comparing Hyper-V Data Exchange to PowerShell Direct

PowerShell Direct is a new feature that will debut with the 2016 version of Hyper-V. Sarah Cooley of Microsoft has written a couple of good reads on this feature so I won’t spend a lot of time on it. You can read her blog article and somewhat more formal documentation. The quick description is that PowerShell Direct allows you to operate PowerShell commands inside a guest by issuing them on the host. I believe that the existence of PowerShell Direct is a testament to how useful Hyper-V Data Exchange could be, if the bar to using it were lower. A few points of comparison:

  • Is PowerShell Direct stronger? No; quicker, easier. There is nothing that can be done with PowerShell Direct that cannot be done by leveraging Hyper-V Data Exchange. The amount of effort to produce the same results is likely to be quite a bit higher for Hyper-V Data Exchange. However, there are things you can do with Hyper-V Data Exchange that you can’t do with PowerShell Direct.
  • Flow direction. With Hyper-V Data Exchange, you can send from the guest to the host and vice versa. The push can be initiated from either side. However, it is a send-only operation. The only way you’ll know if your message was heard is if there is something on the other side that is listening and chooses to push a message in response. With PowerShell Direct, the flow direction is restricted to host-to-guest, so you can’t start the conversation from the guest’s side. However, the script that operates in the guest can return information to the host. In fact, PowerShell Direct can run interactively, so it’s a lot like PowerShell Remoting.
  • Non-Microsoft guest operating system support. Unless things have changed while I wasn’t paying attention, there aren’t any PowerShell hosts on non-Microsoft systems, and if there are, I have no idea if they’ll support PowerShell Direct. Hyper-V Data Exchange is part of the Linux Integration Components. I didn’t check every distribution, but you can go to Microsoft’s master TechNet page for Linux support on Hyper-V and, on the left, choose your distribution. Look for a table item named “Key-Value Pair” (KVP) to see if you have support. The bad news is that to exploit it on Linux, you have to do real programming. I’ll provide links later that have demo code listings.
  • Down-level Windows support. Hyper-V Data Exchange works on every host running a supported version of Hyper-V and any Windows guest that is running the Hyper-V Integration Services. PowerShell Direct only works when both the host and the guest are at the Windows 10/Server 2016 code base or later.

Security Implications of Hyper-V Data Exchange

The security risks presented by Hyper-V Data Exchange are extremely low for four reasons.

  1. Hyper-V Data Exchange is push-only. You cannot force the receiving side to do anything with the data that comes through.
  2. The entry and exit points for the pushed data are tightly controlled and not overridable.
  3. The data is transmitted in string format only (if you’re not familiar with computer data types, a string is treated as a series of unprocessable bits).
  4. To send data to the other side, you must have administrative rights on the source. To receive data incoming from the host on a guest, you must be at least an Authenticated User within the guest’s security context. To receive data incoming from a guest on the host, you must have administrative rights within the host’s security context.

So, while it wouldn’t be accurate to say that Hyper-V Data Exchange presents zero risk, its risk level should not keep anyone awake at night. The greatest threat would be if someone figures out a break-out attack to overcome point #2. With all of these points combined, in order for an attacker to be in a position to exploit Hyper-V Data Exchange, s/he is already in a position to carry out a different assault with considerably less effort than using KVP exchanges.

The security model for PowerShell Direct is different. When you instruct PowerShell on the host to run something on the guest, you must supply credentials that are in context of the guest with sufficient privileges to carry out your desired script. There are issues around storing these credentials securely, but they are not showstopping problems. The bigger issue, especially in comparison to Hyper-V Data Exchange, is that the operator of the host must have access to credentials that are valid on the guest. In hosted environments and in private clouds where Hyper-V administrators do not have privileges within guests and it would be unreasonable to ask clients to provide such credentials, this means that PowerShell Direct runs the risk of being dead on arrival. Hyper-V Data Exchange could still be made to work because there is no sharing of credentials.

What Can be Done with Hyper-V Data Exchange?

I certainly don’t have a comprehensive list of what this technology can be used for. I imagine that if I got ten or fifteen of you in a room with a whiteboard, we could probably fill it up in a half hour of brainstorming. Start with the functionality: you can use Hyper-V Data Exchange to send messages from a Hyper-V host to a guest and vice versa. Imagine the ways that could be useful.

Some ideas that I’ve had, including some that are in projects I’m involved:

  • Host-initiated patching of guests. This is especially useful for standalone Hyper-V hosts. The host signals to all of the guests that they should start a Windows Update cycle. When they report success, they are shut down. Then the host patches and restarts. When it comes back, all the guests are brought online. This means that you don’t need to do anything fancy with staggered schedules. You can have a single defined maintenance window instead.
  • Guest-initiated checkpoints. The loss of AzMan in Hyper-V Server 2012 R2 made some things harder for those shops that want to grant virtual machine control to their owners but don’t want the expense or hassle of SCVMM. Checkpointing would be one of the powers that you could resurrect.
  • State information sharing. As I’ll show you in a bit, the host and the guest have access to a fair bit of information about each other. You could easily extend this to include any status updates that you want. For instance, you could let the guest know that it’s got a checkpoint.
  • Run scripts. You can leverage Hyper-V Data Exchange to kick off scripts or executables. Of course, it’s not “direct” like PowerShell Direct, but you don’t have to share credentials, either. The guest can even initiate things in the host (I hope I don’t have to warn you how potentially dangerous that could be).

I should point out that in terms of raw functionality, PowerShell Remoting, especially with constrained endpoints, could be used to do all of these things as well, as long as the guest is running Windows. As with PowerShell Direct, PowerShell Remoting would be easier than KVP exchange in most cases. Hyper-V Data Exchange still has its merits. To point out a few: there is no concern about the double-hop issue, credential storage, firewall restrictions, VMs being disconnected from a network, or unauthorized message interception.

What is the Architecture of Hyper-V Data Exchange?

There are two major components of Hyper-V Data Exchange.

  • The Hyper-V Virtual Machine Management service (vmms.exe) operates everything on the host side. Its functionality is exposed through WMI.
  • The Hyper-V Data Exchange Service inside the guest is responsible for handling the data exchange. Its functionality is exposed through the Windows registry.

Hyper-V Data Exchange Host Operations

VMMS exposes three functions via WMI for the purpose of sending data to a guest.

  1. AddKvpItems. This function creates new key-value pairs for a guest. Once the pair is created, it exists until it is explicitly removed with RemoveKVPItems. From inside the guest, you can delete or change the registry key-value pair that corresponds to the created item, but it will be recreated automatically when the Hyper-V Data Exchange Service is restarted. Furthermore, checking for the existence of the KVP item on the host side will continue to show that it exists, because the client-side registry key is a view of the KVP data item, not the data item itself.
  2. ModifyKvpItems. This function changes the value(s) of the specified key-value pair(s). From inside the guest, you can change the value of the KVP in the representative registry key, but it will be restored to the last value set by the host when the Hyper-V Data Exchange Service is restarted. Furthermore, querying the KVP data item on the host will not return any value that was set inside the guest for the same reason explained in #1.
  3. RemoveKvpItems. This function removes KVP data exchange pairs from a guest. This is the only way to permanently remove the registry entries that appear inside the guest’s registry.

To determine what the extant KVP names and values for any given virtual machine are, VMMS exposes XML data structures that allow you query a guest for its KVP pairs. You can list both host-to-guest and guest-to-host pairs. These structures are read-only.

If the virtual machine is off, none of the KVPs are retrievable, even the host-to-guest KVPs that are not dependent upon the guest service. However, all of the host-to-guest KVP operations work while the guest is off. The KVPs are retrievable from the moment that the guest is started, even before the guest operating system is fully booted.

If the virtual machine is on and the Data Exchange service is disabled, all host-to-guest KVPs are retrievable and all of the host operations function. All of the guest-to-host values will be invisible.

The three host functions are case-sensitive, which I consider to be a fairly egregious design flaw because the registry is case-insensitive. As an example, you use AddKvpItems to add a KVP with a key named “SaltOrPepper” and a value of “Salt”. You then use ModifyKvpItem to change “saltorpepper” to “Pepper”. This command will fail because the KVP doesn’t exist. So, you use AddKvpItems to add a KVP with a key named “saltorpepper” and a value of “Pepper”. You will now have two KVPs, one with a key named “SaltOrPepper” and one with a key named “saltorpepper”, each with the respective values that you set. Querying the KVPs from the host side will show all entries. The guest will only show the oldest KVP with the original casing, but it will contain the value of the last case-insensitive add/modify. Likewise, RemoveKvpItems will only remove a KVP with a perfectly matched name.

Hyper-V Data Exchange Guest Operations

There are no “functions” inside the guest, per se. Everything is handled by writing to and reading from the Windows registry. When the host issues any of its three commands, the results are instantly visible in the related guest registry keys. When the guest writes to its guest-to-host keys, the host’s virtual machine WMI object immediately changes accordingly.

The root key for Hyper-V Data Exchange is HKEY_LOCAL_MACHINESOFTWAREMicrosoftVirtual Machine. If you access this key on any guest and check the permissions, you’ll see something like the following:

KVP Exchange Root Permissions

KVP Exchange Root Permissions

Authenticated Users has read-only permissions. All other entries in the list have Full Control. This screen shot was taken on a Windows Server 2012 R2 guest, which is why the sppsvc account appears. This is the account used by the Software Protection service; on Server 2012 R2 (and presumably later), the service uses Data Exchange to pass some information involving Automatic Virtual Machine Activation up to the host from the guest. You won’t see this service in the permissions list on Windows desktop or earlier Windows Server guests.

Notice that these permissions are not inherited from above this level. I don’t know what happens if you tinker with the permissions; it’s not something that I would do. All subkeys inherit the permissions that you see here.

There are four subkeys, each with designated functionality.

Auto

KVP Auto Key

KVP Auto Key

The Auto key contains data that is automatically created when the Hyper-V Data Exchange Service starts. The official documentation says that “This data is available to the host as intrinsic data.” There is a special section of the KVP data on the virtual machine WMI data for these entries. Some of these are retrievable via PowerShell, such as the IP addresses.

External

The External key is where KVP items controlled by the three host functions listed earlier will appear. When you use AddKvpItems, ModifyKvpItems, and RemoveKvpItems from the host, look for the results here. As a reminder: creating, modifying, and deleting these pairs from inside the guest is a largely fruitless endeavor. They are only views on the KVP items, not the KVP items themselves. They will be reverted to the host’s KVP item set when the Hyper-V Data Exchange Service starts. Initially, there are no KVP items in this registry key.

Guest

The Guest key is where data is sent from the guest to the host. For most systems prior to 2012 R2, this key will be empty by default. The following screenshot was taken on a 2012 R2 guest that is using Automatic Virtual Machine Activation (AVMA):

KVP with AVMA

KVP with AVMA

All entries that start with Security-SPP are automatically created by SPP and AVMA. These are guest-to-host informational entries only, so don’t think that tinkering with them will magically activate your guest if you aren’t running 2012 R2 Datacenter as your host. I’m not entirely certain what the SessionMonitor and Sessions entries represent.

Any entries that you create here are immediately visible on the host. If you want to send data to the host, simply create a KVP here. From the host side, they’ll show up instantly in the VM’s data structures. I’ll get to the “how” in a bit. The host cannot modify these values. You’ll notice that the SPP service has created a DWORD item; I did not test doing the same, but all items here end up in XML format so it’s still going to come out on the host side as string data.

GuestParameters

The GuestParameters subkey contains data that is automatically generated by the host (as opposed to data generated by the AddKvpItems and ModifyKvpItems functions). Why this is a subkey of Guest and not External is one of the great mysteries of the universe, probably explainable by “Oops!” Some of these KVPs might be valuable to you right away:

KVP Guest Parameters

KVP Guest Parameters

As you can see, the guest knows what operating system its host is running, what the host’s name is, what the virtual machine’s container name is, and its VMID.

Are There Immediately Practical Things Involving Hyper-V Data Exchange?

Failover Cluster Manager uses KVP exchange to populate many of the fields in the VM’s Summary pane:

KVP in Failover Cluster Manager

KVP in Failover Cluster Manager

If you stop or disable the Hyper-V Data Exchange Service, these fields will be blank.

Hyper-V Manager, Failover Cluster Manager, and PowerShell all retrieve the guest’s IP address(es) using KVP Data Exchange.

How to Put Hyper-V Data Exchange to Use

To push and pull data from the guest side, you can manipulate the registry with RegEdit.exe or PowerShell. So, to pull all that information about the host, you can use:

The Scripting Guy has a great blog series on using PowerShell with the registry. Check the Note section near the beginning of that post for links to the other articles.

This is the point where the post stops being as much fun as I’d like, because there’s no doubt that KVP Data Exchange can be tough to use. Some of the things that I’m doing with KVP Data Exchange might be a bit beyond a lot of administrators. I found PowerShell a bit too restrictive for what I wanted and instead cooked up some .Net solutions. For instance, for my synchronized host/guest patching, I have a .Net service that watches the registry for a KVP data item to change to a value that basically says “patch and shut down” and responds accordingly.

There is a decent, but not wonderful, article on TechNet that demonstrates the necessary functions. My issue with the PowerShell examples is that they’re not presented with any degree of reusability; you have to dig into them to figure out what to change to fit the KVP data items that you’ll be using. It’s not overly tough but it could have been presented better. If you’re using Linux guests, it has a C/C++ code sample for reading and manipulating the KVP IC. It’s not very well-documented and contains several unexplained magic numbers, but it should be simple enough to follow along (about as simple as useful C/C++ gets, anyway).

To end the post on a positive note, part 2 will be focused on the implementation of KVP Data Exchange, with almost a pure script/code post that contains wrappers that I’ve developed to make working with KVP Data Exchange much easier.

More in this series:

Part 2: Implementation

Part 3: Linux