“Unable to connect to the MKS” Error – How to Fix it

When was the last time you tried to remote console to a VM only to be greeted with the annoying Unable to connect to the MKS or similar message? Some of the causes leading to this issue range from firewalls blocking the required ports, badly configured DNS, and ESXi services in need of a restart. Today’s post explores a number of ways you can use to remote console to a VMware virtual machine and what you can do when remote connections fail.

Speaking of firewalls, adding a persistent firewall rule to ESXi can be daunting. I discuss how to do this in How to create persistent firewall rules on ESXi.

 

What is MKS?


MKS stands for Mouse, Keyboard and Screen. In vSphere, you can remote console to VMs using the standard vSphere clients and products such as VMware Workstation.

For pre-6.5 vSphere releases, you can use the legacy vSphere client. As shown next, just highlight the VM you want to connect to and click on the Console tab. Alternatively, right-click on the VM and select Open Console. The latter method opens a stand-alone console window for you.

Remote console using the legacy client

Remote console using the legacy client

 

If you have vCenter installed, the vSphere Web Client provides you with two remote console options these being VMRC (the standalone VMware Remote Console) and the Web Console. Clicking on the cog wheel on the VM’s console screen under Summary, gives you both options along with the ability to set the default console and install VMRC if it is not already. VMRC is installed by following the link given in vSphere client or by downloading it from here.

NOTE: Users who have VMware Workstation installed will have noticed that selecting Launch Remote Console brings up Workstation’s VMRC instead of the standalone one. To override this behavior, uninstall the standalone VMRC and reinstall it back.

Back to vSphere Web Client, clicking on the VM’s console screen or selecting Open Console from the VM’s context menu will launch the default remote console.

Remote console using the vSphere Web client

Remote console using the vSphere Web client

 

The behavior is almost identical when using the new vSphere Client (HTML5) or ESXi’s host client (embedded).

Remote console using the vSphere client (HTML)

Remote console using the vSphere client (HTML)

 

Remote console using the host client (ESXi embedded)

Remote console using the host client (ESXi embedded)

 

Interestingly, you can configure a VM such that you connect to it using a remote client such as tightVNC. To do this, add the following lines to the VM’s configuration (Edit Settings -> VM Options tab -> Advanced Settings -> Edit Configuration). You must disable the ESXi firewall for this to work or add a rule to allow 5900 through, this being the default VNC port. If memory serves me right, this worked with ESXi 5.x versions but to be honest, I had little time to try it out on 6.x. With ESXi 6.5, I manage to establish a connection but it keeps asking for a password regardless of the one specified in the VM’s configuration.

Configuring a VM to accept remote connections from VNC

Configuring a VM to accept remote connections from VNC

 

What can go wrong and how to fix it


Some of the MKS errors you might see are as follows:

  • Error connecting: Host address lookup for server <SERVER> failed: The requested name is valid and was found in the database, but it does not have the correct associated data being resolved for.
  • Error connecting: cannot connect to host <host>: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
  • Error connecting: You need execute access in order to connect with the VMware console. Access denied for config file.
  • Unable to connect to MKS: failed to connect to server IP:903.

 

1. Firewalls

Depending on the version of ESXi / vCenter Server installed, TCP ports 902, 903, 443 and 9443 need to be reachable when using the web based and standalone remote consoles. With vSphere 6.5, I managed to replicate a common problem where a personal firewall, or one sitting between the client computer and ESXi networks, is blocking one or more ports especially 902. To do this, I enabled the Windows firewall and created an outbound rule blocking 902 as shown.

Managing Microsoft Windows firewall rules

Managing Microsoft Windows firewall rules

 

When the firewall rule is enabled, the following is what you get when trying to remote console to a VM. The error message might differ slightly depending at which point the connection is dropped.

Remote console connection errors due to a firewall blocking ports 902 and 9443

Remote console connection errors due to a firewall blocking ports 902 and 9443

 

Changing the firewall rule to Allow from Block fixes the issue. Often, the lost ability to remote console occurs right after someone installs or rolls out security software incorporating a personal firewall. From my experience, other primary suspects include updates to group policies governing the Windows firewall and unannounced changes to ACLs on switches / routers. If you’re not the one administering the latter, talk to your colleagues to try and narrow down the source of the problem.

You can use netstat to verify that connections on port 902 are being established when you remote console to a VM. If you don’t see any entries, the issue is definitely firewall / network related. Another trick you can use is to telnet <ESXi IP> 902 from a command prompt or terminal. If you get a reply, it means that ESXi is listening on the port and ready for connections.

Using netstat to inspect network connections

Using netstat to inspect network connections

 

If you’re not running a personal firewall, SSH to ESXi and temporarily disable the ESXi firewall by running esxcli network firewall set –enabled false. Verify that you can connect. If not, it’s a firewall sitting in between the management station and ESXi that’s dropping the connection.

If the problem persists after having ruled out firewalling as a problem source, move on to the next check.

 

2. DNS

Connection to remote console can also be impeded if ESXi and vCenter Server hostnames do not resolve correctly. To determine correct DNS resolution, use ping <ESXi hostname> or nslookup to verify that your ESXi and vCenter Server hostnames are resolving to the correct IP addresses and, likewise, to the correct hostnames. If the issues persists, try adding your ESXi hosts and IP addresses to \windows\system32\drivers\etc\hosts, run ipconfig /flushdns and retry.

192.168.10.10 ESX65-a.lab.local
192.168.10.11 ESX65-b.lab.local
192.168.10.12 ESX65-c.lab.local

 

3. Free Disk Space

If firewalling and DNS are not the culprit, verify that the partition where /var lives has enough free disk space. To do this, run df -h from an SSH session on the ESXi hosting the VM you’re trying to remote console to. Make sure that the partition where /var is located is not full.

Checking for free disk space on ESXi

Checking for free disk space on ESXi

 

Another command you could use is vdh -h.

Making sure /var has free disk space using the vdf -h command

Making sure /var has free disk space using the vdf -h command

 

4. Permissions

Verify that the permissions set on the VM’s VMX file are set correctly (755). To check the permissions, run the command:

ls -l *.vmx

-rwxr-xr-x 1 root root 3995 Aug 18 10:00 Windows 7.vmx

We can see that the permissions are correctly set for the VMX file of a VM called Window 7. 

  • rwx = 111 = 7
  • r-x = 101 = 5
  • r-x = 101 = 5

If the VMX file permissions are not set to 755, run the following command:

chmod 755 'Windows 7.vmx'

Note: On ESXi 6.5, setting the permissions to read only had no effect so the permissions issues might only apply to older versions of ESXi.

 

5. If nothing else works!

If none of the above worked, try doing the following:

  • Check that the time on ESXi is set correctly and in sync with the rest of the infrastructure.
  • vMotion the VM you’re unable to remote console to, to another ESXi host and retry.
  • Restart the management agents on ESXi as per the following KB.
  • Reboot the VM.
  • Reboot ESXi.

Note: Sometimes the fix is as trivial as pressing a key while the focus is given to the console window. This was common with older versions of the legacy client and vSphere releases where nothing is displayed in the VM’s console window. Hitting Enter or another key usually brings the console back to life. An alternative fix is to launch the console in its own window by right-clicking on it and selecting Open Console.

 

Conclusion


Being able to remote console to a VM is, of course, an integral part of any admin’s job. Seeing the dreaded “Unable to connect to the MKS” message and being able to console can be highly frustrating more so when you need to fix something fast. In most cases, the problem lies with a firewall blocking one or more required ports and missing or bad DNS entries. For the rest, it’s usually enough to vMotion a VM to another host or restart the management agent on ESXi.

If you enjoyed reading this post, why not have a look at The Complete List of VMware Articles.

[the_ad id=”4738″][the_ad id=”4796″]

Share this post

Not a DOJO Member yet?

Join thousands of other IT pros and receive a weekly roundup email with the latest content & updates!

65 thoughts on "“Unable to connect to the MKS” Error – How to Fix it"

  • Hi,

    I found this problem when tried to connect to those VMs hosted by one of our ESXi server. If I migrate those VMs to another ESXi servers then I can connect the console with no problem.

    Checked everything e.g. firewall, permission but still with no luck. It turns out the “Config.Defaults.security.host.ruissl” is NOT set.

    The following is how to set this up in vsphere client.

    1. start, log on and connect to the vsphere client
    2. select the ESXi server that has the problem
    3. under the Configuration pane, click the “Advanced Settings” under Software
    4. In the “Advanced Setttings” pop-up, expand “Config->Defaults->security->host”
    5. Make sure the checkbox for “Config.Defaults.security.host.ruissl” checkbox is ticked

    Once I ticked this checkbox on my ESXi server, I can connect to the remote console of the VMs.

    Good luck.

  • Jeremy Martin says:

    It was my system time – thanks!

Leave a comment

Your email address will not be published. Required fields are marked *