High Network Latency on Virtual Windows Server 2012 and 2012 R2 when enduring heavy CPU load.

Today I encountered a situation where 2 new Windows Server 2012 R2 virtual domain controllers (running on VMWare) were unresponsive and experiencing continuis high CPU load. At first the VM’s where assigned 2 vCPU’s but when experiecing the same issues we ended up assigning them 4 vCPU’s. The performance seemed to be improved as the CPU load within the performance monitor did not exceed an avarage of 50% but still peeks of 100% were seen and in these peeks the network latency was really high (ping response 1000ms and above) while the average ping response was <1ms.

Because of this high latency their were authentication issues:

  • ADFS authentication failed occasionally
  • Normal Windows login failed occasionally
  • VPN Authentication failed
  • Wifi Authentication failed

While troubleshooting and searching for any related documents we found these 2 documents

KB Article VMWare:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2008925

How to configure Virtual-Receive-Side Scaling:
http://technet.microsoft.com/en-us/library/dn383582.aspx

By default RSS is disabled in Windows Servers 2012 and Windows Server 2012 R2 while it was enabled in 2008 R2 and lower.
What is Receive-Side Scaling:

Receive-side network processing in multi-core virtual machines is conventionally bottlenecked by the fact that a single virtual processor services all the interrupts from a virtual network adapter. Virtual Receive-side scaling solves this problem by enabling a network adapter to distribute its network processing load across multiple virtual processors in multi-core virtual machines.

Virtual Receive-side scaling requires Virtual Machine Queue (VMQ) support from the physical adapter. If VMQ is disabled, or is not supported then Virtual Receive-side scaling is disabled.

You can check RSS is enabled on your NIC with this Powershell command :  Get-NetAdapterRSS

Name : Ethernet
InterfaceDescription : Microsoft Hyper-V Network Adapter
Enabled : False

To enable RSS use this Powershell command : Set-NetAdapterRSS -Name “AdapterName” -Enabled $True

With vRSS enabled your VM’s should not encounter any network latency and should perform without any problems again.

Leave a Reply

Your email address will not be published. Required fields are marked *

captcha

Please enter the CAPTCHA text

This site uses Akismet to reduce spam. Learn how your comment data is processed.