High Network Latency on Virtual Windows Server 2012 and 2012 R2 when enduring heavy CPU load.
Today I encountered a situation where 2 new Windows Server 2012 R2 virtual domain controllers (running on VMWare) were unresponsive and experiencing continuis high CPU load. At first the VM’s where assigned 2 vCPU’s but when experiecing the same issues we ended up assigning them 4 vCPU’s. The performance seemed to be improved as the CPU load within the performance monitor did not exceed an avarage of 50% but still peeks of 100% were seen and in these peeks the network latency was really high (ping response 1000ms and above) while the average ping response was <1ms.
Because of this high latency their were authentication issues:
- ADFS authentication failed occasionally
- Normal Windows login failed occasionally
- VPN Authentication failed
- Wifi Authentication failed
While troubleshooting and searching for any related documents we found these 2 documents
KB Article VMWare:
How to configure Virtual-Receive-Side Scaling:
http://technet.microsoft.com/en-us/library/dn383582.aspx
By default RSS is disabled in Windows Servers 2012 and Windows Server 2012 R2 while it was enabled in 2008 R2 and lower.
What is Receive-Side Scaling:
Receive-side network processing in multi-core virtual machines is conventionally bottlenecked by the fact that a single virtual processor services all the interrupts from a virtual network adapter. Virtual Receive-side scaling solves this problem by enabling a network adapter to distribute its network processing load across multiple virtual processors in multi-core virtual machines.
Virtual Receive-side scaling requires Virtual Machine Queue (VMQ) support from the physical adapter. If VMQ is disabled, or is not supported then Virtual Receive-side scaling is disabled.
You can check RSS is enabled on your NIC with this Powershell command : Get-NetAdapterRSS
Name : Ethernet
InterfaceDescription : Microsoft Hyper-V Network Adapter
Enabled : False
To enable RSS use this Powershell command : Set-NetAdapterRSS -Name “AdapterName” -Enabled $True
With vRSS enabled your VM’s should not encounter any network latency and should perform without any problems again.