Solved Windows not utilizing over 30% of LAN bandwidth

[Lurickin]

I have my Windows box running with a Solarflare NIC (Model: SFN5162F) which has two 10Gb ports on it. Those links are connected to two of the 10Gb ports on the switch. I also have a Linux server with the same NIC and its two 10Gb ports connecting to the switch. I know this forum is for Windows support so I want to make sure I check and optimize everything I can on the Windows side before moving on to the Linux box. (See update below)

I currently have Jumbo Frames enabled and set to 9000 on both machines and the switch. Every LAN speedtest I have run with iperf and other tools I have never seen the speeds go higher than 3.5Gbps and mostly seem to average around 3Gbps or slightly less. The switch is fully capable of supporting all the bandwidth and the cards are as well but with so many options out there and what you should and could tweak or change or do on the Windows side I'm left scratching my head as to what will actually work. It gets annoying trying 20 different things and rebooting the machine after each one just to make sure that the settings took and everything starts up fresh.

The drivers are up to date with the latest ones available from the manufacturer. Disabled anti-virus and anything else that could even possibly cause an issue during testing and saw no improvement. The NIC supports bidirectional communication of 10Gbps each way and while I know iperf will only really test one of the adapters, I should still expect to see test speeds around at least 9 to 9.5Gbps. I've verified the settings and full duplex is being used.



Update: I opened up two windows and ran iperf as a server with one and a client on the other. I did this on both the Linux box and Windows box. The results from the Linux box are below:

:~$ iperf -c 192.168.1.1
------------------------------------------------------------
Client connecting to 192.168.1.1, TCP port 5001
TCP window size: 2.50 MByte (default)
------------------------------------------------------------
[  3] local 192.168.1.1 port 48559 connected with 192.168.1.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  53.9 GBytes  46.3 Gbits/sec

The Windows box:

>iperf -c 192.168.1.2
------------------------------------------------------------
Client connecting to 192.168.1.2, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  3] local 127.0.0.1 port 53517 connected with 192.168.1.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  2.54 GBytes  2.18 Gbits/sec

I know that these results are produced, at least on the linux side, with the traffic not even traversing the physical interfaces but the differences seem too great between Linux and Windows to be a fluke with iperf itself and still leads me to believe something is going wonky in Windows.

 

[theveterans]

I don't get how Linux got way above 10 Gbps if the link rate maxes out at 10 Gbps. I wouldn't trust the results from Linux if it's showing 46 Gbps. Anyways, maybe network overhead is far more pronounced at this bandwidth level.

Unfortunately, 10 Gbps networking is still limited to enterprise so it'll be very rare for to get another user to post here that he/she is getting throughput of 80 - 97% of the 10 Gbps link rate to show that it isn't limited by OS but rather drivers, networking equipment, SSD drives and/or network overhead.

 

[Lurickin]

I know this is over a month old but I just wanted to post an update. I finally figured out why there was such a limitation on the bandwidth. My SSD has a max read/write speed of around 500MBps and assuming that it gets anywhere near that in real world situations the max throughput over LAN would be around 3,500Mbps. So the limitation appears to be the hard drive and a little bit of Windows overhead as well. To fully utilize all the bandwidth available I'll have to go with a big RAID 10 of SSDs or a RAID 0 and keep lots of backups.

 

[countzero]

or a decent lsi or areca controller and >5 enterprise hdds in raid 5.

the only reason to run 10Gbe anyway.