One-way slowness with Intel X540-TI

[alraz]

Hi

I have NAS4Free system with an Intel X540-TI 10Gbps card connected directly to a Windows Machine which has another X540-TI.
It ised to give me no more than about 1 to 1.6 Gbps when saving and reading to a zfs volume via SMB3.

After much trial and error, found a configuration on the Windows machine called "Interrupt Moderation Rate", which, when disabled, gives me about 400 MB/s of writes to the ZFS volume; however, reading from that same volume gives me only the same 1.3 Gbps:


I suppose I have to tweak the same "Interrupt Moderation Rate" configuration on the NAS4Free side, but I have no clue where or how to do this.

Also, I guess it would be important to note that I configured jumbo frames to 9014 bytes, which gives a little performance boost, but far from the 10 Gpbs these cards are supposed to be able to handle.

Any help will be greatly appreciated.

NAS4Free system specs:
Hostname: nas4free.local
Version: 9.3.0.2 - Nayla (revision 1310)
Build date: Sun Jan 18 01:13:26 CET 2015
Platform OS: FreeBSD Revision: 199506
FreeBSD 9.3-RELEASE-p8 #0 r277306M: Sat Jan 17 21:53:14 CET 2015
Platform: x64-embedded on AMD A10-6800K APU with Radeon(tm) HD Graphics
System: ASUSTeK COMPUTER INC. F2A85-M PRO
System bios: American Megatrends Inc. version: 6601 11/25/2014
CPU frequency: 4100MHz

[b0ssman]

have you done an iperf test? to see what the actual network performance is?

http://n4f.siftusystems.com/index.php/2 ... /comments/

[alraz]

Sorry, I forgot to mention that. In fact, I've tried with iperf too and the result is about the same

With one connection, from windows to Nas4Free, I get about 6.8 Gbps:

C:\Users\alraz\Desktop\1>iperf3.exe -c 192.168.0.253
Connecting to host 192.168.0.253, port 5201
[ 4] local 192.168.0.250 port 56106 connected to 192.168.0.253 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 747 MBytes 6.26 Gbits/sec
[ 4] 1.00-2.00 sec 812 MBytes 6.81 Gbits/sec
[ 4] 2.00-3.00 sec 759 MBytes 6.37 Gbits/sec
[ 4] 3.00-4.00 sec 757 MBytes 6.35 Gbits/sec
[ 4] 4.00-5.00 sec 759 MBytes 6.37 Gbits/sec
[ 4] 5.00-6.00 sec 751 MBytes 6.30 Gbits/sec
[ 4] 6.00-7.00 sec 762 MBytes 6.39 Gbits/sec
[ 4] 7.00-8.00 sec 746 MBytes 6.26 Gbits/sec
[ 4] 8.00-9.00 sec 728 MBytes 6.11 Gbits/sec
[ 4] 9.00-10.00 sec 813 MBytes 6.82 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 7.45 GBytes 6.40 Gbits/sec sender
[ 4] 0.00-10.00 sec 7.45 GBytes 6.40 Gbits/sec receiver



From BSD to Windows, I get about 1.3 Gbps:

C:\Users\alraz\Desktop\1>iperf3.exe -R -c 192.168.0.253
Connecting to host 192.168.0.253, port 5201
Reverse mode, remote host 192.168.0.253 is sending
[ 4] local 192.168.0.250 port 56142 connected to 192.168.0.253 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 153 MBytes 1.29 Gbits/sec
[ 4] 1.00-2.00 sec 153 MBytes 1.28 Gbits/sec
[ 4] 2.00-3.00 sec 155 MBytes 1.30 Gbits/sec
[ 4] 3.00-4.00 sec 155 MBytes 1.30 Gbits/sec
[ 4] 4.00-5.00 sec 155 MBytes 1.30 Gbits/sec
[ 4] 5.00-6.00 sec 155 MBytes 1.30 Gbits/sec
[ 4] 6.00-7.00 sec 155 MBytes 1.30 Gbits/sec
[ 4] 7.00-8.00 sec 155 MBytes 1.30 Gbits/sec
[ 4] 8.00-9.00 sec 155 MBytes 1.30 Gbits/sec
[ 4] 9.00-10.00 sec 155 MBytes 1.30 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 1.51 GBytes 1.30 Gbits/sec sender
[ 4] 0.00-10.00 sec 1.51 GBytes 1.30 Gbits/sec receiver



However, if I increase the number of parallel connections, I can pretty much get the 10 Gbps that the cards are expected to deliver:

C:\Users\alraz\Desktop\1>iperf3.exe -P 10 -R -c 192.168.0.253
Connecting to host 192.168.0.253, port 5201
Reverse mode, remote host 192.168.0.253 is sending
[ 4] local 192.168.0.250 port 56343 connected to 192.168.0.253 port 5201
[ 6] local 192.168.0.250 port 56344 connected to 192.168.0.253 port 5201
[ 8] local 192.168.0.250 port 56345 connected to 192.168.0.253 port 5201
[ 10] local 192.168.0.250 port 56346 connected to 192.168.0.253 port 5201
[ 12] local 192.168.0.250 port 56347 connected to 192.168.0.253 port 5201
[ 14] local 192.168.0.250 port 56348 connected to 192.168.0.253 port 5201
[ 16] local 192.168.0.250 port 56349 connected to 192.168.0.253 port 5201
[ 18] local 192.168.0.250 port 56350 connected to 192.168.0.253 port 5201
[ 20] local 192.168.0.250 port 56351 connected to 192.168.0.253 port 5201
[ 22] local 192.168.0.250 port 56352 connected to 192.168.0.253 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 102 MBytes 858 Mbits/sec
[ 6] 0.00-1.00 sec 127 MBytes 1.07 Gbits/sec
[ 8] 0.00-1.00 sec 124 MBytes 1.04 Gbits/sec
[ 10] 0.00-1.00 sec 102 MBytes 859 Mbits/sec
[ 12] 0.00-1.00 sec 133 MBytes 1.12 Gbits/sec
[ 14] 0.00-1.00 sec 121 MBytes 1.01 Gbits/sec
[ 16] 0.00-1.00 sec 127 MBytes 1.06 Gbits/sec
[ 18] 0.00-1.00 sec 135 MBytes 1.13 Gbits/sec
[ 20] 0.00-1.00 sec 102 MBytes 853 Mbits/sec
[ 22] 0.00-1.00 sec 102 MBytes 857 Mbits/sec
[SUM] 0.00-1.00 sec 1.15 GBytes 9.86 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 1.00-2.00 sec 100 MBytes 839 Mbits/sec
[ 6] 1.00-2.00 sec 120 MBytes 1.01 Gbits/sec
[ 8] 1.00-2.00 sec 126 MBytes 1.06 Gbits/sec
[ 10] 1.00-2.00 sec 97.2 MBytes 816 Mbits/sec
[ 12] 1.00-2.00 sec 139 MBytes 1.16 Gbits/sec
[ 14] 1.00-2.00 sec 127 MBytes 1.07 Gbits/sec
[ 16] 1.00-2.00 sec 120 MBytes 1.01 Gbits/sec
[ 18] 1.00-2.00 sec 139 MBytes 1.16 Gbits/sec
[ 20] 1.00-2.00 sec 97.7 MBytes 820 Mbits/sec
[ 22] 1.00-2.00 sec 98.1 MBytes 823 Mbits/sec
[SUM] 1.00-2.00 sec 1.14 GBytes 9.76 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 2.00-3.00 sec 105 MBytes 879 Mbits/sec
[ 6] 2.00-3.00 sec 125 MBytes 1.05 Gbits/sec
[ 8] 2.00-3.00 sec 122 MBytes 1.02 Gbits/sec
[ 10] 2.00-3.00 sec 107 MBytes 897 Mbits/sec
[ 12] 2.00-3.00 sec 122 MBytes 1.03 Gbits/sec
[ 14] 2.00-3.00 sec 121 MBytes 1.02 Gbits/sec
[ 16] 2.00-3.00 sec 124 MBytes 1.04 Gbits/sec
[ 18] 2.00-3.00 sec 123 MBytes 1.03 Gbits/sec
[ 20] 2.00-3.00 sec 106 MBytes 887 Mbits/sec
[ 22] 2.00-3.00 sec 104 MBytes 875 Mbits/sec
[SUM] 2.00-3.00 sec 1.13 GBytes 9.72 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 3.00-4.00 sec 104 MBytes 873 Mbits/sec
[ 6] 3.00-4.00 sec 124 MBytes 1.04 Gbits/sec
[ 8] 3.00-4.00 sec 119 MBytes 1000 Mbits/sec
[ 10] 3.00-4.00 sec 106 MBytes 886 Mbits/sec
[ 12] 3.00-4.00 sec 127 MBytes 1.06 Gbits/sec
[ 14] 3.00-4.00 sec 118 MBytes 988 Mbits/sec
[ 16] 3.00-4.00 sec 121 MBytes 1.02 Gbits/sec
[ 18] 3.00-4.00 sec 126 MBytes 1.05 Gbits/sec
[ 20] 3.00-4.00 sec 105 MBytes 878 Mbits/sec
[ 22] 3.00-4.00 sec 106 MBytes 891 Mbits/sec
[SUM] 3.00-4.00 sec 1.13 GBytes 9.69 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 4.00-5.00 sec 104 MBytes 873 Mbits/sec
[ 6] 4.00-5.00 sec 120 MBytes 1.01 Gbits/sec
[ 8] 4.00-5.00 sec 126 MBytes 1.06 Gbits/sec
[ 10] 4.00-5.00 sec 102 MBytes 854 Mbits/sec
[ 12] 4.00-5.00 sec 127 MBytes 1.07 Gbits/sec
[ 14] 4.00-5.00 sec 128 MBytes 1.07 Gbits/sec
[ 16] 4.00-5.00 sec 123 MBytes 1.03 Gbits/sec
[ 18] 4.00-5.00 sec 127 MBytes 1.06 Gbits/sec
[ 20] 4.00-5.00 sec 101 MBytes 847 Mbits/sec
[ 22] 4.00-5.00 sec 102 MBytes 852 Mbits/sec
[SUM] 4.00-5.00 sec 1.13 GBytes 9.72 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 5.00-6.00 sec 103 MBytes 866 Mbits/sec
[ 6] 5.00-6.00 sec 122 MBytes 1.03 Gbits/sec
[ 8] 5.00-6.00 sec 123 MBytes 1.03 Gbits/sec
[ 10] 5.00-6.00 sec 101 MBytes 846 Mbits/sec
[ 12] 5.00-6.00 sec 128 MBytes 1.07 Gbits/sec
[ 14] 5.00-6.00 sec 124 MBytes 1.04 Gbits/sec
[ 16] 5.00-6.00 sec 123 MBytes 1.03 Gbits/sec
[ 18] 5.00-6.00 sec 129 MBytes 1.08 Gbits/sec
[ 20] 5.00-6.00 sec 103 MBytes 866 Mbits/sec
[ 22] 5.00-6.00 sec 102 MBytes 858 Mbits/sec
[SUM] 5.00-6.00 sec 1.13 GBytes 9.72 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 6.00-7.00 sec 103 MBytes 862 Mbits/sec
[ 6] 6.00-7.00 sec 125 MBytes 1.05 Gbits/sec
[ 8] 6.00-7.00 sec 125 MBytes 1.05 Gbits/sec
[ 10] 6.00-7.00 sec 103 MBytes 868 Mbits/sec
[ 12] 6.00-7.00 sec 133 MBytes 1.12 Gbits/sec
[ 14] 6.00-7.00 sec 125 MBytes 1.05 Gbits/sec
[ 16] 6.00-7.00 sec 125 MBytes 1.05 Gbits/sec
[ 18] 6.00-7.00 sec 133 MBytes 1.12 Gbits/sec
[ 20] 6.00-7.00 sec 102 MBytes 860 Mbits/sec
[ 22] 6.00-7.00 sec 103 MBytes 861 Mbits/sec
[SUM] 6.00-7.00 sec 1.15 GBytes 9.89 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 7.00-8.00 sec 101 MBytes 851 Mbits/sec
[ 6] 7.00-8.00 sec 129 MBytes 1.08 Gbits/sec
[ 8] 7.00-8.00 sec 124 MBytes 1.04 Gbits/sec
[ 10] 7.00-8.00 sec 100 MBytes 840 Mbits/sec
[ 12] 7.00-8.00 sec 138 MBytes 1.16 Gbits/sec
[ 14] 7.00-8.00 sec 119 MBytes 995 Mbits/sec
[ 16] 7.00-8.00 sec 128 MBytes 1.08 Gbits/sec
[ 18] 7.00-8.00 sec 138 MBytes 1.16 Gbits/sec
[ 20] 7.00-8.00 sec 99.7 MBytes 836 Mbits/sec
[ 22] 7.00-8.00 sec 102 MBytes 855 Mbits/sec
[SUM] 7.00-8.00 sec 1.15 GBytes 9.89 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 8.00-9.00 sec 106 MBytes 889 Mbits/sec
[ 6] 8.00-9.00 sec 130 MBytes 1.09 Gbits/sec
[ 8] 8.00-9.00 sec 125 MBytes 1.05 Gbits/sec
[ 10] 8.00-9.00 sec 106 MBytes 892 Mbits/sec
[ 12] 8.00-9.00 sec 124 MBytes 1.04 Gbits/sec
[ 14] 8.00-9.00 sec 122 MBytes 1.03 Gbits/sec
[ 16] 8.00-9.00 sec 130 MBytes 1.09 Gbits/sec
[ 18] 8.00-9.00 sec 124 MBytes 1.04 Gbits/sec
[ 20] 8.00-9.00 sec 105 MBytes 884 Mbits/sec
[ 22] 8.00-9.00 sec 107 MBytes 898 Mbits/sec
[SUM] 8.00-9.00 sec 1.15 GBytes 9.89 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 9.00-10.00 sec 105 MBytes 877 Mbits/sec
[ 6] 9.00-10.00 sec 126 MBytes 1.06 Gbits/sec
[ 8] 9.00-10.00 sec 127 MBytes 1.07 Gbits/sec
[ 10] 9.00-10.00 sec 105 MBytes 883 Mbits/sec
[ 12] 9.00-10.00 sec 127 MBytes 1.06 Gbits/sec
[ 14] 9.00-10.00 sec 128 MBytes 1.07 Gbits/sec
[ 16] 9.00-10.00 sec 126 MBytes 1.06 Gbits/sec
[ 18] 9.00-10.00 sec 128 MBytes 1.08 Gbits/sec
[ 20] 9.00-10.00 sec 104 MBytes 873 Mbits/sec
[ 22] 9.00-10.00 sec 103 MBytes 865 Mbits/sec
[SUM] 9.00-10.00 sec 1.15 GBytes 9.89 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 1.01 GBytes 867 Mbits/sec sender
[ 4] 0.00-10.00 sec 1.01 GBytes 867 Mbits/sec receiver
[ 6] 0.00-10.00 sec 1.22 GBytes 1.05 Gbits/sec sender
[ 6] 0.00-10.00 sec 1.22 GBytes 1.05 Gbits/sec receiver
[ 8] 0.00-10.00 sec 1.21 GBytes 1.04 Gbits/sec sender
[ 8] 0.00-10.00 sec 1.21 GBytes 1.04 Gbits/sec receiver
[ 10] 0.00-10.00 sec 1.01 GBytes 864 Mbits/sec sender
[ 10] 0.00-10.00 sec 1.01 GBytes 864 Mbits/sec receiver
[ 12] 0.00-10.00 sec 1.27 GBytes 1.09 Gbits/sec sender
[ 12] 0.00-10.00 sec 1.27 GBytes 1.09 Gbits/sec receiver
[ 14] 0.00-10.00 sec 1.20 GBytes 1.03 Gbits/sec sender
[ 14] 0.00-10.00 sec 1.20 GBytes 1.03 Gbits/sec receiver
[ 16] 0.00-10.00 sec 1.22 GBytes 1.05 Gbits/sec sender
[ 16] 0.00-10.00 sec 1.22 GBytes 1.05 Gbits/sec receiver
[ 18] 0.00-10.00 sec 1.27 GBytes 1.09 Gbits/sec sender
[ 18] 0.00-10.00 sec 1.27 GBytes 1.09 Gbits/sec receiver
[ 20] 0.00-10.00 sec 1.00 GBytes 860 Mbits/sec sender
[ 20] 0.00-10.00 sec 1.00 GBytes 860 Mbits/sec receiver
[ 22] 0.00-10.00 sec 1.01 GBytes 864 Mbits/sec sender
[ 22] 0.00-10.00 sec 1.01 GBytes 864 Mbits/sec receiver
[SUM] 0.00-10.00 sec 11.4 GBytes 9.81 Gbits/sec sender
[SUM] 0.00-10.00 sec 11.4 GBytes 9.80 Gbits/sec receiver

[jasch]

I had the exact Problem.
To disable interrupt Moderation.
From shell.

Code: Select all

sysctl -w hw.ix.enable_aim=0

to make permanent add it to System|Advanced|sysctl.conf.
Happy fullspeed.

MFG Alex

[alraz]

I had the exact Problem.
To disable interrupt Moderation.
From shell.

Code: Select all

sysctl -w hw.ix.enable_aim=0

to make permanent add it to System|Advanced|sysctl.conf.
Happy fullspeed.

MFG Alex

Holy hot damn! Thank you so much!



Still need to do some tweaking (and maybe get better CPU because I see it capping), but man that's a lot of difference!
Thanks a lot.

[jasch]

No Problem, i was sitting down yesterday Night with this Problem.
Now iperf gets me arround 1100MB/s in both directions with 1 connection.
U can try some other tweaks too.

System|Advanced|loader.conf

cc_htcp_load=YES
hw.ixgbe.rxd=4096(u can try even bigger, in windows rx is 16MB max)
hw.ixgbe.txd=4096

(u can check actual value from shell "sysctl hw.ix.rxd(be aware from shell u need ix instat of ixgbe)).

System|Advanced|sysctl.conf

net.inet.tcp.cc.algorithm=htcp

u can also have a look here
https://calomel.org/freebsd_network_tuning.html

MFG Alex

[alraz]

Hi

sorry to necromance, but I'm having the exact same issue again since upgrading to 10.3.0.3 (not sure since when this was introduced).

Even with the interrupt moderation setting disabled, I only get about 1.79 GB/s.
Has the name of the setting changed recently?

[alraz]

In fact, I just did some testing and the enable_aim setting seems to be having no effect whatsoever

nas4free: ~# sysctl -w hw.ix.enable_aim=0
hw.ix.enable_aim: 1 -> 0
nas4free: ~# iperf3 -c 192.168.0.250
Connecting to host 192.168.0.250, port 5201
[ 4] local 192.168.0.253 port 60825 connected to 192.168.0.250 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 499 MBytes 4.19 Gbits/sec 0 1.79 GBytes
[ 4] 1.00-2.00 sec 505 MBytes 4.24 Gbits/sec 0 1.79 GBytes
[ 4] 2.00-3.00 sec 504 MBytes 4.23 Gbits/sec 0 1.79 GBytes
[ 4] 3.00-4.00 sec 505 MBytes 4.24 Gbits/sec 0 1.79 GBytes
[ 4] 4.00-5.00 sec 506 MBytes 4.25 Gbits/sec 0 1.79 GBytes
[ 4] 5.00-6.00 sec 506 MBytes 4.25 Gbits/sec 0 1.79 GBytes
[ 4] 6.00-7.00 sec 506 MBytes 4.25 Gbits/sec 0 1.79 GBytes
[ 4] 7.00-8.00 sec 506 MBytes 4.25 Gbits/sec 0 1.79 GBytes
[ 4] 8.00-9.00 sec 505 MBytes 4.24 Gbits/sec 0 1.79 GBytes
[ 4] 9.00-10.00 sec 505 MBytes 4.24 Gbits/sec 0 1.79 GBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 4.93 GBytes 4.24 Gbits/sec 0 sender
[ 4] 0.00-10.00 sec 4.93 GBytes 4.24 Gbits/sec receiver

iperf Done.

nas4free: ~# sysctl -w hw.ix.enable_aim=1
hw.ix.enable_aim: 0 -> 1
nas4free: ~# iperf3 -c 192.168.0.250
Connecting to host 192.168.0.250, port 5201
[ 4] local 192.168.0.253 port 11047 connected to 192.168.0.250 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 388 MBytes 3.26 Gbits/sec 0 1.79 GBytes
[ 4] 1.00-2.00 sec 486 MBytes 4.08 Gbits/sec 0 1.79 GBytes
[ 4] 2.00-3.00 sec 507 MBytes 4.25 Gbits/sec 0 1.79 GBytes
[ 4] 3.00-4.00 sec 505 MBytes 4.23 Gbits/sec 0 1.79 GBytes
[ 4] 4.00-5.00 sec 508 MBytes 4.26 Gbits/sec 0 1.79 GBytes
[ 4] 5.00-6.00 sec 507 MBytes 4.26 Gbits/sec 0 1.79 GBytes
[ 4] 6.00-7.00 sec 505 MBytes 4.23 Gbits/sec 0 1.79 GBytes
[ 4] 7.00-8.00 sec 505 MBytes 4.24 Gbits/sec 0 1.79 GBytes
[ 4] 8.00-9.00 sec 505 MBytes 4.23 Gbits/sec 0 1.79 GBytes
[ 4] 9.00-10.00 sec 507 MBytes 4.25 Gbits/sec 0 1.79 GBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 4.81 GBytes 4.13 Gbits/sec 0 sender
[ 4] 0.00-10.00 sec 4.81 GBytes 4.13 Gbits/sec receiver

iperf Done.

[alraz]

OK, got it figured out.

if someone else is having this same issue, apart from disabling active interrupt moderation as explained above by jasch, you also have to:
* go to system > advanced > sysctl.conf
* add a new setting: net.inet.tcp.tso=0
* reboot your system
* enjoy (optional)