Very slow bandwidth on 2* Gbit NICs

[madgabz]

NAS4Free 9.1.0.1 (revision 847) x64-full
A_M_I_ OEMXSDT Intel(R) Core(TM)2 Duo CPU E6750 @ 2.66GHz 4608MiB RAM

Intel 82801I (ICH9) USB controller Marvell 88SE6121 UDMA133 controller
Intel 82801I (ICH9) USB 2.0 controller

---

Subject:
Very slow bandwidth on 2* Gbit NICs

---

Description:
I cannot seem to get above ~300 mbit/s on my nas4free server. This is on the
main NIC (Builtin realtek NIC), whereas the extra Intel NIC only gets 100
mbit/s, even if it autoselects 1000TX fullduplex)
The only thing between main computer and fileserver is a Netgear GS108T
gig-switch, which is operating at full speed on other test server/clients, using
iperf
I've tried to follow several tweak/tuning suggestions, but nothing has worked so
far.
Suggestions, even for some network troubleshooting?

[madgabz]

As a followup, i just changed cables, and a direct connection from PC to nas, same story, NIC stays at 300 mbit while the other stays at 100 mbit, so it seems that I have a software/driver/IPstack problem... How to troubleshoot?

some more hw info, from dmesg:

age0: <Attansic Technology Corp, L1 Gigabit Ethernet> mem 0xfe9c0000-0xfe9fffff irq 17 at device 0.0 on pci1
age0: 1280 Tx FIFO, 2364 Rx FIFO
age0: Using 1 MSI messages.

em0: <Intel(R) PRO/1000 Network Connection 7.3.2> port 0xec00-0xec1f mem 0xfebe0000-0xfebfffff,0xfeb00000-0xfeb7ffff,0xfebdc000-0xfebdffff irq 17 at device 0.0 on pci3
em0: Using MSIX interrupts with 3 vectors

ifconfig gives:

em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4319b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MCAST,WOL_MAGIC,VLAN_HWTSO>
ether 68:05:ca:16:c6:9f
inet 192.168.15.10 netmask 0xffffff00 broadcast 192.168.15.255
inet 192.168.15.11 netmask 0xffffffff broadcast 192.168.15.11
inet 192.168.15.12 netmask 0xffffffff broadcast 192.168.15.12
inet 192.168.15.13 netmask 0xffffffff broadcast 192.168.15.13
inet 192.168.15.19 netmask 0xffffffff broadcast 192.168.15.19
inet 192.168.15.18 netmask 0xffffffff broadcast 192.168.15.18
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 1000baseT <full-duplex>
status: active
age0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=c319b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MCAST,WOL_MAGIC,VLAN_HWTSO,LINKSTATE>
ether 00:1e:8c:56:b3:d8
inet 192.168.15.14 netmask 0xffffffff broadcast 192.168.15.14
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 1000baseT <full-duplex>
status: active
ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0xc
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

[crowi]

Did you work through this yet?
http://n4f.siftusystems.com/index.php/2 ... /comments/

[madgabz]

sysctl.conf.JPG

Yes, I've done all this! using diskinfo -t I managed to test all harddrives, they all deliver 60-110 MB/s (and confirmed this using dd), so it is not the harddisks/filesystems. Besides, i used iperf to test bandwidth, afaik that is totally separating the filesystem/harddisks from bandwidth measurement. Or...?
I am strongly suspecting, it is some sort of driver/system setting, screwing up the speed! Here's my sysctl.conf:

Code: Select all

MIB	Value	Comment
hw.acpi.power_button_state 	NONE 	Disable power button 
hw.intr_storm_threshold 	9000 	System tuning 
kern.coredump 	0 	Disable core dump 
kern.ipc.maxsockbuf 	16777216 	System tuning 
kern.ipc.nmbclusters 	262144 	System tuning 
kern.ipc.nmbjumbo9 	131072 	System tuning 
kern.ipc.nmbjumbo16 	65536 	System tuning 
kern.ipc.nmbjumbop 	262144 	System tuning 
kern.ipc.somaxconn 	8192 	System tuning 
kern.maxfiles 	65536 	System tuning 
kern.maxfilesperproc 	50000 	System tuning 
net.inet.icmp.icmplim 	300 	System tuning 
net.inet.icmp.icmplim_output 	1 	System tuning 
net.inet.tcp.delayed_ack 	0 	System tuning 
net.inet.tcp.path_mtu_discovery 	0 	System tuning 
net.inet.tcp.recvbuf_auto 	1 	System tuning 
net.inet.tcp.recvbuf_inc 	262144 	System tuning 
net.inet.tcp.recvbuf_max 	4194304 	System tuning 
net.inet.tcp.recvspace 	262144 	System tuning 
net.inet.tcp.rfc1323 	1 	System tuning 
net.inet.tcp.sendbuf_auto 	1 	System tuning 
net.inet.tcp.sendbuf_inc 	262144 	System tuning 
net.inet.tcp.sendbuf_max 	4194304 	System tuning 
net.inet.tcp.sendspace 	262144 	System tuning 
net.inet.udp.maxdgram 	57344 	System tuning 
net.inet.udp.recvspace 	65536 	System tuning 
net.local.stream.recvspace 	65536 	System tuning 
net.local.stream.sendspace 	65536 	System tuning 
security.jail.allow_raw_sockets 	1 	 
security.jail.chflags_allowed 	1 	 

[madgabz]

OK, so I finally got around to do a little more testing!
while running iperf i discovered some weird things: If i run iperf using 1 stream only, the connection maxes out at ~300 mbit! BUT, using several streams utilizes more bandwidth, though capping each stream at roughly ~200 mbit. Eg. using 2 streams utilizes ~400 mbit, each stream ~200 mbit
At 8 streams i seem to be able to max out my gigabit connection, which is really cool, but WHY can i not do this with just 1 stream. At no point did iperf use more than ~30% cpu, unless i had ntop running (a comprehensive network monitoring package). So i switched ntop off during testing.

What is going on here?

Here's the iperf output:

Server listening on TCP port 5001
TCP window size: 256 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.15.10 port 5001 connected with 192.168.15.100 port 63698
[ 6] local 192.168.15.10 port 5001 connected with 192.168.15.100 port 63700
[ 7] local 192.168.15.10 port 5001 connected with 192.168.15.100 port 63701
[ 5] local 192.168.15.10 port 5001 connected with 192.168.15.100 port 63699
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-30.0 sec 586 MBytes 164 Mbits/sec
[ 6] 0.0-30.0 sec 691 MBytes 193 Mbits/sec
[ 7] 0.0-30.0 sec 671 MBytes 188 Mbits/sec
[ 5] 0.0-30.0 sec 698 MBytes 195 Mbits/sec
[SUM] 0.0-30.0 sec 2.58 GBytes 740 Mbits/sec
[ 8] local 192.168.15.10 port 5001 connected with 192.168.15.100 port 63758
[ 8] 0.0-60.0 sec 2.11 GBytes 303 Mbits/sec
[ 4] local 192.168.15.10 port 5001 connected with 192.168.15.100 port 63851
[ 5] local 192.168.15.10 port 5001 connected with 192.168.15.100 port 63852
[ 7] local 192.168.15.10 port 5001 connected with 192.168.15.100 port 63854
[ 6] local 192.168.15.10 port 5001 connected with 192.168.15.100 port 63853
[ 4] 0.0-60.0 sec 1.35 GBytes 193 Mbits/sec
[ 7] 0.0-60.0 sec 1.37 GBytes 196 Mbits/sec
[ 6] 0.0-60.0 sec 1.15 GBytes 164 Mbits/sec
[ 5] 0.0-60.0 sec 1.31 GBytes 188 Mbits/sec
[SUM] 0.0-60.0 sec 5.17 GBytes 740 Mbits/sec
[ 8] local 192.168.15.10 port 5001 connected with 192.168.15.100 port 64009
[ 5] local 192.168.15.10 port 5001 connected with 192.168.15.100 port 64011
[ 6] local 192.168.15.10 port 5001 connected with 192.168.15.100 port 64012
[ 7] local 192.168.15.10 port 5001 connected with 192.168.15.100 port 64013
[ 11] local 192.168.15.10 port 5001 connected with 192.168.15.100 port 64016
[ 9] local 192.168.15.10 port 5001 connected with 192.168.15.100 port 64014
[ 4] local 192.168.15.10 port 5001 connected with 192.168.15.100 port 64010
[ 10] local 192.168.15.10 port 5001 connected with 192.168.15.100 port 64015
[ 8] 0.0-60.0 sec 782 MBytes 109 Mbits/sec
[ 5] 0.0-60.0 sec 853 MBytes 119 Mbits/sec
[ 6] 0.0-60.0 sec 819 MBytes 114 Mbits/sec
[ 7] 0.0-60.0 sec 781 MBytes 109 Mbits/sec
[ 11] 0.0-60.0 sec 819 MBytes 115 Mbits/sec
[ 9] 0.0-60.0 sec 861 MBytes 120 Mbits/sec
[ 4] 0.0-60.0 sec 860 MBytes 120 Mbits/sec
[ 10] 0.0-60.0 sec 850 MBytes 119 Mbits/sec
[SUM] 0.0-60.0 sec 6.47 GBytes 926 Mbits/sec
[ 12] local 192.168.15.10 port 5001 connected with 192.168.15.100 port 64097
[ 5] local 192.168.15.10 port 5001 connected with 192.168.15.100 port 64099
[ 6] local 192.168.15.10 port 5001 connected with 192.168.15.100 port 64100
[ 7] local 192.168.15.10 port 5001 connected with 192.168.15.100 port 64101
[ 9] local 192.168.15.10 port 5001 connected with 192.168.15.100 port 64103
[ 11] local 192.168.15.10 port 5001 connected with 192.168.15.100 port 64105
[ 10] local 192.168.15.10 port 5001 connected with 192.168.15.100 port 64104
[ 4] local 192.168.15.10 port 5001 connected with 192.168.15.100 port 64098
[ 8] local 192.168.15.10 port 5001 connected with 192.168.15.100 port 64102
[ 12] 0.0-60.0 sec 746 MBytes 104 Mbits/sec
[ 5] 0.0-60.0 sec 696 MBytes 97.3 Mbits/sec
[ 6] 0.0-60.0 sec 762 MBytes 107 Mbits/sec
[ 7] 0.0-60.0 sec 746 MBytes 104 Mbits/sec
[ 9] 0.0-60.0 sec 696 MBytes 97.3 Mbits/sec
[ 11] 0.0-60.0 sec 747 MBytes 104 Mbits/sec
[ 10] 0.0-60.0 sec 764 MBytes 107 Mbits/sec
[ 4] 0.0-60.0 sec 719 MBytes 100 Mbits/sec
[ 8] 0.0-60.0 sec 719 MBytes 101 Mbits/sec
[SUM] 0.0-60.0 sec 6.44 GBytes 922 Mbits/sec