Realtek, Atheros, Intel, the thing about speed

[Snufkin]

Some time ago did N4F hardware upgrade and had the opportunity to compare network controllers of three vendors.
Got the results which are interesting not the absolute numbers but the relative increase or decrease in speed.

Good help was Nas4free - Performance Troubleshooting

Here is summary in three parts, 1) Final numbers, 2) Technical details and 3) The problem

PART 1, Final numbers
All HDD's were connected directly to ASUS P5L-MX Intel ICH7, SATA II motherboard (no AHCI and NSQ features)
DD test
- write 88 619 632 bytes/sec
- read 93 660 194 bytes/sec

iPerf test results presented in two numbers depending on N4F role, client/server (or iPerf3 reverse mode)
Samba transfer rate is something in between upload and download speed when moving file to/from MS Windows host.

RealTek
iPerf: 328/579 Mbits/sec
Samba: 20-25 MB/s

Atheros
iPerf: 759/814 Mbits/sec
Samba: 55-60 MB/s

Intel
iPerf: 915/921 Mbits/sec
Samba: 75-80 MB/s

PART 2, Technical details
S/w NAS4Free 11.0.0.4 - Sayyadina (3460)

RealTek
D-Link DGE-528T PCI card (not PCI-E or PCI-X), remains from the previous h/w config, taken as a starting point.
TSO4 (TCP segmentation offload) disabled by default (in FreeBSD driver settings).

Code: Select all

re0@pci0:1:1:0: class=0x020000 card=0x43001186 chip=0x43001186 rev=0x10 hdr=0x00
    vendor     = 'D-Link System Inc'
    device     = 'DGE-528T Gigabit Ethernet Adapter'

Code: Select all

re0: <D-Link DGE-528(T) Gigabit Ethernet Adapter> port 0xc800-0xc8ff mem 0xdfeffc00-0xdfeffcff irq 21 at device 1.0 on pci2
re0: Using defaults for TSO: 65518/35/2048
re0: netmap queues/slots: TX 1/256, RX 1/256

Code: Select all

miibus1: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus1

Atheros
ASUS P5L-MX integrated network controller
TSO4 feature enabled by default. May hang under trafic load (known bug at least with FreeBSD and MS Windows drivers).
To make it stable TSO4 is disabled in ifconfig additional parameters.

Code: Select all

age0@pci0:2:0:0: class=0x020000 card=0x82261043 chip=0x10481969 rev=0xb0 hdr=0x00
    vendor     = 'Qualcomm Atheros'
    device     = 'Attansic L1 Gigabit Ethernet'
    cap 05[48] = MSI supports 1 message, 64 bit enabled with 1 message
    cap 10[58] = PCI-Express 1 endpoint max data 128(128)
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s)

Code: Select all

age0: <Attansic Technology Corp, L1 Gigabit Ethernet> mem 0xdfec0000-0xdfefffff irq 17 at device 0.0 on pci2
age0: 1280 Tx FIFO, 2364 Rx FIFO
age0: Using 1 MSI messages.
age0: Using defaults for TSO: 65518/35/2048

Intel
Intel EXPI9301CT PCI-E card, the planned ultimate goal of upgrade, but not achieved (see The problem below).

Code: Select all

em0@pci0:2:0:0: class=0x020000 card=0xa01f8086 chip=0x10d38086 rev=0x00 hdr=0x00
    device     = '82574L Gigabit Network Connection'
    vendor     = 'Intel Corporation'
    cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
    cap 10[e0] = PCI-Express 1 endpoint max data 128(256) RO NS
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)
    cap 11[a0] = MSI-X supports 5 messages
                 Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]

Code: Select all

em0: <Intel(R) PRO/1000 Network Connection 7.6.1-k> port 0xe800-0xe81f mem 0xdffe0000-0xdfffffff,0xdff00000-0xdff7ffff,0xdffdc000-0xdffdffff irq 16 at device 0.0 on pci1
em0: Using MSIX interrupts with 3 vectors
em0: netmap queues/slots: TX 1/1024, RX 1/1024

PART 3, The Problem
N4F server with Intel EXPI9301CT card works great, but for up to 24 hours, not more.

After server boot both CPU cores are mostly idle.

Code: Select all

last pid:  2526;  load averages:  0.07,  0.10,  0.08                    up 0+00:43:28  23:38:31
656 processes: 3 running, 634 sleeping, 19 waiting
CPU 0:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
CPU 1:  1.2% user,  0.0% nice,  0.0% system,  0.0% interrupt, 98.8% idle
Mem: 92M Active, 976M Inact, 742M Wired, 188M Buf, 14M Free
Swap:

  PID USERNAME     PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
   11 root         155 ki31     0K    48K CPU0    0  42:19  99.54% idle{idle: cpu0}
   11 root         155 ki31     0K    48K RUN     1  42:07  98.82% idle{idle: cpu1}
 2105 transmission  20    0   114M 28800K kqread  1   0:59   1.04% transmission-daemon{transmis
   12 root         -92    -     0K   456K WAIT    0   0:26   0.18% intr{irq257: em0:rx0}
 2519 root          20    0 24216K  4492K CPU1    1   0:04   0.17% top
   12 root         -92    -     0K   456K WAIT    1   0:06   0.05% intr{irq258: em0:tx0}
   12 root         -60    -     0K   456K WAIT    1   0:01   0.04% intr{swi4: clock (0)}
   12 root         -100    -     0K   456K WAIT    0   0:02   0.02% intr{irq20: hpet0 uhc}

Clearly visible MSI-X option (separate rx and tx irq) with rather low interrupt rate.

Code: Select all

interrupt                          total       rate
irq4: uart0                       105878          4
irq14: ata0                       565320         20
irq20: hpet0 uhci0+             10825011        391
irq23: atapci1                    761198         28
irq257: em0:rx0                 27335698        988
irq258: em0:tx0                 32430146       1172
Total                           72023251       2603

At some point suddenly server almost hangs - pings up 1.5 s, huge delay in WebUI response, a lot of Samba error records in journal and one of CPU cores completely loaded with rx interrupts.

Code: Select all

last pid:  9349;  load averages:  2.46,  1.13,  0.49                                       up 0+20:53:18  19:48:21
656 processes: 4 running, 633 sleeping, 18 waiting, 1 lock
CPU 0:  0.0% user,  0.0% nice,  0.0% system,  100% interrupt,  0.0% idle
CPU 1:  0.0% user,  0.0% nice, 26.3% system,  0.4% interrupt, 73.4% idle
Mem: 96M Active, 941M Inact, 751M Wired, 188M Buf, 36M Free
Swap:

  PID USERNAME     PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
   12 root         -92    -     0K   456K CPU0    0   4:41  99.75% intr{irq257: em0:rx0}
   11 root         155 ki31     0K    48K RUN     1  20.6H  73.11% idle{idle: cpu1}
    0 root         -92    -     0K 12912K *em0:r  1   0:39  25.65% kernel{em0 rxq (cpuid 0}
 2105 transmission  20    0   124M 27080K kqread  1  13:15   0.76% transmission-daemon{transmission-dae}
   12 root         -100    -     0K   456K WAIT    0   0:26   0.41% intr{irq20: hpet0 uhc}
 9349 root          20    0 24216K  5348K CPU1    1   0:00   0.21% top
   12 root         -60    -     0K   456K WAIT    1   0:27   0.04% intr{swi4: clock (0)}
   12 root         -92    -     0K   456K WAIT    1   0:36   0.01% intr{irq258: em0:tx0}

But from interrupts point of view nothing was changed

Code: Select all

interrupt                          total       rate
irq4: uart0                        64467          4
irq14: ata0                       356847         21
irq20: hpet0 uhci0+              5715757        339
irq23: atapci1                    436877         26
irq257: em0:rx0                 15136419        898
irq258: em0:tx0                 17076172       1013
Total                           38786539       2302

Enable/disable/change FreeBSD sysctl hw.em tunables, swap, N4F services, IPv6 protocol gave nothing.
Server was unstable and I had to return Intel card to the seller. Guess it was 'fake-Intel' even with Yottamark code.

Comments and question are welcome.

[jonyyybe]

hi there, quite nice your testing and figures but in the same time i reckon that the problem could be from drivers/card issues/incompatibility but not fake(dont think that they tried to do such a fake) - make sense?

[Snufkin]

...i reckon that the problem could be from drivers/card issues/incompatibility but not fake(dont think that they tried to do such a fake) - make sense?

Sure your point makes sense, the reasons may be very different

• Motherboard (Intel 82574L integrated) firmware bug, Packets of Death
• FreeBSD em() driver bugs, Watchdog timeout, TSO4 issue,
• Simply bad PCI-E x1 slot on ASUS motherboard.

Would be nice to build test stand with dedicated LAN segment, full FreeBSD installation and traffic generator, then raise kernel or/and network stack debugging level and browse numerous system logs.
I bought my own free time and server stability for 150 Mbits/sec less speed then it could be. Good deal, huh?