*New 11.4 series Release:
2020-07-03: XigmaNAS - released!

*New 12.1 series Release:
2020-04-17: XigmaNAS - released

We really need "Your" help on XigmaNAS https://translations.launchpad.net/xigmanas translations. Please help today!

Producing and hosting XigmaNAS costs money. Please consider donating for our project so that we can continue to offer you the best.
We need your support! eg: PAYPAL

NVMe M.2 on PCIe 2.0 x4 – bottleneck, but on what?

XigmaNAS Basic Tune-up
Forum rules
Set-Up GuideFAQsForum Rules
Post Reply
Posts: 7
Joined: 24 Dec 2015 16:01
Status: Offline

NVMe M.2 on PCIe 2.0 x4 – bottleneck, but on what?


Post by cudak888 »

I’m chasing after a bottleneck on one of my servers. I’ve been itching to see if its possible to saturate my 10Gbe connection (NIC to NIC) with M.2 SSD’s, and I wound up upgrading a 512GB NVMe quicker than I thought, so I thought I’d run some tests.

I have the M.2 set up with Xigma in a Dell T7500 with a pre-existing Xigma install in it. This server usually gives me around 300-350mb/s (for both read/write) with my RAID-Z2 of 7200rpm drives. I threw the M.2 in as a striped drive with no RAID just for tests – it’s a Samsung PM981 M.2 with a theoretical 3000 MB/s read and 1800 MB/s write cap. It’s installed in the box with a PCIe adapter card (GLOTRENDS PA09_HS) which, near as I can see it, seems to be a direct passthrough device which doesn’t require any drivers. I already tested the card and confirmed its speeds when I had it installed in a Windows box.

This T7500 NAS is direct connected to my main box with Mellanox Connect-X2 PCie NICs running static IPv4’s. Jumbo packet on Windows is set at 9614 and receive buffers are 4096; these are the max settings for the Mellanox. On Xigma, I have the MTU at 9014, with “mtu 9014 rxcsum txcsum tso4 lro” set.

The main box has a Samsung 970 EVO M.2 in it and reads/writes at 2,200-2,400mb/s, so that’s fine as well.

However, when running AJA System Test (biased towards video editing tools) on Windows to check speed, I’m seeing anywhere from 480 to 600 MB/s write speed to the server’s M.2, and 530-580 MB/s read speed. This seems abnormally low – given everything I’ve set up to now, I would think that the 10Gbe connection should be saturated enough to be the weak link at this point.

This said, I’ve been cycling through all the possible bottlenecks, and I’m a bit baffled. The only one I can think of is the Dell T7500’s PCIe 2.0 limitation, but the adapter card is on an X4 slot. Theoretically, it be capping out somewhere in the 1,500mb/s range to the server itself, and at least 900-950mb/s when tested to the Windows box, shouldn't it?

Curious to hear what the forum thinks.

Posts: 779
Joined: 23 Jun 2012 07:36
Status: Offline

Re: NVMe M.2 on PCIe 2.0 x4 – bottleneck, but on what?


Post by sleid »

It all depends on the choice of the size of the test file you have selected in the software.
This speed can be normal with small files (256k to 1M) and abnormally low for 1G files and more. - Ingva (revision 7569)
FreeBSD 12.1-RELEASE-p4 #0 r360543M: Fri May 1 21:19:01 CEST 2020
X64-embedded sur Intel(R) Atom(TM) CPU C2750 @ 2.40GHz Boot UEFI
ASRock C2750D4I 2 X 8GB DDR3 ECC
Pool of 2 vdev Raidz1: 3 x WDC WD40EFRX + 3 x WDC WD30EZRX

Post Reply

Return to “XigmaNAS Basic Tune-up”