LSI 9240-8i performance: x4 slot = x8 slot in X9SCM?
Posted: 02 Dec 2014 22:14
I’m not sure this example proves that the full SATA 3 bandwidth is pretty much unusable for conventional 5400 RPM HDDs, but I’m hoping it’ll save me money without sacrificing the current level of performance on my home server.
I have the need to add two more HBAs so I can add more HDDs via PCI pass-through to the VMWare ESXi virtual machines. My Supermicro X9SCM motherboard has four physical PCIe x8 slots, but the actual bandwidth of the slots are x8, x8, x4, x4, with the last x4 slot being PCH connected, not CPU. My current HBA in one x8/x8 slot is an LSI 9240-8i (2308 based) flashed to IT mode, used by the NAS4Free VM for a 5 drive (advanced format-4KB sector, 3 drives on port0, 2 drives on port1) RAIDZ1 array (for testing, destined to be a RAIDZ2 6 drive array as soon as the next drive arrives). I realized that the 9240-8i was designed for an x8/x8 slot and I only had two of those available…so I thought I was about to be forced to switch to a new MB, as in an ASRock C226WS+. Thankfully, I decided to test the 9240-8i in the CPU connected x4 slot first…and it worked just fine, as did the PCH x4 slot. So then my worry was bandwidth, how bad would this hurt performance? Simple answer: not one bit! Even the x4 PCH based slot worked great. My first bandwidth test was simple, just a ZFS scrub of 1.37TB of uncompressed data. The second test was to duplicate ( cp –R /mnt/tank/dataset1/source /mnt/tank/dataset1/target ) a 10GB directory within a dataset (dedup off).
Scrub Results:
PCIe x8 CPU slot - 1h14m (about 308MB/s)
PCIe x4 CPU slot – 1h15m (ditto)
PCIe x4 PCH slot – 1h15m (ditto)
Copy Results:
PCIe x8 CPU slot – 1m16s (about 131MB/s)
PCIe x4 CPU slot – (ditto)
PCIe x4 PCH slot – (ditto)
Note that the average CPU utilization during the scrub was about 15-20%, never jumping over 30%. The copy was much less. The files were pretty much unfragmented and nothing else was running within ESXi or the NAS4Free VM. I built up the total 1.37TB size by duplicating a 10GB directory (mostly larger file sizes) 10 times into a 100GB directory, then duplicating that 10 times into a 1TB directory.
Granted, these were very simple tests, but what really surprised me was that there was no meaningful difference at all. So at this point, instead of buying a couple more 9240-8i, I think I’ll test out a less expensive LSI SATA 2 HBA first and see what that does.
One additional nice thing: both ESXi and NAS4Free always found and reconnected/reconfigured themselves to the 9240-8i as I moved it around the different slots, no input from me required.
Configuration:
Supermicro X9SCM-F, E3-1245v2, 16GB ECC/1600 memory, 1 LSI 9240-8i/IT Mode, 5 WD Red 2TB
ESXi 5.5 Update 2 – VM: NAS4Free 9.2.0.1 (3 threads, 6GB memory) with pass-through to 9240-8i
I have the need to add two more HBAs so I can add more HDDs via PCI pass-through to the VMWare ESXi virtual machines. My Supermicro X9SCM motherboard has four physical PCIe x8 slots, but the actual bandwidth of the slots are x8, x8, x4, x4, with the last x4 slot being PCH connected, not CPU. My current HBA in one x8/x8 slot is an LSI 9240-8i (2308 based) flashed to IT mode, used by the NAS4Free VM for a 5 drive (advanced format-4KB sector, 3 drives on port0, 2 drives on port1) RAIDZ1 array (for testing, destined to be a RAIDZ2 6 drive array as soon as the next drive arrives). I realized that the 9240-8i was designed for an x8/x8 slot and I only had two of those available…so I thought I was about to be forced to switch to a new MB, as in an ASRock C226WS+. Thankfully, I decided to test the 9240-8i in the CPU connected x4 slot first…and it worked just fine, as did the PCH x4 slot. So then my worry was bandwidth, how bad would this hurt performance? Simple answer: not one bit! Even the x4 PCH based slot worked great. My first bandwidth test was simple, just a ZFS scrub of 1.37TB of uncompressed data. The second test was to duplicate ( cp –R /mnt/tank/dataset1/source /mnt/tank/dataset1/target ) a 10GB directory within a dataset (dedup off).
Scrub Results:
PCIe x8 CPU slot - 1h14m (about 308MB/s)
PCIe x4 CPU slot – 1h15m (ditto)
PCIe x4 PCH slot – 1h15m (ditto)
Copy Results:
PCIe x8 CPU slot – 1m16s (about 131MB/s)
PCIe x4 CPU slot – (ditto)
PCIe x4 PCH slot – (ditto)
Note that the average CPU utilization during the scrub was about 15-20%, never jumping over 30%. The copy was much less. The files were pretty much unfragmented and nothing else was running within ESXi or the NAS4Free VM. I built up the total 1.37TB size by duplicating a 10GB directory (mostly larger file sizes) 10 times into a 100GB directory, then duplicating that 10 times into a 1TB directory.
Granted, these were very simple tests, but what really surprised me was that there was no meaningful difference at all. So at this point, instead of buying a couple more 9240-8i, I think I’ll test out a less expensive LSI SATA 2 HBA first and see what that does.
One additional nice thing: both ESXi and NAS4Free always found and reconnected/reconfigured themselves to the 9240-8i as I moved it around the different slots, no input from me required.
Configuration:
Supermicro X9SCM-F, E3-1245v2, 16GB ECC/1600 memory, 1 LSI 9240-8i/IT Mode, 5 WD Red 2TB
ESXi 5.5 Update 2 – VM: NAS4Free 9.2.0.1 (3 threads, 6GB memory) with pass-through to 9240-8i