Only 55 MB/s read performance?

[actyler1001]

Hi kids! I need some help trying to diagnose my ZFS/iSCSI home NAS/Shared storage solution performance issue. I have 6, standard 7200 RPM traditional SATA drives segregated into 3 different mirror vdev's and then created 1 ZFS volume striped across the vdev's.. Everything I have read said that this will give you best performance while still granting redundancy. So the way I figure I should get approximately 3 times the performance of a single disk with the benefit of the stripe and sacrificing 3 drives to redundancy.

From what I have been able to figure is that a single drive's read performance stabilizes at about 60 MB/s.. So in theory, I should have absolutely no problem completely saturating a single 1 Gbps NIC link at approximately 100-130 MB/s. Unfortunately I am only getting about 55 MB/s read when I copy data off this NAS. What the hell? Where do you guys think my break down is? Here is more info on the setup:

Server: two 6 core CPU AMD.
16 GB DDR 2 RAM - Runs about 85% utilized based on Nas4Free UI.
older sata controller - SATA II

Connection to network - iSCSI presentation of ZFS volume using multi-path I/O over two 1 GB Links. Read speed test performed when copying files from a VM hosted within VMDK on ZFS volume.

[Parkcomm]

Hey actyler1001,

Your logic is sound - I can saturate a 1G link with 2 x mirrored vdevs - and slower disks. Also Sata II.

So you bottlenecks could be:

Your CPU (you seem to have plenty of grunt, but maybe you are running other programs at the same time)

Code: Select all

top

You NIC

Code: Select all

systat -vmstat

Check the interrupts

your sata controller - especially if you are using a SATA multiplier (SATA Mux)

your disks:
check SMART

Your config:
I think you have some complications in there that it might be better to simplify (ie iSCSI + mulitpath + VM etc any one of which coiuld have a problem). I also don't quite understand the config as described, is the VM a guest on the nas4free host? Anyway without knowing anything else, test a single NIC, no VMs, etc and get that working first.

Your test environment:
What software are using to test throughput? (I like https://www.freebsd.org/cgi/man.cgi?iostat)
Are you maxing the throughput on the test machine?

[b0ssman]

please go through this list step by step
http://n4f.siftusystems.com/index.php/2 ... /comments/

[Parkcomm]

I could have saved a lot of typing there

[actyler1001]

Thanks a ton guys. I will look through the article and get back to you with my findings.. At a glance it looks like it might be 512 vs 4k sector related. The drives I have in this NAS were basically laying around the house and are a mixed bag.. I also agree that testing directly from an NFS or CIF share is probably a better approach. Currently was just using a Windows 10 box with a 1 GB connected to the same switch and watching was Windows Explorer would report for transfer speed.

[actyler1001]

Ya, so.... The drive review doesn't look good. I am having some trouble looking up all of the specifications, but this is what I have found so far:

vdev 1
ada0 WDC WD1002FBYS-02A6B0 953870MB WD-WMATV8594822 ahcich0 AMD SB7x0/SB8x0/SB9x0 AHCI SATA controller
(SATA II and 32MB Cache says Newegg)
ada1 ST31000524NS 953870MB 9WK4EMQN ahcich1 AMD SB7x0/SB8x0/SB9x0 AHCI SATA controller
(SATA II 32MB Cache says Newegg)

vdev 2
ada3 WDC WD1002FAEX-00Z3A0 953870MB WD-WCATRC428924 ahcich3 AMD SB7x0/SB8x0/SB9x0 AHCI SATA controller
(SATA III 64MB Cache says Newegg)
ada4 ST31000524NS 953870MB 9WK4G685 ahcich4 AMD SB7x0/SB8x0/SB9x0 AHCI SATA controller
(SATA II 32MB Cache says Newegg)

vdev 3
ada2 ST1000DM003-9YN162 953870MB W1D0EJT2 ahcich2 AMD SB7x0/SB8x0/SB9x0 AHCI SATA controller
(SATA III 64MB Cache says Newegg)
ada5 WDC WD10EACS-14ZJB0 953870MB WD-WCASJ1941265 ahcich5 AMD SB7x0/SB8x0/SB9x0 AHCI SATA controller
(SATA II 16MB cache says Newegg - I realize this is a green drive and that is BAD for NAS.. I hacked the firmware to disable green drive power down feature and thought I would give it a try)

nas4free: ~# zdb | grep ashift
ashift: 9
ashift: 9
ashift: 12


So, that all said.. Keep the WDC WD1002FAEX-00Z3A0 & ST1000DM003-9YN162.. Then buy 4 new RED WD drives.. Then backup all my data, blow up the pool and re-create?

Or would you suggest a different route?

[Parkcomm]

it looks like it might be 512 vs 4k sector related

Probably not. You are looking to increase throughput by 300% - you might get 10% of that through sector alignment. (if alignment is the only issue). However why don't you start by formatting all drives with the 4k sector alignment - Disks|Format| Enable Advanced Format (4KB sector)

Having said that zpools do tend to operate at the speed of the worst performing disk, so mixed bags like yours can be slower than expected, but 3 x vdev of your cavier green would still easily saturate a 1G link.

If I had your setup - I'd replace the green for sure, but I'd keep the rest until I needed more space or they started playing up.

[actyler1001]

Thanks Parkcomm.. Quick question though. As I am swapping drives and formatting with the new 4KB sector, do I need to rebuild the pool and vdevs or can I start converting the vdev as drives are replaced and the vdev/pool resilvered?

[b0ssman]

an existing pool can not be converted to ashift 12

[actyler1001]

Well hell.....

[actyler1001]

What is the easiest way to find out if your drive supports a 4kb sector size?

[b0ssman]

the smart report should tell you.

[b0ssman]

but as Parkcomm that will not be your main problem with the speed of your system.

[actyler1001]

Ok.. I will step through the performance troubleshooting link you gave me and report back before doing anything too rash. At the risk of sounding stupid.. How do I generate the SMART report? I see it is enabled and I have configured it for e-mail delivery, but it hasn't seemed to work thus far. A manual way to run it from the CLI?

[Parkcomm]

Is it here Diagnostics|Information|S.M.A.R.T.?