troubleshooting ZFS performance

[rs232]

Hi all, I though it's a good idea to open a topic to gather infor from users on how you did or you would troubleshoot poor transfer rate when using zfs.
I do experience problems and my scenario is rather complex.
I run Nas4free into a vmware VM (ESXi 5.0 host) with data disk raw mapped (virtual) and data accessed via samba and I do have the zfs kernel tune.
Things I can think about are:
When I see that consistent data transfer dropping to pretty much 1/10 randomly, what is it causing it exactly? Is it samba? Is it zfs? Is it the storage itself?
So I guess what I'm asking is: is there any real-time way to monitor e.g. the buffers? or else?
Any best practice please post below.

Thanks!
rs232

[Tomsk]

My N4F server is still in the experimental stage at the moment so I am able to conduct a few tests.
I'm using two SSD in RAID0 on my desktop so I know there's no transfer bottlenecks there.

SAMBA/CIFS shares

Transfering to a single 3TB HD (UFS) I get >100MB/s on writes and reads. On reads however, after approx. 4GB has been transferred the speed drops to about ~80MB/s.

Transfering to a 3x2TB RAIDZ1 or a 4x1TB RAIDZ1 on my server write speed fluctuates, peaking at>110MB/s and dropping to zero. Read speed seems to be OK and steady at >100MB/s
I've experimented with some MTU and buffer settings but it makes little difference.

Edit
Similar results with ftp.

[rs232]

I really appreciated you answer! Thanks!!
That's the type of input I was looking for. If you ask me I'm rather disappointed with the performance of ZFS in general despite the high spec of the VM where I run nas4free.
I have 4x2Tb and do se jumbo frames (9K). My ZFS RAIDZ1 performance is unpredictable, e.g. 75MB/s, then down to 10, up to 25, pretty much random. One of the things I've noticed is that copying via CIFS (writing) using teracopy on windows, often it gets to the end on e.g. 70MB/s however when it reaches 100% it stays there for even a minute, like verifying the checksum or something. Probably I made the mistake to enable deduplication and the 6GB of RAM might not be enough.
I'm seriously thinking to revert back to software raid using JBOD or raid0 (as I back up data offsite).
Is there any chance you can test JBOD, RAID-0 and RAID-5 on the very same HW and see how it goes compared to your previous test?

Again... thanks!

[raulfg3]

if you have the oportunity to test, have a look at OMV

[Tomsk]

I set up my 3TB drives as a software RAID0.
Formatted as UFS and mounted via SAMBA - read & writes were a consistant ~120MB/s
Formatted and used via ZFS as a single softRAID0 device, write speeds fluctuated bewteen 120 and 0 MB/s. Read speeds were not tested.

The 3TB drives setup as softRaid1 and used via ZFS still results in some occasional drops to 0MB/s, but not not as bad as a RAIDZ setup.

[rs232]

That's a priceless input thanks for that. Did you get the chance to test JBOD?

[xbmcg]

RAID0 is kind of JBOD (but R/W acess is distributed across all disks in a round-robin maner, while JBOD just appends next disk when the first one is full), so RAID0 delivers usually a better R/W performance.

[rs232]

RAID0 is kind of JBOD (but R/W acess is distributed across all disks in a round-robin maner, while JBOD just appends next disk when the first one is full), so RAID0 delivers usually a better R/W performance.

That was already clear thanks. However there are cases where JBOD is faster than raid0 e.g. when one of the disks in the pool has different firmware or slightly different built (more/less platter disks) but same capacity thus sold as same product. In these cases raid0 *might* be slower than JBOD.

Hence the request to test this for reference, and to complete the test.

[rs232]

Tomsk can you please also give us an input on the network HW you're testing this on?
I mean is this a home network or a business one, what's the brand/model of the network devices and what's the MTU in use? (Jumbo frames enabled I guess)

Many thanks again

[Tomsk]

RAID0 is kind of JBOD (but R/W acess is distributed across all disks in a round-robin maner, while JBOD just appends next disk when the first one is full), so RAID0 delivers usually a better R/W performance.

That's what I figured, so I'm not going to load up one drive with ~3TB of data just to see what happens when the next drive has to be used!

That was already clear thanks. However there are cases where JBOD is faster than raid0 e.g. when one of the disks in the pool has different firmware or slightly different built (more/less platter disks) but same capacity thus sold as same product. In these cases raid0 *might* be slower than JBOD.

Hence the request to test this for reference, and to complete the test.

If this was the case then it's not the performance of ZFS that is in question but the performance of each HD. All my test HD's are identical anyway.

Tomsk can you please also give us an input on the network HW you're testing this on?
I mean is this a home network or a business one, what's the brand/model of the network devices and what's the MTU in use? (Jumbo frames enabled I guess)

Many thanks again

My server is in my signature below.
My main PC is an i5@3Ghz with 16GB of RAM.
Both are using onbaord ethernet interfaces which are Realtek.
The switch is a HP 1400-8g
MTU is windows default - 1472


I don't think I'll be doing more tests as it seems to me that it is ZFS that is slowing performance on writes, probably due to error correction overheads.

My server is for my home and will be used mainly for data backups and storage of my DVD (& future Blu-ray) collection, so the erratic write speed is only a minor incovenience and annoyance that I can live with.

[rs232]

Is it possible that these ZFS spikes are experienced only when data is accessed/shared via CIFS? e.g. do you get the same results doing rsync over ssh?

[ku-gew]

I would check the router, if you are using one. If you are using a switch you can check it anyway, but it should be fine.
My router cannot go past 70-75 MB/s and I think the performances of the net are affected by it more than the bare throughput value would let me think.

[rs232]

the route is definitely the bottleneck and even in the past with jumbo frames enabled I got to maximum 80MB/sec. And that's more than enough for me.
My question was related to ZFS thought as the speed of my transfer goes from 80 to 20 then 70 then 0 the 35 than [add random numbers]...

[raulfg3]

to minimize spikes in SAMBA transfer , play with Jumbo Frames ( on / OFF and different sizes), and play with diferents RX and TX buffer 64240 is default, but I have good experience using 64240/2=32120 and 64240x2=128480

use allways same big ISO file for transfer to / from NAS to do the test comparative.

[rs232]

I'm copying data (reading) off the NAS using rsync over ssh.

Code: Select all

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
raid5       4.74T  2.51T      1     29   127K   136K
  raidz1    4.74T  2.51T      1     29   127K   136K
    da2p2       -      -      1     12  31.8K  49.5K
    da3p2       -      -      1     11  31.7K  48.1K
    da4p2       -      -      1     12  31.7K  49.5K
    da5p2       -      -      1     11  31.7K  47.8K
----------  -----  -----  -----  -----  -----  -----

Can anybody explain me why read operation is 1 and write operations is 11-12?
I currently get 10MB/sec reading from nas4free with this zfs pool

[raulfg3]

strange I have a similar ZFS pool, only difers on ada1 for me and da2p2 for you.

and I have:

Code: Select all

Pool information and status

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
NAS-A       8.48T   598G      6     17   461K  82.8K
  raidz1    8.48T   598G      6     17   461K  82.8K
    ada1        -      -      4      9  94.2K  22.0K
    ada3        -      -      4      9  82.3K  22.0K
    ada2        -      -      4      9  88.9K  22.0K
    ada4        -      -      4      9  97.9K  22.0K
    ada5        -      -      4      9  99.0K  22.0K
----------  -----  -----  -----  -----  -----  -----

Do you use partitions instead of all disk (device)?