XigmaNAS

I gave kmem 1.5G because I wasn't sure how much to reserve for userspace: I use mostly Samba and rsync.
Given the 1.5G kmem, I then allocated 1G to ARC (maxsize), but I haven't changed the minsize because I want to leave ZFS and the kernel free to decide.

That extension was taken into account, but never installed because somehow I thought it was difficult to install. I'm not sure now, but I put in System|Advanced|sysctl.conf (and apparently also in loader.conf) the following lines:
kern.maxvnodes 250000
vfs.zfs.arc_max 512M
vm.kmem_size 1G

That, I notice now, are not what I wanted. I change them to 1.5G and 1G, could someone tell me where they should go (loader vs sysctl) and whether additional variables are needed?
Thanks

you might also try setting:

vfs.zfs.vdev.min_pending=2
vfs.zfs.vdev.max_pending=3

this limits the number of simultaneous I/O that ZFS sends to your drives. If the drives don't order the seeks well, this can help. Turning off write caching on the drives may also help with or without these settings for the same reason.

Installing the ZFS Kernel tune extension is very simple. viewtopic.php?f=71&t=1278

You can either create a script or type out the following commands at the command line.
mkdir /mnt/data/zfskerntune
cd /mnt/data/zfskerntune
fetch http://www.peach.ne.jp/archives/nas4fre ... 031.tar.gz
tar xvf zfskerntune-20121031.tar.gz
cd zfskerntune
./zfskerntune-install.php

Then refresh webgui and set the memory size in ZFS Kernel Tune extension. Restart and test.

If you plan on doing changes yourself they go in the loader.conf file. I would avoid making the arc size very large if you only have 2GB RAM. My opinion is that it helps to set kmem and arc sizes (min/max) that don't interfere with each other when you have smaller amounts of memory. So for 2GB RAM I would use a maximum of kmem=1.5G and arc=.5G so that each gets its own memory space and there is less chance for memory contention problems. Also note that ZFS Kernel Tune extension adds other settings to loader.conf file that can affect performance.

Here is what ZFS Kernel Tune has when 2GB memory is selected
# ZFS v28 kernel tune
vfs.zfs.prefetch_disable="1"
vfs.zfs.txg.timeout="5"
vfs.zfs.vdev.max_pending="10"
vfs.zfs.vdev.min_pending="4"
vfs.zfs.write_limit_override="0"
vfs.zfs.no_write_throttle="0"

# X32 2048M
#vm.kmem_size="1400M"
#vfs.zfs.arc_min="400M"
#vfs.zfs.arc_max="400M"

# X64_2GB
#vm.kmem_size="1536M"
#vfs.zfs.arc_min="512M
#vfs.zfs.arc_max="512M"

Again my opinion is the easiest solution would be to just let the ZFS Kernel Tune extension make the changes.

00Roush

Hello, the problem didn't really disappear, even after I tweaked the settings.
Now I read this: http://dtrace.org/blogs/ahl/2014/02/10/ ... -throttle/

Here’s how that scenario would play out with the old write throttle assuming full quiesced and syncing transaction groups (you may want to refer to my last blog post for a refresher on the mechanism and some of the numbers). With a target of 5 seconds to write out its contents, the currently open transaction group would be limited to 500MB. Recall that after 7/8ths of the limit is consumed, the old write throttle starts inserting a 10ms delay, so the first 437.5MB would come sailing in, say, with an average latency of 780µs, but then the remaining writes would average at least 10ms (scheduling delay could drive this even higher). With this artificially steady rate, the delay would occur 7/8ths of the way into our 5 second window, with 1/8th of the total remaining. So with 5/8ths of a second left, and an average latency of 10ms, the client would be able to write only and additional 500KB worth of data. More simply: data would flow at 100MB/s most of the time, and at less than 1MB/s the rest.

It seems a reasonable explanation? even if I am far below the disks capabilities, I get full throttle until the end of the 5 seconds interval, then an unreasonably low rate, then back again.

Has anyone enough experience/knowledge on the topic?

If it is the explanation, it would probably be RAM related. As you say, your drives can easily handle the load. But perhaps ZFS doesn't think you have enough memory to buffer up past 5 seconds.

There are two tunables you might try.

vfs.zfs.no_write_throttle = 1 (instead of 0). Might make the problem disappear (or get more abrupt) if it is the throttle causing the stalls.
vfs.zfs.write_limit_override = 34000000 Not sure if this is recommended or not, but it might force syncs before the throttle gets involved.

I will keep it in mind next time I am physically close to it, but probably I will just try 9.2.x first.

I also wanted to suggest turning off AIO writes as another test. Letting the mid-point throttling take effect might prevent it from getting to the big slow down.

AIO is off, since AIO forces everything async and I want writes to stay as they were meant to be.

The final solution to the problem of the dips in the transfer rate from/to the HP N40L was found by updating to NAS4free 9.3.x, no other tweaks needed. As I expected it was something in the implementation of ZFS. Now it saturates the 100 Mbps links perfectly and constantly.