Poor/Inconsistent iSCSI write performance

[coachmark2]

Hello,

First time poster, and quite impressed with N4F. Hoping someone can point me in the right direction on this. My box specs:

x64 embedded
CPU: Pentium G620
RAM: 16GB ECC 1600
Mobo: SuperMicro M-ATX LGA1155 (Dual Intel NICs, only one in use)
Case, PSU, etc
Drives: 2x3TB WD RED as a mirrored vdev all alone in a zpool

I have created an iSCSI target of 1.5TB and have successfully attached my media server, a Windows Server 2012 R2 VM, to it. However, my performance is not good. Reads for the most part max out around 100MB/s, but they're pretty consistent. Writes, however, are pretty terrible. They start at 110-100MB/s and then degrade over the course of a second or two down to 0MB/s. Then about five seconds later, they'll gradually rise to 30-50MB/s and then crash back down to zero. This continues until the transfer is completed or cancelled.

What I've done so far:
-It's not network related. iperf between the two shows 900+mbits/sec even on an extended test of 120 seconds.
-Using SMB for reads and writes results in >100MB/s consistent
-Tried modifying some iSCSI parameters like MaxOutstandingR2T
-Tried setting target to 4096 logical block length
-CPU is between 15-40% during a large write operation, RAM is at 6GB/16GB
-CPU on my server is at 5% peak during transfer

What would be the next step in troubleshooting?

[b0ssman]

iscsi uses synchronous writes.

try adding a zil.

[coachmark2]

What would a suitable ZIL drive be for my array plus an eye towards future expansion? And are you reasonably confident that this will fix the issue? Would rather not go drop $100 on an SSD that doesn't fix the problem.

[b0ssman]

see

http://forums.freenas.org/index.php?thr ... nas.13633/

[00Roush]

First try installing zfs kernel tune... viewtopic.php?f=71&t=1278
Then restart nas4free and try tests again.

Another option is to create iscsi volume with block size of 4k or 64k. This is done via command line command like this...

zfs create -V 50G -b 64k media/scsidisk

This will create a 50 GB zvol named scsidisk with a block size of 64k. Default is 8k however typically windows does not typically write in 8k blocks. Instead it uses 4k and 64k for most operations.

Test this out with 4k as well to see if either helps. In my limited testing I found zfs kernel tune, 64k block size, and 64k NTFS allocation unit size provided good performance for file transfers.

Let me know if it helps.
00Roush

[STAMSTER]

FreeBSD's ZFS implementation is fully functional; the only missing features are kernel CIFS server and iSCSI, but at least the latter can be added using externally available packages.[71] Samba can be used to provide a userspace CIFS server.
http://en.wikipedia.org/wiki/ZFS

If your HDD's have 4 KB sectors, try to format them in that way (i.e. not to emulate 512b).

Good news is your memory seems to be sufficient for that workload... others might say - add more RAM...

@b0ssman Please correct me if I'm wrong, but iSCSI writes are async?! Or perhaps N4F works in sync mode by default!?

In general, ZFS performance is far from the best when you initiate lots of sync requests without a SLOG device... it's pretty logical - system needs to wait the slowest disk in a pool to confirm it's writes... if your disk is slow, it'll be the bottleneck.

I'm also considering a SLOG device (MLC based SSD with 240 GB min.)...

[b0ssman]

yes sorry iscsi is async. got it mixed up in my head with nfs.