bad performance after adding mirrored SLOG

[Thomymaster]

Hi

Yesterday i installed my mirrored SSDs (ADATA SX900 64GB) to enably "sync=always" in my ZFS dataset which is used to provide and ESXi datastore.
Before this, i could get approx. 100MB/s with CrystalDiskMark on a VM, but now i am stuck to a maximum of 25MB/s. The command "gstat" shows me the same. The SSDs are connected to the onboard S-ATA2 controller.

I have 16GB of RAM and 4 1.5TB drives in a striped-mirror config running the latest NAS4Free. Further i have enabled these kernel tune values:

vm.kmem_size 14336M
vfs.zfs.arc_max 12288M
vfs.zfs.arc_min 12288M
vfs.zfs.prefetch_disable 0
vfs.zfs.zil_disable 0
vfs.zfs.txg.timeout 5
vfs.zfs.vdev.max_pending 10
vfs.zfs.vdev.min_pending 4
vfs.zfs.write_limit_override 0

What causes this issue?

Best,
Thomas

[substr]

As long as you didn't partition the SSD (in which case it needs to be done very carefully to maintain alignment), then I think the speed you are seeing is normal. This is why expensive solutions like FusionIO exist.

What is the blocksize or recordsize for the VM storage and is compression set?

[Thomymaster]

Hi

Compression is set to LZ4, blocksize of the zVol is 64KB. Blocksize for the VMFS 5 is 1MB which is set by VMWare. Can you help me how to partition my SSD?

[substr]

Try creating a new zvol with -b 16k and test the performance.

Code: Select all

zfs create -b 16k -o compression=lz4 -o sync=always -V 100G tank/testvol

If you haven't partitioned the SSD, then you don't need to. If you have partitioned it, somebody else will have to help make sure it is aligned. I've done it before, but I couldn't explain it.

[Thomymaster]

I tried with your settings but there was no improvement

[substr]

You could try -b 8k, but if that doesn't help, it's just a basic small write throughput limitation of the SSD. When they are used as log devices, they are much slower than people expect.

[Thomymaster]

And why should this be a throughput limitation? My SSD can do at least 21K writes at 4K random which gives 84MB/s and not 25

[substr]

A guess: It can do 21K random 4K writes with caching enabled or with multiple simultaneous write requests. It can do a quarter that many when each write has to be completed immediately before continuing (aka only one write at a time), which is what sync=always does.

There is also the possibility that the guest OS are breaking writes down into smaller blocks.

What is your guest OS, what file system, and what cluster/block size there?