sync = disabled vs SSD ZIL: too much difference

[lucapsg]

At home I have an All-in-one server based on Supermicro X10SL7-F MB, practically something very similar to this: https://b3n.org/freenas-9-3-on-vmware-esxi-6-0-guide/
I know that the performance of the combination ESXi + NFS + ZFS can be disastrous.
So to improve them I added to my ZFS pool (2x2TB SATA HDD in mirror) a Samsung SSD 840 EVO over-provisioned at 8GB (https://www.thomas-krenn.com/en/wiki/SS ... ing_hdparm) as SLOG/ZIL.
The result was very disappointed.
CristalDiskMark in Windows 2008 R2 x64 VM reports:

1.png

It seemed to have read that by doing so you could get "comparable" performance with sync = disabled, right?
Later I tried with a second SSD, the first mirrored, then striped, and also increased the number of NFS server threads, from 4 to 32 (https://b3n.org/freenas-vs-omnios-napp-it/) but without significant improvements.
So I again removed the SSD from the pool, I set sync = disabled and I redid the test:

2.png

Does it seem possible?
In the end I created a mirror pool with only the two SSDs and I was literally speechless:

3.png

The write performance are absurd!
I do not understand!!!
Note that all HDDs are connected to the internal controller LSI 2308, while the two SSDs are connected to two SATA3 ports of the integrated Intel Lynx Point controller, both passed through to N4F via VT-d.

I know that the SSD is not an Intel S3500 but it is possible that the difference is due to the kind of SSD?
Where can I compare the performance obtained from other, perhaps with configurations similar to mine?
Any other suggestions?

[Onichan]

The SSD makes a huge difference, see https://b3n.org/ssd-zfs-zil-slog-benchm ... omparison/

Though your tests are so bad I would think the cheapest SSD would be better than what you have. I would probably plug the SSD into my desktop and run CrystalDiskMark directly on it to compare. Could be some hardware issue with your NAS.

Here is my tested from a while back using a 4 Disk Intel 335 180GB ZFS RAID10 with lz4 compression with sync enabled to my Proxmox host over a direct fiber connection using NFS.

Code: Select all

VM with sata disk in raw mode:
-----------------------------------------------------------------------
CrystalDiskMark 5.1.0 x64 (C) 2007-2015 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

   Sequential Read (Q= 32,T= 1) :   172.057 MB/s
  Sequential Write (Q= 32,T= 1) :   162.124 MB/s
  Random Read 4KiB (Q= 32,T= 1) :     8.124 MB/s [  1983.4 IOPS]
 Random Write 4KiB (Q= 32,T= 1) :     8.346 MB/s [  2037.6 IOPS]
         Sequential Read (T= 1) :   199.987 MB/s
        Sequential Write (T= 1) :   157.879 MB/s
   Random Read 4KiB (Q= 1,T= 1) :     5.546 MB/s [  1354.0 IOPS]
  Random Write 4KiB (Q= 1,T= 1) :     4.438 MB/s [  1083.5 IOPS]

  Test : 1024 MiB [G: 0.2% (0.1/32.0 GiB)] (x1)  [Interval=5 sec]
  Date : 2016/01/04 21:54:29
    OS : Windows Server 2012 R2 Server Standard (full installation) [6.3 Build 9600] (x64)


Other test using virtio disk driver in raw mode
-----------------------------------------------------------------------
CrystalDiskMark 5.1.0 x64 (C) 2007-2015 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

   Sequential Read (Q= 32,T= 1) :   305.341 MB/s
  Sequential Write (Q= 32,T= 1) :   269.511 MB/s
  Random Read 4KiB (Q= 32,T= 1) :    28.336 MB/s [  6918.0 IOPS]
 Random Write 4KiB (Q= 32,T= 1) :    13.240 MB/s [  3232.4 IOPS]
         Sequential Read (T= 1) :   311.799 MB/s
        Sequential Write (T= 1) :   211.079 MB/s
   Random Read 4KiB (Q= 1,T= 1) :     7.137 MB/s [  1742.4 IOPS]
  Random Write 4KiB (Q= 1,T= 1) :     5.652 MB/s [  1379.9 IOPS]

  Test : 1024 MiB [E: 0.2% (0.1/34.0 GiB)] (x1)  [Interval=5 sec]
  Date : 2016/01/04 21:49:30
    OS : Windows Server 2012 R2 Server Standard (full installation) [6.3 Build 9600] (x64)