ZFS as ESXi datastore

[JamesP]

I have ESXi running on bare metal and one of the VM's is a NAS4FREE VM that is exporting a directory (via NFS) that I'm using as a datastore directory for other VM's on the same hardware. i.e., ESXi boots first from an SSD and then launches the NAS4FREE VM which uses a datastore on the same SSD. Other VM's are then launched one by one and these other VM's use the datastore hosted on NAS4FREE. Just to be clear, the 'Other VM's' do not know anything about NAS4FREE and they think they are simply accessing a locally attached drive.

NAS4FREE is performing really well on my network and Windows hosts are able to read and write at 60-100MBytes/s over CIFS.

My virtual machines running on ESXi are not so lucky. They can read at 100MB/s but writes are around 6MB/s. Is there anything I can do to speed this up? Or is there just too much overhead going from the VM through ESXi through NFS to NAS4FREE?

Thanks,
JP

[PaganGod]

This is a known issue with NFS datastores hosted on NAS4Free running on ZFS, and ESX/ESXi. Basically it boils down to VMware wanting all writes to be synchronous, which really slows down performance, typically to the rates you are reporting. There are workarounds (switch to iSCSI, install mirrored SLC SSDs for ZIL) as well as ways to improve performance at expense of risk (disable sync on ZFS) which you can find with a Google search. This also happens on FreeNAS, and I think the most useful articles I found on the topic were on the FreeNAS forums. Here is one:

http://forums.freenas.org/showthread.ph ... se-is-fast

[JamesP]

This is a known issue with NFS datastores hosted on NAS4Free running on ZFS, and ESX/ESXi.

Thanks for this! I've tried moving the datastore to iSCSI and while it's much better, it still does not seem very consistent. Using dd, sometimes I get 50MB and sometimes 150MB. I'll look into adding a ZIL drive if I can find a reasonably priced one

James

[PaganGod]

If you are talking about an SSD for ZIL, I believe you will only want to consider SLC (Single Level Cell) because of how intensive the writes are. You don't need it to be particularly large, so the trade off of small size for SLC technology is worth it. MLC will fail under ZIL use much more quickly.

[JamesP]

Well, I went for the new Intel DC S3700 SSD, but I haven't installed it yet Not sure if it's SLC or MLC, but it appears to be very promising for a ZIL drive. We'll see!

JP

[PaganGod]

http://ark.intel.com/products/71915/Int ... s-25nm-MLC
It's MLC.

I'd recommend you Google ssd zil mlc slc and read any of the first page of hits. MLC is fine for L2ARC which is a read cache function since MLC are sensitive to early failure from excessive write use and L2ARC gets mostly reads. Of course if you have a lot of RAM in the box you'd only benefit from L2ARC if you have a large data set, larger than will fit in the RAM cache. For ZIL, which is basically all writes, you want SLC which can handle the repeated writes to often the same cells (bits).

One thing I am not sure of is whether the writes are also buffered to RAM, and the ZIL is only used to replay them in the event the server crashes or loses power. I believe that is the case, however you are always writing your ZIL somewhere and the benefit of a dedicated SLC SSD for ZIL is that it is going to different disks from the actual ZFS pool holding the data, and also of course SSD is so much faster than rotating disks. It basically allows writes to commit very fast.

If I'm wrong on any of this I'd appreciate some of the "pros" here letting me know, but I read extensively on many of these topics before doing my first (or second) build.

[JamesP]

http://ark.intel.com/products/71915/Int ... s-25nm-MLC
It's MLC.

I read in a recent post that the MLC/SLC difference isn't as critical with the new drives... Unfortunately, I don't have a link to that post any more... I'm hoping I can get 100MB/sec performance in which case I'm limited by GbE and not the MLC.

One thing I am not sure of is whether the writes are also buffered to RAM, and the ZIL is only used to replay them in the event the server crashes or loses power. I believe that is the case, however you are always writing your ZIL somewhere and the benefit of a dedicated SLC SSD for ZIL is that it is going to different disks from the actual ZFS pool holding the data, and also of course SSD is so much faster than rotating disks. It basically allows writes to commit very fast.

My understanding is the same as yours in that that the ZIL is a write-only device, unless something 'bad' happens, such as a power failure. Only then does the system actually read back from the ZIL.

If I'm wrong on any of this I'd appreciate some of the "pros" here letting me know, but I read extensively on many of these topics before doing my first (or second) build.

I'd love to hear some 'pro' comments too!

James

[PaganGod]

I read in a recent post that the MLC/SLC difference isn't as critical with the new drives... Unfortunately, I don't have a link to that post any more... I'm hoping I can get 100MB/sec performance in which case I'm limited by GbE and not the MLC.

If you find the article please link here for future searcher's benefit! Even a 3 year old SATA-II MLC drive should easily be able to sustain enough throughput to saturate a single Gigabit Ethernet link, at least for typical I/O mix. Definitely for sequential reads and writes. Actually, since the ZIL is effectively just sequential log data writes, the performance won't be a concern.

Here's a fairly current article about SLC/MCL/TLC in general from late 2012. I'd not consider what they say about the duration of the different technologies in years and more look at the relative differences. Also, your drive might well be eMLC (Enterprise MLC) which could have a longer endurance (lifetime) than consumer MLC. Still, I think the SLC recommendation stands for anything mission critical, production or high-volume.

http://www.anandtech.com/show/6459/sams ... f-tlc-nand

[cancerman]

Read the review on anandtech for the S3700. It's rated for 10x capacity writes every day for 5 years. It should be fine. Under provision it for even more durability.