ZFS Benchmarks on HP Microserver. N4F vs N4F VMWare vs OI

[ahnooie]

I've been considering a switch to N4F (currently using OI), about the only thing holding me back is seeing if I can get Crashplan on it which I think will be easier once FreeBSD 9.1 is out of RC and the pkg_add stuff works. In the meantime, I did some performance testing of CIFS with several configurations:

HP Microserver N40L
ZFS Pool 3x250GB Seagate 7200 RPM drives.
4GB Memory
Sync: standard
Atime: off
Compression: on
NF4: large read/write: enabled


[*] If using ESXi drives mapped using RDM (-r switch) except one test where I tried VMDKs (indicated). I didn't use -z because N4F thought the drives were 0 bytes, I can use -z in OI but to do apples to apples as much as possible I mapped them using -r in OI. I'm not sure whether or not it makes a performance difference, I'm guessing it does not.
[*] Guests were given the full 4GB mem.
[*]I installed the gui version of OI.
[*] Used VMXNET3 in N4F, used the en1000g in OI since I couldn't figure out how to enable the VMXNET3 driver.
[*] Used OI 151a5-live, N4F 9.1.0.1.457

Tested from Windows 8 machine using CrystalDiskMark, each test repeated 5 times and averaged, test size 50MB.

All numbers in MB/s

Sequential Read
ESXi+OpenIndiana: 35
ESXi+NAS4Free: 62
NAS4Free (usb embedded): 100
ESXi+NAS4Free using VMDKs: 47

512K Read
ESXi+OpenIndiana: 35
ESXi+NAS4Free: 62
NAS4Free (usb embedded): 100.5
ESXi+NAS4Free using VMDKs: 46

4K Read
ESXi+OpenIndiana: 4.7
ESXi+NAS4Free: 5.1
NAS4Free (usb embedded): 9.1
ESXi+NAS4Free using VMDKs: 4.0

4K Read Queue Depth 32
ESXi+OpenIndiana: 62
ESXi+NAS4Free: 5.8
NAS4Free (usb embedded): 9.43
ESXi+NAS4Free using VMDKs: 4.3

---

Sequential Write
ESXi+OpenIndiana: 83
ESXi+NAS4Free: 34
NAS4Free (usb embedded): 99
ESXi+NAS4Free using VMDKs: 23

512K Write
ESXi+OpenIndiana: 77
ESXi+NAS4Free: 31
NAS4Free (usb embedded): 94
ESXi+NAS4Free using VMDKs: 27

4K Write
ESXi+OpenIndiana: 4.2
ESXi+NAS4Free: 2.1
NAS4Free (usb embedded): 8.0
ESXi+NAS4Free using VMDKs: 3.7

4K Write Queue Depth 32
ESXi+OpenIndiana: 22
ESXi+NAS4Free: 2.4
NAS4Free (usb embedded): 8.3
ESXi+NAS4Free using VMDKs: 3.9

---

So virtualization has a huge hit, the one USB embedded test I did probably maxed out my ethernet. I like running everything virtualized, but I may have to reconsider and just run it without VMWare... I have two Microservers so one can be the SAN and NFS share to ESXi.
OI is better at write speed while N4F is better at reads.
OI performs better at high queue depth, so for databases and vmware storage it would probably perform better... although the tests may come out completely different using NFS.

[ChriZathens]

Excuse me for the offtopic... I would like to ask you a few questions about N40L since I am considering to buy one.. (I thought that posting here is better than sending you a PM)
OK, here we go:

1. I read in specs that it has "2 DIMM slots Unbuffered ECC memory". Have you tried non ECC memory? Does it work?
2. The internal disk is a 250GB HDD. I thought about adding 4x2TB HDDs + 1 small 2.5" for the OS and then use an external DVD to install N4F full. Can this be done out of the box? I read that people who want to use a 5disk array needed to flash a hacked bios, but it is not clear to me if it is needed even if you want to use only the 4 disks in the array.
3. If I use 4 HDDs, an internal dvd and an internal usb stick, will it allow full installation on the usb stick? My best guess is yes, but branded computers often have strange limitations (I know it is not recommended to install full on usb, but can it be done?)
4. Can we perhaps use the esata port to add an additional disk ?

I am sure that you perhaps cannot answer all my questions, but any response would be much appreciated!
Cheers!

[ahnooie]

Excuse me for the offtopic... I would like to ask you a few questions about N40L since I am considering to buy one.. (I thought that posting here is better than sending you a PM)
OK, here we go:

• I read in specs that it has "2 DIMM slots Unbuffered ECC memory". Have you tried non ECC memory? Does it work?
• The internal disk is a 250GB HDD. I thought about adding 4x2TB HDDs + 1 small 2.5" for the OS and then use an external DVD to install N4F full. Can this be done out of the box? I read that people who want to use a 5disk array needed to flash a hacked bios, but it is not clear to me if it is needed even if you want to use only the 4 disks in the array.
• If I use 4 HDDs, an internal dvd and an internal usb stick, will it allow full installation on the usb stick? My best guess is yes, but branded computers often have strange limitations (I know it is not recommended to install full on usb, but can it be done?)
• Can we perhaps use the esata port to add an additional disk ?

I am sure that you perhaps cannot answer all my questions, but any response would be much appreciated!
Cheers!

[*] I have not tried non-ECC, but I'm not sure why you'd be concerned enough about integrity to run ZFS if you weren't also running ECC memory.
[*] I know it can be done, I don't know if you need to hack the BIOS. I just run 3 drives + 1 SSD (I put a L2ARC vmdk there) and usually boot off the USB on the three HP Microservers I have.
[*] I think I saw the option to do a full installation of N4F on a USB stick (full embedded install without swap), you don't want to put a swap on it or you'll wear it out. Also, you don't need a DVD drive to install it. Just use the N4F USB boot image, boot from USB and install to another USB. The microserver has an internal USB port so you can install N4F on that. I don't have a DVD drive on mine.
[*] I have not done so, but I've heard of others that have (they may have had to do a Bios hack) to get a total of 6 drives.