Slow transfer speed

[Dest]

Hi,

I’m experiencing slow performance with my newly build Nas4Free setup. I’ve been looking online to find a solution but nothing has resolved my problem so far. I’m hoping to find a solution through this forum post.
I’m a windows user, I have no experience with FreeBSD.

I’m virtualizing Nas4Free on an VMWare ESXi server (5.1.0.U1) with the following hardware; Supermicro X8DTH-6; Intel Xeon L5618; 24 GB DDRL3-1333 EEC Memory. The Motherboard has an LSI SAS2008 controller on board, which is flashed in IT mode(latest firmware). I’m passing the sas controller though to the VM with Intel VT-d. There are 4 WD RED 3TB disk’s connected to the sas controller. There are no other VM’s active during testing. The systems are connected through a gigabit network (no jumbo frames support on switch).
I’m using Nas4Free x64 9.1.0.1.804 in Full mode, with 8 GB RAM.

I did the following steps after installing Nas4Free;
• Edit /boot/loader.conf to resolve interrupt problems with sas controller, viewtopic.php?f=15&t=1684
• Edit /boot/loader.conf to enable ZFS prefetch, change vfs.zfs.prefetch_disable="1" to vfs.zfs.prefetch_disable="0"
• Creating a vdev(Raidz1) with the 4 WD RED’s (support 4K sectors)
• Creating a Pool with the vdev
• Creating a dataset (Compression: lzjb; Dedup: Off; Sync: standard; Access Time: On; Canmount; Extended attributes)
• Remove nop from pool (performance) viewtopic.php?f=59&t=1494
• Activating CIFS/SMB service
• Activating FTP Service

I’ve tested the CIFS/SMB speed with NAS performance tester 1.4 (http://www.808.dk/?nastester). I’ve tested with a 4 GB file.
I’ve tested with the following CIFS/SMB settings:
• SMB2; Local Master Browser: yes; Time server: no; Large read/write: yes; Use sendfile: no; Asynchronous I/O (AIO): no
Average write: 64,56 MB/sec Average read: 63,58 MB/sec
• SMB2; Local Master Browser: yes; Time server: no; Large read/write: yes; Use sendfile: no; Asynchronous I/O (AIO): yes
Average write: 58,65 MB/sec Average read: 68,11 MB/sec
• NT1; Local Master Browser: yes; Time server: no; Large read/write: yes; Use sendfile: no; Asynchronous I/O (AIO): no
Average write: 66,79 MB/sec Average read: 65,38 MB/sec
• NT1; Local Master Browser: yes; Time server: no; Large read/write: yes; Use sendfile: no; Asynchronous I/O (AIO): yes
Average write: 57,97 MB/sec Average read: 61,14 MB/sec
• NT1; Local Master Browser: yes; Time server: no; Large read/write: yes; Use sendfile: yes; Asynchronous I/O (AIO): yes
Average write: 68,29 MB/sec Average read: 61,84 MB/sec

ftp speed is a litle bit faster. Transfer speed is also not stable.
When I go to Stats|Graph|System load during file transfers the LAN en CPU graph goes wildly up and down (Lan between 50 MB/s to 85 MB/s; CPU between 5 % to 64 %). The HDD activity led on the HDD enclosure light up for only a second at a time. It takes about 3 seconds between HDD activity. It looks like reading and writing is done in burst.

I found this great topic on Nas4Free performance, http://n4f.siftusystems.com/index.php/2 ... leshooting. And I’ve been using this site to troubleshoot.

Iperf: 937 Mbits/sec

dd:
Write: 20971520000 bytes transferred in 25.692166 secs (816261273 bytes/sec)
Read: 20971520000 bytes transferred in 11.370161 secs (1844434735 bytes/sec)
This would indicate that the sequential disk speed is 778,45 MB/s during write and 1758,99 MB/s during read operations. I find these numbers very high. I was more expecting something around 300 MB/s.(single disk speed is around 110 MB/s) During the write test the HDD activity led’s where constantly blinking.

Faulty Drive: scrub ok en no smart errors

CPU: I have only encountered a peak maximum load of 69% so far

Power Daemon is disabled

Card is connected through PCI-express 8x (Intel VT-d)

Ashift 12

File copy to my old nas (windows 2008 r2 with hardware raid 5) goes at about 112 MB/s from my pc. So I don’t suspect my pc.

I’ve realize it’s a lot to read through, but I wanted to show that I did some research/testing on my own bit wasn’t able to resolve the problem.

If anyone has an idea how to resolve this performance issue (+ stability of the transfer speed). Or has some configurations changes I could test. It would be a big help.
I’m expecting a steady file transfer from around 100MB/s with this setup.

[b0ssman]

just to rule out other problems.

have you tried a native nas4free setup?

[Dest]

I haven’t tried a native setup yet. I didn’t want to mess up my ESXi server as there are other VM’s running on it.

I’m thinking on doing the following steps to make sure it isn’t related to virtualization:
• Destroy pool and vdev, creating a new vdev en pool with 1 disk an testing a file transfer (this way I have a performance baseline)
• Install a Windows 2008 R2 server VM, connect the sas card to this VM and test a file transfer with 1 disk
• Create a bootable usb stick with Nas4Free embedded and run it native on the server

@b0ssman
I see you are also using a sas controller with the LSI sas2008 controller (IBM ServeRAID M1015) and the same HDD’s (WD30EFRX) as I’m using. What are you transfer speeds trough gigabit Ethernet? This way I have an indication what transfer speed to expect (is 100 MB/s possible?).

[b0ssman]

i am using encryption and i am getting about 90mb/sec.

you can install nas4free on a usb stick and boot from it.
this should leave your vmware setup alone and you can test it.

[Dest]

I’ve redone the CIFS/SMB speed test with NAS performance tester 1.4.
• Nas4Free in VM, 1 disk: Average write: 63,78 MB/sec Average read: 75,79 MB/sec
• W2008r2 in VM, single disk: Average write: 116,69 MB/sec Average read: 91,67 MB/sec
• Native Nas4Free, 4 disk in raidz1: Average write: 90,67 MB/sec Average read: 91,35 MB/sec

A native Nas4free setup is 20 MB/sec to 30 MB/sec faster in my case. Cpu load during transfer is around 3 % lower than in a VM. A windows VM is still the fastest setup.

[b0ssman]

i have found that samba never reaches the transfer speed of a native windows setup for cifs.

see what speed you can get with ftp.

but as to what you can do about your vm setup i can not help you.