Optimization of ZFS with metadata on SSD possible

[himbrr]

Hi All,

since two years I'm using nas4free (9.1.0.1 rev 636) with a SSD for the OS and 5x WD20EFRX for data as a single raidZ1 pool.
I mentioned that the write speed is very slow (synchronous write with dd reaches only 54MB/s). Also browsing folders with many files is very slow.

So my thought is to outsource the metadata of the filesystem to the SSD. This should optimize the lookup process of files and folders. The read speed of the data should remain the same. But you often have programs which check if files were changed, added or removed.

Does ZFS support such optimizations?

best regards,
himbrr

[b0ssman]

have you installed zfskerntune and made sure that prefetch is enabled?
without that browsing directories was painfully slow on my machine.

zfs supports two methods of using a ssd to improve performance.
l2arc for readign, and zil for synchronous writes.

[himbrr]

no I haven't installed zfskerntune yet... should try this

How could I use l2arc and zil?

Is there any documentation on ZFS | Datasets and Volumes? Currently my data is directly stored on an ZFS Pool without dedup. etc.

[b0ssman]

how much ram does your pc have?

[himbrr]

The system has 8 Gb RAM... But I've read in the wiki that dedup is painful. And this is not effective for media data...

I see that I could add a device as cache manually via the webui, but I can't select the ssd.
Is it possible to use the ssd for the os and for caching?

[b0ssman]

have you got a full install and do you really need it?

the recommended install is the embedded on a usb stick. this would free up your ssd.

[00Roush]

This should help with slow folder browsing... http://www.forums.nas4free.org/viewtopi ... =21&t=5401

Your write speeds seem slow for the disks. Your disks look to be 4k sector disks... do you know if you formatted your disks with advanced format option? You can check with zdb | grep ashift command.

What is dd read speeds? What hardware you running?

As was mentioned L2ARC and ZIL can help improve performance but my opinion is that is only the case if the hardware is already being strained.

For me I have found the best performance with just a couple of changes from defaults. Samba buffers (send and receive) 0, Samba DOS attribute unchecked, Samba AIO checked (can be unstable), Samba Max Protocol SMB2 (default in latest versions) and ZFS kernel tune installed.

00Roush

[himbrr]

have you got a full install and do you really need it?

Yes I have a full install running... I will migrate this to the embedded version at the weekend.

What is dd read speeds?

Code: Select all

/testfile is located at the ssd
/mnt/Management/testfile is locatet at the raidZ pool

---- Write from /dev/random ---
 #  dd if=/dev/random of=/mnt/Management/testfile bs=4k count=250k
256000+0 records in
256000+0 records out
1048576000 bytes transferred in 19.201713 secs (54608461 bytes/sec)

#  dd if=/dev/random of=/testfile bs=4k count=250k
256000+0 records in
256000+0 records out
1048576000 bytes transferred in 18.376837 secs (57059656 bytes/sec)

---- Read to /dev/null ---
#  dd if=/mnt/Management/testfile of=/dev/null bs=4k count=250k
256000+0 records in
256000+0 records out
1048576000 bytes transferred in 1.650008 secs (635497539 bytes/sec)

#  dd if=/testfile of=/dev/null bs=4k count=250k
256000+0 records in
256000+0 records out
1048576000 bytes transferred in 1.123679 secs (933163361 bytes/sec)

---- Write from /dev/null---
#  dd if=/dev/null of=/mnt/Management/testfile bs=4k count=250k
0+0 records in
0+0 records out
0 bytes transferred in 0.000118 secs (0 bytes/sec)

#  dd if=/dev/null of=/testfile bs=4k count=250k
0+0 records in
0+0 records out
0 bytes transferred in 0.000073 secs (0 bytes/sec)

---- Read the zeros to /dev/null ---
#  dd if=/mnt/Management/testfile of=/dev/null bs=4k count=250k
0+0 records in
0+0 records out
0 bytes transferred in 0.000160 secs (0 bytes/sec)

#  dd if=/dev/null of=/testfile bs=4k count=250k
0+0 records in
0+0 records out
0 bytes transferred in 0.000102 secs (0 bytes/sec)

Yes the SSD is slower as the Raid...
SSD r/w specs from the vendor: 285MB/s | 275MB/s
HDD r/w specs from the vendor: 145 MB/s | 145 MB/s
There must be something wrong...

Your disks look to be 4k sector disks... do you know if you formatted your disks with advanced format option?

I haven't enabled the advanced format option on the pool.

Code: Select all

#  zdb | grep ashif
ashift: 9

What hardware you running?

Platform x64-full on Intel(R) Celeron(R) CPU G530 @ 2.40GHz
System ASUSTeK COMPUTER INC. P8H77-I
System bios American Megatrends Inc. version: 0809 08/01/2012
8GB memory
The ZFS Raid is running as Z1 Software Raid with 5x WD20EFRX and the OS is running on a OCZ-VERTEX2 (60 GB) SSD.
For the embedded installation I will use a USB3.0 Patriot Tab 8GB.

For me I have found the best performance with just a couple of changes from defaults. Samba buffers (send and receive) 0, Samba DOS attribute unchecked, Samba AIO checked (can be unstable), Samba Max Protocol SMB2 (default in latest versions) and ZFS kernel tune installed.

My current setup of SMB is:
Samba buffers (send and receive) 84240,
Samba DOS attribute checked,
Samba AIO unchecked ,
Samba Max Protocol SMB2 (default in latest versions)
Large read/write checked
Sendfile unchecked
EA support unchecked

ZFS kernel tune is installed, but I don't know if it's enabled... After the selection of the options and the restart, anything of the extension has been reset.

I think there is a lot of improvement possible

EDIT: I have added the result of the benchmark over smb, after i have adjusted the options for the DOS attribute and AIO.

[crowi]

I haven't enabled the advanced format option on the pool.

That causes a lot delay in processing, you need to enable it if you have 4k drives, otherwise the disk performance becomes very bad

ZFS kernel tune is installed, but I don't know if it's enabled... After the selection of the options and the restart, anything of the extension has been reset.

That is normal, but the values stay in the system

[himbrr]

That causes a lot delay in processing, you need to enable it if you have 4k drives, otherwise the disk performance becomes very bad

Okay.. this are very bad news after running a system for 2 years with now 6TB data...

[b0ssman]

you dont use /dev/random for test you use /dev/zero

/dev/random is very cpu intensive. thats where the strange values come from.

please post the values from your loader.conf

[himbrr]

you dont use /dev/random for test you use /dev/zero

Code: Select all

#  dd if=/dev/zero of=/mnt/Management/testfile bs=4k count=250k
256000+0 records in
256000+0 records out
1048576000 bytes transferred in 3.675862 secs (285259887 bytes/sec)
#  dd if=/dev/zero of=/testfile bs=4k count=250k
256000+0 records in
256000+0 records out
1048576000 bytes transferred in 3.854861 secs (272013957 bytes/sec)

please post the values from your loader.conf

Where is the loarder.conf located?

[b0ssman]

i think there i a gui for it under System|Advanced

or /cf/boot
or /boot

[ChriZathens]

And also, as 00Roush mentioned, uncheck Enable store DOS attributes
I have seen that it slows down folder browsing significantly when checked

[himbrr]

please post the values from your loader.conf

Code: Select all

#  cat /boot/loader.conf
kernel="kernel"
bootfile="kernel"
kernel_options=""
kern.hz="100"
hw.est.msr_info="0"
hw.hptrr.attach_generic="0"
kern.maxfiles="65536"
kern.maxfilesperproc="50000"
kern.cam.boot_delay="8000"
autoboot_delay="5"
isboot_load="YES"
zfs_load="YES"
# ZFS kernel tune
vm.kmem_size="4608M"
vfs.zfs.arc_min="3072M"
vfs.zfs.arc_max="3072M"
vfs.zfs.prefetch_disable="0"
vfs.zfs.txg.timeout="5"
vfs.zfs.vdev.max_pending="10"
vfs.zfs.vdev.min_pending="4"
vfs.zfs.write_limit_override="0"
vfs.zfs.no_write_throttle="0"

[ChriZathens]

I also have 8GB Ram, but ZFS kernel tune settings are different in my case:

# ZFS kernel tune
vm.kmem_size="6656M"
vfs.zfs.arc_min="5120M"
vfs.zfs.arc_max="5120M"
vfs.zfs.prefetch_disable="0"
vfs.zfs.txg.timeout="5"
vfs.zfs.vdev.max_pending="10"
vfs.zfs.vdev.min_pending="4"
vfs.zfs.write_limit_override="0"
vfs.zfs.no_write_throttle="0"

Can you change your settings (or run zfs kernel tune ) and check again if things get better?

[crowi]

But the main issue is the wrong (not 4k) alignment, which slows down the disks by 25% or more.
In this case the tuning parameters have less influence in speeding up the system than a correct aligment/format.

[himbrr]

But the main issue is the wrong (not 4k) alignment, which slows down the disks by 25% or more.
In this case the tuning parameters have less influence in speeding up the system than a correct aligment/format.

Yes this should be result a lot of performance, but after deleting the unnecessary data, I have still 4,6TB data at the RAID.
I couldn't store this at another volume...

Can you change your settings (or run zfs kernel tune ) and check again if things get better?

I will do this, after the re-installation of the nas4free system. Currently I have the issue, that I can't run any jails...

[00Roush]

DD results show your write speeds on your pool are about 272 MB/sec. Alignment might not be correct but write speeds look to be more than enough to saturate gigabit. Correcting it will probably help local disk performance but based on your crystaldiskmark tests after changes show performance looks pretty good.

Are things working better than when you first posted?

[himbrr]

Are things working better than when you first posted?

Yes of course! I think the real slowness came from the wrong SMB settings.
I will post the benchmark again, after I have reinstalled nas4free on the usb stick and added the SSD as cache for the raid.

[substr]

I would set your min_pending =2 and max_pending=6.

The higher values are too optimistic for budget drives.

[00Roush]

Are things working better than when you first posted?

Yes of course! I think the real slowness came from the wrong SMB settings.
I will post the benchmark again, after I have reinstalled nas4free on the usb stick and added the SSD as cache for the raid.

Well that is good to hear. Let us know how it goes.

Cheers,
00Roush

[himbrr]

Hello,

I have migrated my nas4free on ssd to a usb stick.
After I have formated the ssd as ZFS Pool Device I tried to add it to the ZFS pool as cache.
But it didn't showed up in the devicelist.

Do you have any ideas, how I could add the ssd as cache device to the pool?

EDIT: Found it! Had to do this manually via commandline...

loader.conf options after ZFS kernel tune (8GB RAM)

Code: Select all

# ZFS kernel tune
vm.kmem_size="6656M"
vfs.zfs.arc_min="5120M"
vfs.zfs.arc_max="5120M"
vfs.zfs.prefetch_disable="0"
vfs.zfs.txg.timeout="5"
vfs.zfs.vdev.max_pending="10"
vfs.zfs.vdev.min_pending="4"
vfs.zfs.write_limit_override="0"
vfs.zfs.no_write_throttle="0"

EDIT2: Benchmarks
I have added the Benchmark over SMB as attachment. This was done before adding a log partition to the pool.
It's now a little bit slower 0,1 MB/s on writing and has now 600 IOps lesser.

Here is the dd Benchmark:

Code: Select all

nas4free: Management # dd if=/dev/zero of=testdata bs=1024000 count=10240
10240+0 records in
10240+0 records out
10485760000 bytes transferred in 36.071499 secs (290693770 bytes/sec) --> ~ 290,69 MB/s write speed

nas4free: Management # dd if=testdata of=/dev/zero bs=1024000
10240+0 records in
10240+0 records out
10485760000 bytes transferred in 33.369388 secs (314232912 bytes/sec) --> ~ 314,23 MB/s read speed

Cheers,
himbrr

[crowi]

Hi HImbrr,
I think the Crystaldiskmark won't show correct values on a network drive, it will be limited by the LAN speed.
Which size has your log device?

Can you post the output of: Disks|ZFS|Pools|Information ?

[b0ssman]

a zil is for synchronous writes. smb is async.

[himbrr]

Hi,

an you post the output of: Disks|ZFS|Pools|Information ?

Code: Select all

pool: Management
 state: ONLINE
  scan: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	Management  ONLINE       0     0     0
	  raidz1-0  ONLINE       0     0     0
	    ada1    ONLINE       0     0     0
	    ada2    ONLINE       0     0     0
	    ada3    ONLINE       0     0     0
	    ada4    ONLINE       0     0     0
	    ada5    ONLINE       0     0     0
	logs
	  ada0p2    ONLINE       0     0     0
	cache
	  ada0p1    ONLINE       0     0     0

errors: No known data errors

Which size has your log device?

The log device has 10GB and the cache 45 GB. This fit on the 60GB SSD (55GB usable).