Page 1 of 1
Optimization of ZFS with metadata on SSD possible
Posted: 15 Jul 2014 20:58
by himbrr
Hi All,
since two years I'm using nas4free (9.1.0.1 rev 636) with a SSD for the OS and 5x WD20EFRX for data as a single raidZ1 pool.
I mentioned that the write speed is very slow (synchronous write with dd reaches only 54MB/s). Also browsing folders with many files is very slow.
So my thought is to outsource the metadata of the filesystem to the SSD. This should optimize the lookup process of files and folders. The read speed of the data should remain the same. But you often have programs which check if files were changed, added or removed.
Does ZFS support such optimizations?
best regards,
himbrr
Re: Optimization of ZFS with metadata on SSD possible
Posted: 15 Jul 2014 21:36
by b0ssman
have you installed zfskerntune and made sure that prefetch is enabled?
without that browsing directories was painfully slow on my machine.
zfs supports two methods of using a ssd to improve performance.
l2arc for readign, and zil for synchronous writes.
Re: Optimization of ZFS with metadata on SSD possible
Posted: 15 Jul 2014 21:50
by himbrr
no I haven't installed zfskerntune yet... should try this
How could I use l2arc and zil?
Is there any documentation on ZFS | Datasets and Volumes? Currently my data is directly stored on an ZFS Pool without dedup. etc.
Re: Optimization of ZFS with metadata on SSD possible
Posted: 15 Jul 2014 21:55
by b0ssman
how much ram does your pc have?
Re: Optimization of ZFS with metadata on SSD possible
Posted: 15 Jul 2014 21:57
by himbrr
The system has 8 Gb RAM... But I've read in the wiki that dedup is painful. And this is not effective for media data...
I see that I could add a device as cache manually via the webui, but I can't select the ssd.
Is it possible to use the ssd for the os and for caching?
Re: Optimization of ZFS with metadata on SSD possible
Posted: 16 Jul 2014 07:12
by b0ssman
have you got a full install and do you really need it?
the recommended install is the embedded on a usb stick. this would free up your ssd.
Re: Optimization of ZFS with metadata on SSD possible
Posted: 16 Jul 2014 09:26
by 00Roush
This should help with slow folder browsing...
http://www.forums.nas4free.org/viewtopi ... =21&t=5401
Your write speeds seem slow for the disks. Your disks look to be 4k sector disks... do you know if you formatted your disks with advanced format option? You can check with zdb | grep ashift command.
What is dd read speeds? What hardware you running?
As was mentioned L2ARC and ZIL can help improve performance but my opinion is that is only the case if the hardware is already being strained.
For me I have found the best performance with just a couple of changes from defaults. Samba buffers (send and receive) 0, Samba DOS attribute unchecked, Samba AIO checked (can be unstable), Samba Max Protocol SMB2 (default in latest versions) and ZFS kernel tune installed.
00Roush
Re: Optimization of ZFS with metadata on SSD possible
Posted: 16 Jul 2014 18:35
by himbrr
b0ssman wrote:have you got a full install and do you really need it?
Yes I have a full install running... I will migrate this to the embedded version at the weekend.
00Roush wrote:What is dd read speeds?
Code: Select all
/testfile is located at the ssd
/mnt/Management/testfile is locatet at the raidZ pool
---- Write from /dev/random ---
# dd if=/dev/random of=/mnt/Management/testfile bs=4k count=250k
256000+0 records in
256000+0 records out
1048576000 bytes transferred in 19.201713 secs (54608461 bytes/sec)
# dd if=/dev/random of=/testfile bs=4k count=250k
256000+0 records in
256000+0 records out
1048576000 bytes transferred in 18.376837 secs (57059656 bytes/sec)
---- Read to /dev/null ---
# dd if=/mnt/Management/testfile of=/dev/null bs=4k count=250k
256000+0 records in
256000+0 records out
1048576000 bytes transferred in 1.650008 secs (635497539 bytes/sec)
# dd if=/testfile of=/dev/null bs=4k count=250k
256000+0 records in
256000+0 records out
1048576000 bytes transferred in 1.123679 secs (933163361 bytes/sec)
---- Write from /dev/null---
# dd if=/dev/null of=/mnt/Management/testfile bs=4k count=250k
0+0 records in
0+0 records out
0 bytes transferred in 0.000118 secs (0 bytes/sec)
# dd if=/dev/null of=/testfile bs=4k count=250k
0+0 records in
0+0 records out
0 bytes transferred in 0.000073 secs (0 bytes/sec)
---- Read the zeros to /dev/null ---
# dd if=/mnt/Management/testfile of=/dev/null bs=4k count=250k
0+0 records in
0+0 records out
0 bytes transferred in 0.000160 secs (0 bytes/sec)
# dd if=/dev/null of=/testfile bs=4k count=250k
0+0 records in
0+0 records out
0 bytes transferred in 0.000102 secs (0 bytes/sec)
Yes the SSD is slower as the Raid...
SSD r/w specs from the vendor: 285MB/s | 275MB/s
HDD r/w specs from the vendor: 145 MB/s | 145 MB/s
There must be something wrong...
00Roush wrote:Your disks look to be 4k sector disks... do you know if you formatted your disks with advanced format option?
I haven't enabled the advanced format option on the pool.
00Roush wrote:What hardware you running?
Platform x64-full on Intel(R) Celeron(R) CPU G530 @ 2.40GHz
System ASUSTeK COMPUTER INC. P8H77-I
System bios American Megatrends Inc. version: 0809 08/01/2012
8GB memory
The ZFS Raid is running as Z1 Software Raid with 5x WD20EFRX and the OS is running on a OCZ-VERTEX2 (60 GB) SSD.
For the embedded installation I will use a USB3.0 Patriot Tab 8GB.
00Roush wrote:For me I have found the best performance with just a couple of changes from defaults. Samba buffers (send and receive) 0, Samba DOS attribute unchecked, Samba AIO checked (can be unstable), Samba Max Protocol SMB2 (default in latest versions) and ZFS kernel tune installed.
My current setup of SMB is:
Samba buffers (send and receive)
84240,
Samba DOS attribute
checked,
Samba AIO
unchecked ,
Samba Max Protocol SMB2 (default in latest versions)
Large read/write checked
Sendfile unchecked
EA support unchecked
ZFS kernel tune is installed, but I don't know if it's enabled... After the selection of the options and the restart, anything of the extension has been reset.
I think there is a lot of improvement possible
EDIT: I have added the result of the benchmark over smb, after i have adjusted the options for the DOS attribute and AIO.
Re: Optimization of ZFS with metadata on SSD possible
Posted: 16 Jul 2014 20:40
by crowi
I haven't enabled the advanced format option on the pool.
That causes a lot delay in processing, you need to enable it if you have 4k drives, otherwise the disk performance becomes very bad
ZFS kernel tune is installed, but I don't know if it's enabled... After the selection of the options and the restart, anything of the extension has been reset.
That is normal, but the values stay in the system
Re: Optimization of ZFS with metadata on SSD possible
Posted: 16 Jul 2014 21:51
by himbrr
crowi wrote:That causes a lot delay in processing, you need to enable it if you have 4k drives, otherwise the disk performance becomes very bad
Okay.. this are very bad news after running a system for 2 years with now 6TB data...
Re: Optimization of ZFS with metadata on SSD possible
Posted: 16 Jul 2014 22:10
by b0ssman
you dont use /dev/random for test you use /dev/zero
/dev/random is very cpu intensive. thats where the strange values come from.
please post the values from your loader.conf
Re: Optimization of ZFS with metadata on SSD possible
Posted: 16 Jul 2014 22:29
by himbrr
b0ssman wrote:you dont use /dev/random for test you use /dev/zero
Code: Select all
# dd if=/dev/zero of=/mnt/Management/testfile bs=4k count=250k
256000+0 records in
256000+0 records out
1048576000 bytes transferred in 3.675862 secs (285259887 bytes/sec)
# dd if=/dev/zero of=/testfile bs=4k count=250k
256000+0 records in
256000+0 records out
1048576000 bytes transferred in 3.854861 secs (272013957 bytes/sec)
b0ssman wrote:please post the values from your loader.conf
Where is the loarder.conf located?
Re: Optimization of ZFS with metadata on SSD possible
Posted: 17 Jul 2014 07:14
by b0ssman
i think there i a gui for it under System|Advanced
or /cf/boot
or /boot
Re: Optimization of ZFS with metadata on SSD possible
Posted: 17 Jul 2014 09:36
by ChriZathens
And also, as 00Roush mentioned, uncheck Enable store DOS attributes
I have seen that it slows down folder browsing significantly when checked
Re: Optimization of ZFS with metadata on SSD possible
Posted: 17 Jul 2014 13:23
by himbrr
b0ssman wrote:please post the values from your loader.conf
Code: Select all
# cat /boot/loader.conf
kernel="kernel"
bootfile="kernel"
kernel_options=""
kern.hz="100"
hw.est.msr_info="0"
hw.hptrr.attach_generic="0"
kern.maxfiles="65536"
kern.maxfilesperproc="50000"
kern.cam.boot_delay="8000"
autoboot_delay="5"
isboot_load="YES"
zfs_load="YES"
# ZFS kernel tune
vm.kmem_size="4608M"
vfs.zfs.arc_min="3072M"
vfs.zfs.arc_max="3072M"
vfs.zfs.prefetch_disable="0"
vfs.zfs.txg.timeout="5"
vfs.zfs.vdev.max_pending="10"
vfs.zfs.vdev.min_pending="4"
vfs.zfs.write_limit_override="0"
vfs.zfs.no_write_throttle="0"
Re: Optimization of ZFS with metadata on SSD possible
Posted: 17 Jul 2014 13:48
by ChriZathens
I also have 8GB Ram, but ZFS kernel tune settings are different in my case:
# ZFS kernel tune
vm.kmem_size="6656M"
vfs.zfs.arc_min="5120M"
vfs.zfs.arc_max="5120M"
vfs.zfs.prefetch_disable="0"
vfs.zfs.txg.timeout="5"
vfs.zfs.vdev.max_pending="10"
vfs.zfs.vdev.min_pending="4"
vfs.zfs.write_limit_override="0"
vfs.zfs.no_write_throttle="0"
Can you change your settings (or run zfs kernel tune ) and check again if things get better?
Re: Optimization of ZFS with metadata on SSD possible
Posted: 17 Jul 2014 14:17
by crowi
But the main issue is the wrong (not 4k) alignment, which slows down the disks by 25% or more.
In this case the tuning parameters have less influence in speeding up the system than a correct aligment/format.
Re: Optimization of ZFS with metadata on SSD possible
Posted: 17 Jul 2014 14:29
by himbrr
crowi wrote:But the main issue is the wrong (not 4k) alignment, which slows down the disks by 25% or more.
In this case the tuning parameters have less influence in speeding up the system than a correct aligment/format.
Yes this should be result a lot of performance, but after deleting the unnecessary data, I have still 4,6TB data at the RAID.
I couldn't store this at another volume...
ChriZathens wrote:Can you change your settings (or run zfs kernel tune ) and check again if things get better?
I will do this, after the re-installation of the nas4free system. Currently I have the issue, that I can't run any jails...
Re: Optimization of ZFS with metadata on SSD possible
Posted: 17 Jul 2014 21:38
by 00Roush
DD results show your write speeds on your pool are about 272 MB/sec. Alignment might not be correct but write speeds look to be more than enough to saturate gigabit. Correcting it will probably help local disk performance but based on your crystaldiskmark tests after changes show performance looks pretty good.
Are things working better than when you first posted?
Re: Optimization of ZFS with metadata on SSD possible
Posted: 19 Jul 2014 22:21
by himbrr
00Roush wrote:Are things working better than when you first posted?
Yes of course! I think the real slowness came from the wrong SMB settings.
I will post the benchmark again, after I have reinstalled nas4free on the usb stick and added the SSD as cache for the raid.
Re: Optimization of ZFS with metadata on SSD possible
Posted: 19 Jul 2014 22:33
by substr
I would set your min_pending =2 and max_pending=6.
The higher values are too optimistic for budget drives.
Re: Optimization of ZFS with metadata on SSD possible
Posted: 20 Jul 2014 11:36
by 00Roush
himbrr wrote:00Roush wrote:Are things working better than when you first posted?
Yes of course! I think the real slowness came from the wrong SMB settings.
I will post the benchmark again, after I have reinstalled nas4free on the usb stick and added the SSD as cache for the raid.
Well that is good to hear. Let us know how it goes.
Cheers,
00Roush
Re: Optimization of ZFS with metadata on SSD possible
Posted: 24 Jul 2014 23:06
by himbrr
Hello,
I have migrated my nas4free on ssd to a usb stick.
After I have formated the ssd as ZFS Pool Device I tried to add it to the ZFS pool as cache.
But it didn't showed up in the devicelist.
Do you have any ideas, how I could add the ssd as cache device to the pool?
EDIT: Found it! Had to do this manually via commandline...
loader.conf options after ZFS kernel tune (8GB RAM)
Code: Select all
# ZFS kernel tune
vm.kmem_size="6656M"
vfs.zfs.arc_min="5120M"
vfs.zfs.arc_max="5120M"
vfs.zfs.prefetch_disable="0"
vfs.zfs.txg.timeout="5"
vfs.zfs.vdev.max_pending="10"
vfs.zfs.vdev.min_pending="4"
vfs.zfs.write_limit_override="0"
vfs.zfs.no_write_throttle="0"
EDIT2: Benchmarks
I have added the Benchmark over SMB as attachment. This was done before adding a log partition to the pool.
It's now a little bit slower 0,1 MB/s on writing and has now 600 IOps lesser.
Here is the dd Benchmark:
Code: Select all
nas4free: Management # dd if=/dev/zero of=testdata bs=1024000 count=10240
10240+0 records in
10240+0 records out
10485760000 bytes transferred in 36.071499 secs (290693770 bytes/sec) --> ~ 290,69 MB/s write speed
nas4free: Management # dd if=testdata of=/dev/zero bs=1024000
10240+0 records in
10240+0 records out
10485760000 bytes transferred in 33.369388 secs (314232912 bytes/sec) --> ~ 314,23 MB/s read speed
Cheers,
himbrr
Re: Optimization of ZFS with metadata on SSD possible
Posted: 25 Jul 2014 09:21
by crowi
Hi HImbrr,
I think the Crystaldiskmark won't show correct values on a network drive, it will be limited by the LAN speed.
Which size has your log device?
Can you post the output of: Disks|ZFS|Pools|Information ?
Re: Optimization of ZFS with metadata on SSD possible
Posted: 25 Jul 2014 09:30
by b0ssman
a zil is for synchronous writes. smb is async.
Re: Optimization of ZFS with metadata on SSD possible
Posted: 26 Jul 2014 12:12
by himbrr
Hi,
crowi wrote:an you post the output of: Disks|ZFS|Pools|Information ?
Code: Select all
pool: Management
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
Management ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ada1 ONLINE 0 0 0
ada2 ONLINE 0 0 0
ada3 ONLINE 0 0 0
ada4 ONLINE 0 0 0
ada5 ONLINE 0 0 0
logs
ada0p2 ONLINE 0 0 0
cache
ada0p1 ONLINE 0 0 0
errors: No known data errors
crowi wrote:Which size has your log device?
The log device has 10GB and the cache 45 GB. This fit on the 60GB SSD (55GB usable).