[RESOLVED] No performance increase with 4 disks over 2!

[chrisf4lc0n]

Hi I just added another 2 disks to my original ZFS Mirror Pool and it looks like that now:

Code: Select all

nas4free:~# zpool status
  pool: ZFS_RAID
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        ZFS_RAID    ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada0    ONLINE       0     0     0
            ada2    ONLINE       0     0     0
          mirror-1  ONLINE       0     0     0
            ada3    ONLINE       0     0     0
            ada4    ONLINE       0     0     0

errors: No known data errors

My read speeds have only gone up by something like 10 Mb/s to around 80 Mb/s, when write speeds have significantly gone down to around 30 Mb/s from around 50 Mb/s??
I am using 3x320GB HDDs 2 of which are exactly the same Caviar Blue and the 3rd one is a 2.5" Fujitsu from my old laptop plus 1x500GB 5400RPM Samsung SpinPoint salvaged from the Maxtor USB external HDD...
Could it the different size in my config cause bottlenecking, or would it be the 2.5" Fujitsu?

Edit: I am also using OEM JMB363 4x SATA II controller...

[raulfg3]

you need to check all chain involved ( Disk, ZFS, NIC, RAM) etc...
http://constantin.glez.de/blog/2010/01/ ... still-best

Do you tune your system to your new 8GB RAM? viewtopic.php?f=71&t=1278&p=19924

To tune your SMB conection you need to change parameters one by one and observe result, keep in mind that Windows need to be tunned too: viewtopic.php?f=74&t=3805

[chrisf4lc0n]

loader.conf:

Code: Select all

kernel="kernel"
bootfile="kernel"
kernel_options=""
kern.hz="100"
hw.est.msr_info="0"
hw.hptrr.attach_generic="0"
kern.maxfiles="65536"
kern.maxfilesperproc="50000"
kern.cam.boot_delay="8000"
autoboot_delay="5"
isboot_load="YES"
zfs_load="YES"
kern.maxvnodes="250000"
# ZFS kernel tune
vm.kmem_size="6656M"
vfs.zfs.arc_min="5120M"
vfs.zfs.arc_max="5120M"
vfs.zfs.prefetch_disable="0"
vfs.zfs.txg.timeout="5"
vfs.zfs.vdev.max_pending="1"
vfs.zfs.vdev.min_pending="1"
vfs.zfs.write_limit_override="131072000"
vfs.zfs.no_write_throttle="0"

smb.conf:

Code: Select all

[global]
encrypt passwords = yes
netbios name = nas4free
workgroup = Poland
server string = NAS4Free Server
security = user
max protocol = SMB2
dns proxy = no
# Settings to enhance performance:
strict locking = no
read raw = yes
write raw = yes
oplocks = yes
max xmit = 65535
deadtime = 15
getwd cache = yes
socket options = TCP_NODELAY SO_SNDBUF=128480 SO_RCVBUF=12848 
# End of performance section
unix charset = UTF-8
store dos attributes = yes
local master = yes
domain master = yes
preferred master = yes
os level = 35
time server = yes
guest account = ftp
map to guest = Never
display charset = LOCALE
max log size = 100
syslog only = yes
syslog = 1
load printers = no
printing = bsd
printcap name = /dev/null
disable spoolss = yes
log level = 1
dos charset = CP437
smb passwd file = /var/etc/private/smbpasswd
private dir = /var/etc/private
passdb backend = tdbsam
idmap config * : backend = tdb
idmap config * : range = 10000-39999

[Backup]
comment = Samsung Backup
path = /mnt/Samsung/
writeable = yes
printable = no
veto files = /.snap/.sujournal/
hide dot files = yes
guest ok = no
inherit permissions = yes
vfs objects = shadow_copy2 
shadow:format = auto-%Y%m%d-%H%M%S
shadow:snapdir = .zfs/snapshot
shadow:sort = desc
shadow:localtime = yes

[ZFS_RAID]
comment = ZFS Pool
path = /mnt/ZFS_RAID/
writeable = yes
printable = no
veto files = /.snap/.sujournal/
hide dot files = yes
guest ok = no
inherit permissions = yes
vfs objects = shadow_copy2 zfsacl 
shadow:format = auto-%Y%m%d-%H%M%S
shadow:snapdir = .zfs/snapshot
shadow:sort = desc
shadow:localtime = yes
veto files = /.zfs/

ifconfig:

Code: Select all

re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 6000
        options=80098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
        ether b8:97:5a:28:bf:63
        inet 192.168.0.16 netmask 0xffffff00 broadcast 192.168.0.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet 1000baseT <full-duplex>
        status: active

Same MTU 6000 on Windows.

[chrisf4lc0n]

I doubt it is controller or Samba config fault as I am still getting ~50Mb/s when copying to UFS disk...
So it is either one of the disks which is bottlenecking or there is a problem with the ZFS config somewhere!

[kkd]

hi,
i had fujitsi hdd and it was extremly slow because of it's low cache.
and image what kind of hdd u had in an usb external gadget.... maybe not the fastest one was built in.
it worths a try to make a smart test, but i thing it was a bad choise to use 2 slow hdds even if the raid is 0.

[Lee Sharp]

You do not have a 4 disk pool, but a pair of two disk pools. So you do not get the full boost in performance. If you can, pull off all the data, and build a totally new 4 disk pool. You will see a much bigger boost in performance.

[raulfg3]

I do not see AIO active in SMB, revise other post to see what parameters have more influence on speed, for me are AIO, and SMB=NT1, and size of buffers, ( I use 64240x2=128480) and works better for my NIC ( Intel em0).

try enable / disable large read write and use send file, some times disable it's better

[chrisf4lc0n]

Here we go I have done a wide range of tests:

1. 320GB Fujitsu and 500GB Samsung SpinPoint in Mirror RAID both of them on JMB363 Controller:

Code: Select all

nas4free:~# zpool status
  pool: ZFS_RAID
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        ZFS_RAID    ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada0    ONLINE       0     0     0
            ada2    ONLINE       0     0     0

errors: No known data errors

Fujitsu with SpinPoint.jpg

2. 2x320GB Caviar Blue in Mirror RAID on the buil-in mobo controller:

Code: Select all

nas4free:~# zpool status
  pool: ZFS_RAID
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        ZFS_RAID    ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada3    ONLINE       0     0     0
            ada4    ONLINE       0     0     0

errors: No known data errors

2x Caviar.jpg

3. All disks in 1 Mirror Pool:

Code: Select all

nas4free:~# zpool status
  pool: ZFS_RAID
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        ZFS_RAID    ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada0    ONLINE       0     0     0
            ada2    ONLINE       0     0     0
            ada3    ONLINE       0     0     0
            ada4    ONLINE       0     0     0

errors: No known data errors

All 4 in 1 Pool.jpg

4. 2 Mirror Pools with 2 disks each:

Code: Select all

nas4free:~# zpool status
  pool: ZFS_RAID
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        ZFS_RAID    ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada0    ONLINE       0     0     0
            ada2    ONLINE       0     0     0
          mirror-1  ONLINE       0     0     0
            ada3    ONLINE       0     0     0
            ada4    ONLINE       0     0     0

errors: No known data errors

2x2 Pool.jpg

5. UFS 640GB Samsung SpinPoint on JMB363 controller used as backup for comparison:

Samsung.jpg

Also before I installed the JMB363 controller and only had 2x320GB Caviar in place I had around 80 Mb/s write speeds too, so what the heck??

[chrisf4lc0n]

1. Cannot use NT1... Windows cannot access shares when enabled
2. AIO on makes performance degrade even further.
3. Sendfile is not working either...
4. Single disk in UFS seems to perform alright and it was fine with just 2 disks at the beginning floating for read/write ~80Mb/s...

[chrisf4lc0n]

And this is a test with Stripped RAID:

Stripped.jpg

That means whatever RAID config I use now seems to be performing worse than a single UFS disk!

[raulfg3]

1. Cannot use NT1... Windows cannot access shares when enabled

strange , works for me very well ( I use Win7 in the other side and NT1 and anonymous in my NAS)

2. AIO on makes performance degrade even further.

Sorry,not for me , I notice a great boost performance from 50MB to 90MB

3. Sendfile is not working either...

, I do not notice increase or decrease performance so I disable, but I mount at least 5 more NAS for friends and some times increase performance ( and others no), so my suggest is to do test a choose better option.

I notice better performance if double the send and recive buffers ( 128480 instead 64240 by default) <- In a old NAS that use realtek 8111C NIC I need to use 32120 to increase stability ( less spikes).

[chrisf4lc0n]

It is not Samba's config fault, single UFS drive performs right!

[kkd]

sendfile function is not working:
viewtopic.php?f=78&t=2031&p=20655&hilit=sendfile#p20655

[raulfg3]

thanks, this is coherent with my reply I ask about old install, perhaps in the time that still works, sorry if my answer confuses or generate noise in the thread.

[chrisf4lc0n]

OK, I think I have found the problem, for some reason all drives report UDMA6 and PIO892!

Code: Select all

ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <ST3500630AS 3.AFM> ATA-7 SATA 2.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C)
ada0: Previously was known as ad4
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <SAMSUNG HD502HI 1AG01118> ATA-7 SATA 2.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad6
ada2 at ahcich2 bus 0 scbus3 target 0 lun 0
ada2: <ST2000DM001-1CH164 CC24> ATA-8 SATA 3.x device
ada2: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada2: Previously was known as ad10
ada3 at ahcich3 bus 0 scbus4 target 0 lun 0
ada3: <WDC WD3200AAKS-22B3A0 01.03A01> ATA-8 SATA 2.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 305245MB (625142448 512 byte sectors: 16H 63S/T 16383C)
ada3: Previously was known as ad12
ada4 at ahcich4 bus 0 scbus5 target 0 lun 0
ada4: <WDC WD3200AAKS-00V1A0 05.01D05> ATA-8 SATA 2.x device
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 305245MB (625142448 512 byte sectors: 16H 63S/T 16383C)
ada4: Previously was known as ad14

But why no AHCI?
I need to go back to BIOS...

[chrisf4lc0n]

Code: Select all

May 28 10:21:58 nas4free kernel: GEOM: ada0: the primary GPT table is corrupt or invalid.
May 28 10:21:58 nas4free kernel: GEOM: ada0: using the secondary instead -- recovery strongly advised.
May 28 10:21:58 nas4free kernel: GEOM: ada1: the primary GPT table is corrupt or invalid.
May 28 10:21:58 nas4free kernel: GEOM: ada1: using the secondary instead -- recovery strongly advised.
May 28 10:21:58 nas4free kernel: GEOM: ada3: the primary GPT table is corrupt or invalid.
May 28 10:21:58 nas4free kernel: GEOM: ada3: using the secondary instead -- recovery strongly advised.
May 28 10:21:58 nas4free kernel: GEOM: ada4: the primary GPT table is corrupt or invalid.
May 28 10:21:58 nas4free kernel: GEOM: ada4: using the secondary instead -- recovery strongly advised.

Found it! GPT table was corrupt and it degraded the performance of my disks!
This is the result after fixing the problem:

Result.jpg

Code: Select all

nas4free:/mnt/ZFS_RAID# dd bs=1M count=128 if=/dev/zero of=test
128+0 records in
128+0 records out
134217728 bytes transferred in 0.867861 secs (154653479 bytes/sec)

[raulfg3]

please post how to fix GPT table to help others in same situation.

[chrisf4lc0n]

As requested by a senior member of the forum a guide on how to fix GPT Table:

1. Backup all your data from the partition you will be fixing the GPT from, I know it can be a pain, but I do not know how to do that without destroying the partition. For those using mirror raid it should not be a problem if only 1 disks has got GPT corrupt, as it will start rebuilding once added back to the pool.
2. I had to destroy the pool as all the drives had GPT corrupt!
3a. In SSH:

Code: Select all

zpool destroy -f "name of your pool"

for example:

Code: Select all

zpool destroy -f ZFS_RAID

Code: Select all

gpart destroy -F gpt "name of the disk where gpt will be destroyed"

3b. In my case all 4 partitions, so:

Code: Select all

gpart destroy -F gpt ada0
gpart destroy -F gpt ada1
gpart destroy -F gpt ada3
gpart destroy -F gpt ada4

4. Now you need to recreate GPT:

Code: Select all

gpart create -s gpt ada0
gpart create -s gpt ada1
gpart create -s gpt ada3
gpart create -s gpt ada4

5. I also prefer having 4kb sectors so:

Code: Select all

gnop create -S 4096 ada0
gnop create -S 4096 ada1
gnop create -S 4096 ada3
gnop create -S 4096 ada4

6. Re-create pool:

Code: Select all

zpool create -fm /mnt/ZFS_RAID ZFS_RAID mirror /dev/ada0.nop /dev/ada1.nop mirror /dev/ada3.nop /dev/ada4.nop

7. Check if the pool has been created with 4kb sectors by:

Code: Select all

zdb -C ZFS_RAID | grep ashift

if it is 12 then you have got 4kb sectors if it is 9 something has gone wrong.

8. For those who has only got 1 disk with GPT corrupt, you just need to detach device from the pool by in my example lets say the pool is a mirror pool with ada0.nop and ada1.nop, where ada0.nop GPT is corrupt:

Code: Select all

zpool detach ZFS_RAID /dev/ada0.nop 

then you go to 3b and follow the instruction to 5 where you:

Code: Select all

zpool attach ZFS_RAID /dev/ada0.nop 

Hope it will help.

[raulfg3]

thanks a lot.