Geli+ZFS=very slow - is it normal?

[halfer]

From: http://sourceforge.net/apps/phpbb/freen ... =80&t=8007

Said "pepak":

I wonder: What kinds of speeds are you getting with ZFS volumes stored on GELI-encrypted disks? I am rather concerned, because even after upgrading my NAS's hardware I am seeing speeds not higher than 5 MB/s. I find it unacceptable given the specs, which are way over what FreeNAS is supposed to require:

- CPU: Core 2 Duo 6550
- RAM: 1 GB DDR2
- Network: 1 Gbit
- Disk controller: Areca 1120 in a PCI-X slot
- Disk: to FreeNAS, it appears like one big volume; in any case, they are quite new 1.5 TB Samsungs.
- FreeNAS: 7.1 stable
- GELI: 256-bit AES, software
- ZFS: single disk pool (no RAID, no mirroring)

Some googling revealed this thread: http://forums.freebsd.org/showthread.php?t=13788
where someone claims that a patch is needed "to get acceptable performance with ZFS+GELI". I wonder: is that patch included in FreeNAS?

[halfer]

Said "serguei00":

0.7.2 Sabanda (revision 7S-5255)
amd64-embedded on Intel(R) Pentium(R) Dual CPU E2160 @ 1.80GHz
6GB of DDR2 RAM
Network: 1 Gbit
GELI: 128-bit AES - default
ZFS: 4 x disk 1,5TB in pool ZRAID1


typical speed on write no more then 10MB/sec

Code: Select all

                     capacity     operations    bandwidth
    pool           used  avail   read  write   read  write
    ------------  -----  -----  -----  -----  -----  -----
    zstore        2.43T  3.01T      0     91  64.8K  7.77M
      raidz1      2.43T  3.01T      0     91  64.8K  7.77M
        ad10.eli      -      -      0     35  21.4K  2.60M
        ad12.eli      -      -      0     35  12.7K  2.60M
        ad14.eli      -      -      0     35  21.6K  2.60M
        ad16.eli      -      -      0     40  9.27K  2.60M
    ------------  -----  -----  -----  -----  -----  -----

                     capacity     operations    bandwidth
    pool           used  avail   read  write   read  write
    ------------  -----  -----  -----  -----  -----  -----
    zstore        2.43T  3.01T      1     12   104K   262K
      raidz1      2.43T  3.01T      1     12   104K   262K
        ad10.eli      -      -      1      6  26.6K  89.2K
        ad12.eli      -      -      0      7  17.4K  89.2K
        ad14.eli      -      -      1      7  30.0K  88.8K
        ad16.eli      -      -      1      6  30.1K  85.3K
    ------------  -----  -----  -----  -----  -----  -----

                     capacity     operations    bandwidth
    pool           used  avail   read  write   read  write
    ------------  -----  -----  -----  -----  -----  -----
    zstore        2.43T  3.01T      0     32  17.9K  4.03M
      raidz1      2.43T  3.01T      0     32  17.9K  4.03M
        ad10.eli      -      -      0     17  5.17K  1.35M
        ad12.eli      -      -      0     17  5.75K  1.35M
        ad14.eli      -      -      0     17  1.56K  1.35M
        ad16.eli      -      -      0     17  5.46K  1.35M
    ------------  -----  -----  -----  -----  -----  -----

what can I do?
may be change sector size in

Code: Select all

    geli init -s 4096

[halfer]

Said "dani-l":

Yes, aligning your geli volumes (happens when sectorsize is 4096) solves the problem, IF you are using Advanced Format/4k secsize drives.
I went from 17 Mbyte/s to 31 Mb/s avg.write and avg.read over Gigabit. I maxed the onboard NIC and need to buy a dedicated one for further testing.

You init your geli volumes as serguei said.

On FN7 you check the secsize by

Code: Select all

    diskinfo -v ad6.eli 

for instance.

Then you need to format the new geli volume as a zfs storage pool device. You can do that over the GUI.

After this you need to create a zpool from your newly created ad#.eli.nop volumes. YOu must do this over the console so you can control the ashift value.

When using FN7, you run

Code: Select all

    zpool create anyname raidz ashift=12 ad6.eli.nop ad7.eli.nop 

or using FN8 you do everything over the GUI and then check the ashift values by

Code: Select all

    zdb -C