Very high cpu usage/poor performance with ZFS and writes

[Eliminateur]

Hello,
i'm experiencing a very high CPU usage on my new box that's essentially killing all performance.
Box is a Intel G31PR(latest bios) motherboard with a P4 651 (3.4Ghz HT, cedar mill) and 2GB of RAM.
The controller is an ICH7 plain, with no AHCI so NF is limiting me to 150MBps transfers.
NIC is RT8111 gigabit onboard, gigabit network
NF is 9.2.0.1 - Shigawire (revision 972) x64, kernel tune is on, ZFS tuning has been set for 2GB of RAM, HPET as system timer
there's a single ZFS pool set up as RAIDZ1 with 3x1.5TB HDD (ST31500341AS, 7200rpm, 512BPS, SATAII, each drive is capable of more than 100MB/s sustained speed. all 3 drives have the same FW version).
Samba is setup as SMB2/local, AIO, large transfers, SO_RCVBUF and SO_SNDBUF at 524288

The problem is that when i start copying data to the NF the performance is very poor and erratic, with SMB it shows choppiness and speed around 50~60MB/s, FTP fares somewhat better but then drops.
CPU Usage is at 95+%(of both "HT" cores), with ZFS compression it is even more erratic, cpu load doesn't changes.
I then did some tests with DD and file copies:

can't use compression ON with DD as freebsd dev/random performance is horrible (~50MB/s in my machine with 50% cpu load), so take it with a grain of salt

Compression OFF, using ZERO:

Code: Select all

nas4free: /mnt # dd if=/dev/zero of=/mnt/tank/Anime/file1 bs=8m count=500
500+0 records in
500+0 records out
4194304000 bytes transferred in 34.874673 secs ([b]120267909 [/b]bytes/sec)

~120MB/s which is not very good as that's the speed of a SINGLE drive, but that speed should be enough to saturate the gigabit ethernet conection

zpool stats concur

Code: Select all

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tank        1.87G  4.06T      1  1.20K    613   125M
  raidz1    1.87G  4.06T      1  1.20K    613   125M
    ada1        -      -      0    575    306  62.2M
    ada2        -      -      0    576    306  62.3M
    ada3        -      -      0    585      0  63.5M
----------  -----  -----  -----  -----  -----  -----

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tank        2.83G  4.06T      1  1.07K    715   107M
  raidz1    2.83G  4.06T      1  1.07K    715   107M
    ada1        -      -      0    509    204  53.4M
    ada2        -      -      0    508    102  53.4M
    ada3        -      -      0    488    408  51.2M
----------  -----  -----  -----  -----  -----  -----

top shows ~50% cpu usage and gstat shows a high busy percentage for all drives

Code: Select all

last pid: 23660;  load averages:  2.97,  1.71,  0.94                                 up 0+02:04:09  01:52:06
39 processes:  2 running, 37 sleeping
CPU:  1.6% user,  0.0% nice, 35.9% system,  3.9% interrupt, 58.6% idle
Mem: 43M Active, 43M Inact, 676M Wired, 22M Buf, 1184M Free
ARC: 511M Total, 29M MFU, 273M MRU, 207M Anon, 2079K Header, 947K Other
Swap: 2047M Total, 2047M Free

now the weird part is that gstat shows that the 3rd drive has 20% less busy time than the other 2!
0 and 1 remain between 85 and 97%~, in red all the time, but ada3 between 67 to 87!, mostly in 67 and all the time in purple, check out the difference in ms/w!, this drive is newer than the others(a refurbish of a failed one from that batch)

Code: Select all

dT: 1.010s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      2      0      0    0.0      2      8    3.9    0.4| ada0
    9    603      1      0    0.1    600  66862   12.2   97.8| ada1
    0      2      0      0    0.0      2      8    5.2    0.5| ada0s1
    0      0      0      0    0.0      0      0    0.0    0.0| ada0s2
    0      0      0      0    0.0      0      0    0.0    0.0| ada0s3
   10    601      1      0    0.2    598  66609   12.3   97.8| ada2
   10    599      0      0    0.0    597  67993    8.7   76.7| ada3

Now let's test on the network:

FTP transfer of a single large file to the NAS
Erratic speed, cpu usage though the roof

Code: Select all

last pid: 29435;  load averages:  2.96,  2.03,  1.42                                 up 0+02:29:26  02:17:23
40 processes:  2 running, 37 sleeping, 1 zombie
CPU:  2.4% user,  0.0% nice, 50.2% system, 40.6% interrupt,  6.9% idle
Mem: 39M Active, 44M Inact, 702M Wired, 22M Buf, 1162M Free
ARC: 512M Total, 23M MFU, 375M MRU, 111M Anon, 2257K Header, 1157K Other
Swap: 2047M Total, 2047M Free

  PID USERNAME      THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
29249 root            1  89    0 62696K 15308K RUN     0   0:21 58.98% proftpd

you can see the erraticness in the network, with drops to zero included, also top speed is averaging ~106MB/s, again 10MB/s slower than when reading, same FTP...

Code: Select all

          input          (re0)           output
   packets  errs idrops      bytes    packets  errs      bytes colls
         2     0     0        132          2     0        396     0
      3844     0     0    5733736       3837     0     254784     0
     64352     0     0   95976326      64388     0    4249290     0
     64717     0     0   96484954      64721     0    4275466     0
     68855     0     0  102643134      68871     0    4546431     0
     66512     0     0   99156296      66522     0    4392896     0
     55658     0     0   82970540      55660     0    3673920     0
         3     0     0        198          3     0        946     0
     10518     0     0   15678388      10517     0     695130     0
     68130     0     0  101573496      68138     0    4499076     0
     65915     0     0   98263544      65916     0    4352149     0
     72256     0     0  107711472      72268     0    4770752     0
     65132     0     0   97106376      65138     0    4299666     0
     63118     0     0   94092764      63126     0    4167494     0
     73729     0     0  109907290      73734     0    4866820     0
     59704     0     0   88999366      59710     0    3941528     0
     62970     0     0   93880596      62967     0    4156842     0
     71995     0     0  107322750      72007     0    4754291     0
     64430     0     0   96046220      64437     0    4253876     0
     71760     0     0  106971944      71769     0    4736992     0
     62288     0     0   92855816      62294     0    4111796     0

SMB transfer of a single large file

Again top showing the cpu getting killed:

Code: Select all

last pid: 30673;  load averages:  2.30,  1.71,  1.42                                 up 0+02:33:48  02:21:45
38 processes:  3 running, 35 sleeping
CPU:  8.6% user,  0.0% nice, 52.4% system, 32.9% interrupt,  6.1% idle
Mem: 29M Active, 43M Inact, 701M Wired, 22M Buf, 1172M Free
ARC: 512M Total, 1K MFU, 356M MRU, 152M Anon, 2678K Header, 1164K Other
Swap: 2047M Total, 2047M Free

  PID USERNAME      THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
 8049 root            1  86    0 82788K 19356K RUN     1   4:48 50.00% smbd

network speed is all over the place:

Code: Select all

            input          (re0)           output
   packets  errs idrops      bytes    packets  errs      bytes colls
     42042     0     0   63439194      42942     0    2912424     0
     75146     0     0  113546380      76677     0    5200314     0
     37334     0     0   56247780      38100     0    2586637     0
     78589     0     0  118657410      80151     0    5435886     0
     40110     0     0   60447380      40938     0    2777780     0
     56646     0     0   85489892      57753     0    3920154     0
     33252     0     0   50276861      33968     0    2305088     0
     44161     0     0   66607224      45101     0    3059984     0
     58872     0     0   88896837      60069     0    4074412     0
     25376     0     0   38302440      25882     0    1756371     0
     79420     0     0  119958272      81018     0    5496478     0
     76142     0     0  115039612      77675     0    5267694     0
     38285     0     0   57741066      39072     0    2649512     0
     61560     0     0   92889456      62788     0    4257752     0
     67621     0     0  102146176      69031     0    4683060     0
     44574     0     0   67325644      45525     0    3087026     0
     46240     0     0   69675560      47163     0    3200163     0
     78792     0     0  119026976      80380     0    5450686     0
     50105     0     0   75644986      51128     0    3467700     0
     44378     0     0   66887580      45299     0    3072494     0
     79251     0     0  119646654      80803     0    5480774     0

Windows is showing an average of 73MB/s~

Read tests:
reading a file is not showing the same erratic behaviour, it starts at ~80% network then after a little more than half the file is transfered it drops to around 57% and stays there, but the transfer graph is flatter.

CPU usage is still through the roof:

Code: Select all

CPU: 11.2% user,  0.0% nice, 37.6% system, 35.3% interrupt, 15.9% idle
Mem: 34M Active, 47M Inact, 678M Wired, 22M Buf, 1187M Free
ARC: 512M Total, 12M MFU, 497M MRU, 272K Anon, 1756K Header, 1047K Other
Swap: 2047M Total, 2047M Free

  PID USERNAME      THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
 8049 root            1  90    0 86824K 23368K RUN     0   3:08 57.96% smbd

you can easily see the drop in performance here, CPU usage also drops:

Code: Select all

          input          (re0)           output
   packets  errs idrops      bytes    packets  errs      bytes colls
     27966     0     0    2010492      64831     0   96488580     0
     29482     0     0    2121780      69468     0  103262075     0
     29263     0     0    2103582      67870     0  100970058     0
     27382     0     0    1970076      64515     0   96077110     0
     30463     0     0    2188398      70379     0  104727138     0
     30152     0     0    2166000      69315     0  103037552     0
     29556     0     0    2123342      67530     0  100248002     0
     30601     0     0    2195770      70095     0  104448506     0
     21676     0     0    1552296      48629     0   72038937     0
     13060     0     0     937080      28714     0   42614220     0
     27136     0     0    1950096      62721     0   93410386     0
     18945     0     0    1358946      43831     0   64964822     0
     19921     0     0    1427106      43875     0   64969882     0
     19808     0     0    1420062      44484     0   66147154     0
     18414     0     0    1322028      41731     0   62220042     0
     20527     0     0    1459614      46978     0   69578295     0
     19590     0     0    1418500      44373     0   65774374     0

with FTP the speeds are MUCH better, a solid 10MB/s more reaching 90%+ of the connection but it has the same drop

Code: Select all

 packets  errs idrops      bytes    packets  errs      bytes colls
     33109     0     0    2185194      28422     0  114730426     0
     32606     0     0    2151996      27276     0  113629798     0
     33594     0     0    2217618      29961     0  115448840     0
     29690     0     0    1959540      20988     0  103360830     0
     29492     0     0    1946472      26328     0  104963565     0
     32327     0     0    2133582      29589     0  112333250     0
     33516     0     0    2212056      30994     0  114397604     0
     30729     0     0    2028114      27870     0  112994556     0
     34125     0     0    2252250      33388     0  118170854     0
     31999     0     0    2112348      23621     0  111804194     0
     30866     0     0    2037156      26120     0  108785584     0
     29870     0     0    1971420      26584     0  108504330     0
     22052     0     0    1455894      19753     0   77006738     0
     17987     0     0    1187142      13505     0   61662670     0
     16747     0     0    1105302      12847     0   61329272     0
     18718     0     0    1235802      13626     0   62231770     0
     18385     0     0    1213410      14831     0   63294738     0
     16677     0     0    1100682      13451     0   58835133     0
     18783     0     0    1239678      14697     0   63030224     0
     18361     0     0    1211962      14614     0   61782360     0
     18665     0     0    1231890      14774     0   61792728     0

gstats or zpool do not show high busy, lower than 20%


now even if this is an "old" P4, it's still one of the latest models out(in fact it's the latest P4 core), and they had excellent integer performance, it shouldn't be doing this!.
¿why is ftp pumping a solid 10MB/s more than SMB?
¿why are read speeds tanking after some time?(the ARC cache is at 512MB and the tanking is waaay past that)
why are interrupts eating so much CPU?

any ideas?

[b0ssman]

you cpu is woefully underpowerd.

see
http://www.cpubenchmark.net/low_end_cpus.html

its slower than the first generation atom.

[Eliminateur]

i'd take the cpu passmark with a grain of salt asit's a very synthetic test.
for example a nice comparison here http://ixbtlabs.com/articles2/cpu/intel ... page1.html shows it not very far from the E2160 in real usage scenarios(look for the ones that use integer) whilst the passmark shows the E2160 at 997 vs 405 for the 651....
i do have a Pentium D805 i could test, but it's prescott and lower speed so i dont think it will make a difference.
(i should also test a Xeon 3075 i have laying around but that one is used atm and the X3210 is too overkill for this)

[b0ssman]

no the xeon is not overkill.

because those systems have ecc. that is exactly what you NEED for zfs.

see

http://forums.freenas.org/index.php?thr ... zfs.15449/

[Eliminateur]

no the xeon is not overkill.
because those systems have ecc. that is exactly what you NEED for zfs.
see
http://forums.freenas.org/index.php?thr ... zfs.15449/

bossman, i'm not going to use ECC, those are s775 Xeon that can be popped in any S775 desktop motherboard(in which i've used both of them before).
the 3075 is the equivalent of the very first C2D, the E6600 but with 1333MHz bus (conroe 2.66Ghz/4M/1333)
the X3210 is similar to the Q6600 with 1333MHz bus but 2.12GHz clock vs 2.66 of the 6600
for the usage i'm planning on for this nas it's very overkill (this will be a video files storage server, which means low speed streaming reads and very high sustained sequential writes needed)

[crowi]

i'm not going to use ECC

Then you should strongly reconsider using ZFS at all.

Anyway, 2GB RAM is way to less for ZFS, you need 1GB (ECC) RAM per 1 TB disk space plus 1-2GB RAM for the system in order to have your filesystem work properly.
http://constantin.glez.de/blog/2010/04/ ... erformance

[Eliminateur]

i've switched to a C2D E4600 (2.4/2M/800, Conroe-2M) and the performance issue is *mostly* gone, i still see a mess of performance with SMB but the peaks are almost full BW and variations not as drastic.
CPU usage is still very high showing ~84% of both cores

Code: Select all

 packets  errs idrops      bytes    packets  errs      bytes colls
     76915     0     0  115959006      78638     0    5425248     0
     66625     0     0  100267555      68006     0    4665937     0
     45342     0     0   68200647      46394     0    3176075     0
     80693     0     0  121047976      82106     0    5626957     0
     68760     0     0  103313141      70111     0    4794014     0
     63456     0     0   95319373      64927     0    4482851     0
     31598     0     0   46782839      32347     0    2272814     0
     42435     0     0   62556519      43354     0    2992144     0
     78796     0     0  118207283      80554     0    5506092     0
     64092     0     0   95863696      65437     0    4453265     0
     70728     0     0  106163522      72275     0    4943918     0
     73211     0     0  109689379      74739     0    5148535     0
     29642     0     0   43734458      30290     0    2095001     0
     69540     0     0  104108315      71042     0    4862674     0
     79538     0     0  119324235      81203     0    5524455     0
     69106     0     0  103561051      70544     0    4793535     0
     66326     0     0   99404422      67875     0    4652554     0
     78804     0     0  118080798      80447     0    5501156     0
     48181     0     0   71749712      49173     0    3353349     0
     63979     0     0   96113643      65374     0    4462251     0
     79827     0     0  119661084      81579     0    5612169     0

i know that ZFS recommends at least 4GB and i know ECC is recommended, but ECC requires a server board and EXPENSIVE ram, both things i'm not going to buy for a recycled-parts home server(and putting more very expensive and very hard to get DDR2 ram is out of the quesiton for now).
i'll see how it works with 2GB, so far memory usage is at 43% (i've setup a 640MB ARC limit) even with smbd transferring heavily and transmission running so i dont see why i should put more if i have 932MB free i could even increase the ARC max to 768 or even a gig

[crowi]

know that ZFS recommends at least 4GB and i know ECC is recommended

That is not quite true. If you want to have your data stored safely, ECC is prerequisite for using ZFS
Not using ECC RAM is a timebomb, which can lead to a complete loss of data due to silent corruption.

Please read and understand the article b0ssman recommended before:
http://forums.freenas.org/index.php?thr ... zfs.15449/

In your system softRAID is the way to go.

[Eliminateur]

softraid does not give me the data resiliency i need, i wouldnt even bother setting up a NAS with softraid, i chose ZFS due to the inherent resiliency to bad blocks/silent corruption due to checksum.
otherwise i'd leave the drives inside my PC and have more storage available.
Also 99% of the files in the datastore have checksum values in their filename which are also checked regularly outside of the NF box so a corruption wouldn't go unnoticed

and again, having to buy costly and unavailable ECC platform(i'd have to find an ECC-capable CPU, then find if a normal MB can support it when they dont advertise that, it's hit or miss, or a server board that's simply not available in my country, plus ecc ram costs more than twice than normal ram -which is alreayd quite expensive-) is outside the scope of this "no cost" build made of repurposing old hardware.
i'll keep it in mind but for now it's outside of the monetary scope

[b0ssman]

we just want to make sure that you understand the implications of zfs and ecc. That when you memory goes bad reading data will destroy your data.

We have told you about the general requirements and implications so that when you data gets corrupted that you are on your own.

[Eliminateur]

i'll take it into consideration, if a ram goes bad and something breaks i'll see how to recover the data, no problems.
btw, are there any kind of "community HCL" with a list of known desktop boards that support ECC?(without being server boards)

[00Roush]

Lots of Asus AMD boards support ECC memory. From socket 939 all the way up to AM3+. I think Intel is more specific and it is limited more to server chipsets but I am not as familar with Intel ECC support.

Not sure if anyone really answered you basic questions of why CPU usage is so high. FTP generally is considered to have the lowest overhead (lowest CPU usage). When CPU usage is high it tends to provide better performance vs SMB. In my experience it seems like Samba has higher CPU usage vs native Windows SMB. ZFS also uses more CPU resources vs softRAID. Combine the two and CPU usage can be more than 50% on older CPUs (even on newer CPUs). Another thing possibly adding on to your CPU usage is the interrupt usage which could be related to Realtek NIC. Typically Realtek NICs have used more CPU resources as they are not supported as well in FreeBSD or just don't have the offload functions like Intel NICs.

Recommend you do iperf tests and review CPU usage during max network load and no disk load. This can help isolate how much CPU usage is needed just for network. Also this will give you an idea of the maximum possible speeds you can expect to get between N4F box and client.

Any chance you could try DD write tests again with a smaller block size like bs=1M? Sanity check to see if performance is similar vs bs=8M.

My opinion is for the best performance you would want to start with this setup.
NAS4Free x64 full version (not embedded) with no swap (not used) and no special settings or changes from default.
ZFS kernel tune extension installed and set to 2GB.
Samba setup -- SMB2 (default), Send/Receive buffer size 0 (disables and lets the OS autotune), AIO enabled, DOS attributes disabled (speeds up browsing folders with lots of files), and the rest at defaults.

Hope it helps,
00Roush

[Eliminateur]

00rush, yeah i know about Asus AMD boards supporting ECC, in fact i have two of them here at work and one of them with a mix of ecc/non ecc ram, i'm in the process of trying to find a used one as all i can find in new models are those uber-expensive 990FX ultra-gamer models (it also doesn't helps that all FX CPU are quite expensive, cheapest one here is ~130$, so i'm also trying to get a used phenom II x3 or x4).
letme piece your recomendations:
NAS4Free x64 full version -->it's what i'm using, only special setting is an added 16MB VDEV cache (it made no practical difference)
ZFS kernel tune --> for 2GB it sets a very low 512MB or ARC_MAX, i upped this gradually to 640, 768 and now 1GB safely, plenty of ram available even when running samba and transmission.
samba ->already in SMB2, but i did set buffer to 65535, the default and 512K, didn't notice much difference in real performance, never considered "0", AIO is enabled, dos att disabled, the zfs poll as no atime

problem about using DD is that i have LZ4 comp on the pools, so a DD with zeroes gives a very false synthetic top, and random source has a 50MB/s top speed with high cpu usage.
i've ran iperf and it goes ~860Mbps both ways

[00Roush]

I ran through a bunch of testing to try and recreate the original issues you had mentioned and I think I have been able to.

To test I had a N4F VM I cut down to 2GB RAM and limited CPU usage. Tested reading and writing files (SMB) ranging from 2GB to 16GB.

I think read issues could have been related to using AIO. Read speeds seemed to fall off after a few seconds with AIO on. In my testing read speeds didn't seem to drop off when I disabled AIO. Adding more CPU power helped as well.

Poor write speed and erratic write speeds could be related to not enough CPU power and compression. Adding CPU power brought SMB write speeds up to about the maximum possible over gigabit (110 MB/sec) and overall fairly consistent. However enabling compression (default or LZ4) caused average write speeds to fall due to what looks to be IO stalls. CPU usage was at about 50-60%. Added more CPU cores and memory (8GB) and IO stalls look to go away.

If your main goal is performance I would say try disabling compression and test. ZFS set compression=off pool

Hope it helps.

00Roush