[fixed] Slow CIFS performance

[catwiesel]

Hi,

I am relatively new to NAS4Free, BSD and NFS, please bare with me.

I bought and set up a nas4free box as my backup device. However, I do get miserable slow transfer speeds which confuses me.

Hardware:
ASRock Board + DualCore AMD CPU 2,5GHz
8GB DDR3 RAM
6x Hitachi 7200rpm 3TB HDDs

I left most nas4free settings alone.

I installed the current (1 week ago) x64 nas4free version on a usb stick (embedded, memdisks, no writes on usb)

What I did set up:
- users, passwords
- ZFS RAID-Z (1 parity / "raid5") with all 6 drives
- zfs pool/dataset (/mnt/pool1/dataset1)
- CIFS
- (ssh)

now, transfer speeds, measured with windows 7 (cifs)
[i tried two different gbit switches. one consumer grade, the other a "real" one]
- uploading 4gbs in small-ish files, multiple subfolders (install files like firefox setup, thunderbird setup and so on) i get ~4mb/s
- downloading the 4gbs small-ish files i get maybe 6mb/s

- uploading a 4gb big file I get 10mb/s
- downloading the 4gb file I get 12mb/s

- a dd if=/dev/zero of=test bs=1M count=4096 finishes in ~12secs [i realise this value is "lying" since caching is not disabled. cant figure out the option to do so in nas4free)

neither ram nor cpu are going anywhere noticeable, so I dont think that I hit the cpu/ram ceiling.

I have tried to disable zfs sync, as it has been stated this would increase performance. it might, but it was negleteable. also, I chose zfs for data security reasons, hence I should let sync enabled.

I did enable the cifs AIO (default values), enable tuning and (tried to) enable sendfile (it deactives itself apparently)

I tried to transfer files via ftp, after i finally got that to work (host entries, sigh!), it appears to be able to transfer files up to 40MiB/s (thats mb/s, right?)
That is better and would actually be acceptable, still, the cpu/ram limit was nowhere hit.

Well, to make matters short. Since I have no real experience with nas4free, samba and tuning said systems, I do need help.

I can post logs, try stuff. No real data is on the nas so far, so I could even start from the beginning.
But please help me figure out what the real limit for this machine is and reach it.

I will not be able to purchase or add more/other hardware. And I really would like to keep using zfs in "raid5" mode. (no, i dont care for spare drives. yes i know this may seem stupid. but that is not the issue here.)

[raulfg3]

ok, first you need to install zfskerntune, and tune your systm for 8GB RAM
do test using same big ISO file.
once done.

Tune SMB:
-Set NT1 and anonimous ( NOT SMB2 at this point, perhaps later when you have better speed).
-SET AIO
change buffer size to /2 and to x2 and do test on both cases and compare with actual buffer size 64240.
-Enable tunning of some kernel variables in WebGUI (General).
Repeat Test, and repeat change default send and receive values to x2 and /2 (64240x2=128480).

Finally use best values for you.


At this point you must notice 40MB in SMB transfer ( expected for your FTP test , but for your hardware must be 80MB ), and you can play with SMB2 instead NT1 if you want.

[catwiesel]

When I try to tune the memory with the zfs kernel tune I get the error:

The following input errors were detected:

config read error.


can i not just add the appropriate settings in sysctl.conf ?

[raulfg3]

can i not just add the appropriate settings in sysctl.conf ?

yes, but I do not understand why you have this error.

open zfskerntune.sh script using a text ditor (in your PC) and see what are the apropiate values for your RAM

[catwiesel]

i "fixed" it. (the script error, not the speed problem)

I logged in as root, used the fetch and tar command in my shared folder.
I ran the script as root and got the error in the webinterface logged in as admin (the webadmin "root")

When I went to look for the values in the .sh script you mentioned, I noticed/realised that all relevant files were owned by root in a folder who is owned by [username]. seeing as the error might have something to do with not being able to read a file because unix file permissions were set wrong, I did a chown on all files to [username] and the script ran.

I dont fully understand WHY ownership by root would stop/hinder a script run in the webinterface, but apparently it did.
Lucky, because I found the values in the script, but would have had no clue in how to set them up properly by hand.

going to test now. I will be back with the results

[catwiesel]

What the hell?! I dont get it...

When I built this/my nas, I built a second one.

same board, cpu, psu, case, same hdds, same usb sticks...

there is however one difference. the second one only got 3 hdds and 4gb ram (it is for a buddy of mine)
I set both of them up almost identical. different user/pass, everything else. identical.
my system - his system
6hdds - 3hhds
8gb ram - 4gb ram

both have a zfs raidz1 with all hdds. both have cifs shares and ssh enabled, nothing else

So I unpacked his nas to apply any fine tuning settings if I found any. and to be honest, I wanted to delete the zfs and try "normal" raid5 to see if zfs is actually the problem here.
Luckily I was curious enough to try if his setup gets the same speed than mine or not.

And what do I see? i fraking download with 100+mb/s and upload with ~80mb/s over cifs.

this is a brand new windows7 pc / core i5 with a low end 7200hdd, so I expect the numbers I am seeing are as good as it gets with this client pc.

now, what i absolutely dont understand, why does the same hardware/setup give me what I expect, maybe more, and the other gives me 10% of that?

the obvious answer must be that in the difference of the system lies the answer. we have 3 more hdds and more ram.
I can not imagine that more ram means slower speeds. So either more drives slow zfs down (which would be the first time I'd hear that) or the settings/setup are not identical although I did my best to make sure they are. Or something is broken.

I need to find out what is going on!

At least my buddies NAS is working superb...

[raulfg3]

the obvious answer must be that in the difference of the system lies the answer. we have 3 more hdds and more ram

not as obvious as you think, in my experience if a pool decrease speed in a noticeable way is sympton of a failed / failing disk.

Please pass a S.M.A.R.T. test to your disk to discard in first time a faulty disk.

Nex you can continue tunning, and YES more RAM = More Speed ( but only if RAM is tunned)

[catwiesel]

i have thought about a broken harddrive too, quick smart test is passing with flying colours.

also, all drives are brand spanking new. this would be the first time I have a "broken" drive on arrival...

[catwiesel]

okay, the slow cifs/zfs performance is history.

I have however no idea what was wrong.
(the hdd tests were all fine)

So an hour ago I just redid the whole thing:
- deleted the usb stick
- installed nas4free again on the stick
- set up user/groups
- formated the hdds for zfs
- set up a virtual device zfs1 with all 6 drives (1 parity raidz)
- set up a pool pool1
- set up dataset dataset1
- created folders
- set up cifs shares
- tested: ~100mb/s read and write [it might get faster with tuning or not. but with the client machine i have here with a low-end 7200rpm drive i wont know)

(note: at first I did two devices/pools/datasets with 3 drives each to compare with the other 3-drive-nas if the performance would be equal then. it was. so I tried 6 drives again and hurray, the performance did not decrease but stayed the same)

i just can not figure out what caused the problems in the first place. especially since I set up two nas4free devices and did exactly the same on both, just like I did an hour ago.

still, thank you for your time and help!
(it did not really help in fixing, but it gave me hope and more importantly, an insight into how to further tune the performance)

Catwiesel

consider this thread closed.

[sodalimon]

Hi there,

Catwiesel, I couldn't help but notice, you've been using 6 drives for a RAIDZ1 array.

If you take a look into the ZFS best practices guide: http://www.solarisinternals.com/wiki/in ... mendations
and this post: viewtopic.php?f=59&t=1494#p6410

I haven't practiced any issues about that... But just to make sure I'm on the right side (and it kind of makes sense too)...

Edit: fixed a typo...

[catwiesel]

as i mentioned before, I am well aware of best practice guides but dont care about how many drives i have in my z1 array and how many you people think i should have. thank you very much

[cancerman]

Ouch! I assume by 'you people', you mean experts. And it's not how many drives we think you should have. It's how many are recommended for better performance. You made a post complaining about performance and you got an answer telling you how to get better performance. Don't abuse free support.

[catwiesel]

Cancerman is right, my reply was out of line.

Thank you for pointing it out to me.

And thank you, sodalimon, to you I should have written
At the time of your post, the issue had been resolved. It had noting to do with the 6 drives in the z1 setup.
And even before I came here to ask for help, I have read the best practices guides.
I am aware of the controversety of 6-drives setup as z1, unfortunately, in this case, it is a requirement, not an option.

Still, thank you for your help!

[ku-gew]

By the way, if you are using desktop class drives, keep in mind that vibrations from nearby hard disks are able to kill over 50% of the performances of the drives.
If two setups differ only by the number of drives and one is getting different performances, maybe it's a vibration problem.

[catwiesel]

as mentioned before, the problem has been resolved by reinstalling nas4free as well as setting the zfs pool up anew.

i do not know what killed the performance, but neither the number of drives, nor the vibrations appear to have been the cause.
the current 110mb/s is more than I expected. Maybe it could be more, but I dont know if the harddrives or the gigabit ethernet are the bottleneck here.
and to be honest, it is a mute point. I wont exchange the drives for faster ones and I can't put a second NIC in the NAS. More importantly however, this device is for data backup and the source data can not even be sent that fast...

by the bye, the case I chose has rubber vibration thingys and cages for the harddrives with which I seperated them 3+3