SAMBA stalls on big files (2 GB and higher)

[fallen2109]

Dear All,

Saturday I have tried to copy some ISO files (4 GB and bigger) - total size of the transfer 14 GB. I was surprised to see that my file transfer performance goes up and down and finally stalled to 0 KB/s. After some time a message pops-up asking if I want to retry. I have tried multiple time (with deleting the target directory completely) and sometimes I even get warning about duplicates files/folders. Anybody seen such behavior ?
I was then on SMB3 (all settings by default - see my previous topic - viewtopic.php?f=21&t=9902 ), but have tried also NT2 and SMB2 and SAMBA behaves exactly the same when set with the defaults.

The only remedy for me was to set back Asynchronous IO to ON with default 1024 read and write size. When AIO is ON - I get also varying transfer performance and it goes down to 0 KB/s, but at least it does not stall! Crystal-Mark does give decent sequential read and write (110 / 80 MB/s) performance - even on huge 32 GiB file - also tested with AIO on. What worries me is the jigsaw transfer performance of SAMBA - start with 110 MB/s and then goes down and zigzags around 40-50 MB/s, occasionally hitting 0 KB/s, but not stalling.

I am on the latest 10.2.0.2.2223 release as of time of this post.

Any clues or advice will be highly appreciated.

[raulfg3]

Please post how exactly has your SMB config, I copy & paste this from your old post:

I am on the latest NAS4FREE (v10.2.0.2.1962). I am using "Local User"mode, SMB send buffer is 514000 and receive is 1028000 respectively. Large R/W, Async I/O (default 1024 size) and DOS attributes are all enabled. I also have the following Aux parameters:
getwd cache = yes
smb ports = 139

My suggest:
delete getwd cache = yes in auxiliary parameter

and reduce buffer size to 128k or 64k=65536, your actuallt buffers excedd the buffer size of your NIC.

Please post what is your NIC and google about phisicall buffer size of your NIC, better size in N4F is same ( or x2) as your NIC have.

[fallen2109]

In other words - put the buffer size back to default?

The aux parameters:
getwd cache = yes
smb ports = 139

are already removed by me .

[raulfg3]

Yes, first try default values and test stability and performance, sometimes if you try to improve performance , stability goes wrong.

[fallen2109]

Well - believe me I did try with default parameters. It lead me nowhere. Here are the results when all settings are on default (buffers are both set to 128480). See the sequence of events?

1

NAS_SMB_SMALL_01.png

2

NAS_SMB_SMALL_02.png

3

NAS_SMB_SMALL_03.png

When I go back to the big buffer size receive - 1028000 and send 514000 I do hit 0 kB/s, but my transfers complete at least.

1

NAS_SMB_LARGE_01.png

Unfortunately there is a bug or incompatibility in the latest versions of NAS4Fee as every time I change things on the SMB configuration, I do a save
and then reboot - I am getting a page fault and my whole configuration is gone. So after each change in the SAMBA configuration - I make a backup of the configuration, before rebooting and re-installing the complete NAS4free from scratch. A really tedious and slow process, but I do not have any other options. I did my ZFS configuration a year ago and I don't want to loose it by starting from scratch. Also all my shares (10 in total) and other staff like rsync needs to be done from scratch and I can not afford loosing the time.

[raulfg3]

Unfortunately there is a bug or incompatibility in the latest versions of NAS4Fee as every time I change things on the SMB configuration

not sure

Please use a new cat6 cable and try to eliminate other elements from equation ( eg: the switch) = try a cross cable), my suspect is a bad cat5 cable or something simmilar.

SAMBA & N4F are well tested ( of course not all configurations can be tested), but inestable output like you have are simptom of a bad hardware ( bad cable tipically).


PD: Can you execute iperf and post result with defaul values?.

and please read this usefull info: viewtopic.php?f=74&t=7238

[raulfg3]

I am getting a page fault and my whole configuration is gone. So after each change in the SAMBA configuration - I make a backup of the configuration, before rebooting and re-installing the complete NAS4free from scratch. A really tedious and slow process, but I do not have any other options. I did my ZFS configuration a year ago and I don't want to loose it by starting from scratch

.

Ok, plase do this, when you have time to do:

1- save your actual config as config.xml ( no encription)
2 - shutdown your NAS a plug a NEW USBkey
3 - poweron a boot from Cd ( you need latest N4F ISO)
4 - do a embeded install on your new USB key (this copy files from liveCD to new USB)
5 - shutdown
6 - boot from new USBkey ( perhaps you need to configure BIOS to do it), remember to retire your CD from CDROM reader.
7 - enter on webGUI (by default 192.168.1.250)
8 - restore your previosly saved config.
9 - Done, now you have a new embeded install and all your shares / users / rsync jobs are restored.

Repeat copy test using default values and post iperf from this start from scratch boot.


PD: If you do not have CDROM Reader, you can do the same using live USB, in this case you need 2 USB, one with live USB that you only need to use one time to install on the second USB, once installed, you can retire liveUSB, and boot from new USB key.

[fallen2109]

Two questions:

1) I am using LAGG (link aggregation with LACP) over 2 x GigaBit - do you think this might be the culprit of the problem?

2) As I am using a FULL and not EMBEDDED release. Will your procedure work in this case? In other words - can I take the config.xml
out of the EMBEDDED release and restore it in a FULL release? If not - do you have any other suggestions may be?

Thank you for the help so far .

[raulfg3]

2) As I am using a FULL and not EMBEDDED release. Will your procedure work in this case? In other words - can I take the config.xml
out of the EMBEDDED release and restore it in a FULL release? If not - do you have any other suggestions may be?

YES, this is the magic of N4F, unless you REALLY need full instal, embeded is the recomended way, is easy to upgrade to latest version from webGUI.

PD1: Do not format your actual full install, only change BIOS order to boot from new embeded install on USBkey.

If something goes wrong, you only need to change BIOS boot order to select your actual HD and voila your NAS is online again

1) I am using LAGG (link aggregation with LACP) over 2 x GigaBit - do you think this might be the culprit of the problem?

YES, LACP do not double banwith only usefull in small bussines enviroment when several clients access to your NAS in home enviroment to move big ISO films one after other, do not improve performance. Please try whitout LAGG and you notice best stability.

[fallen2109]

YES, this is the magic of N4F, unless you REALLY need full instal, embeded is the recomended way, is easy to upgrade to latest version from webGUI.

PD1: Do not format your actual full install, only change BIOS order to boot from new embeded install on USBkey.

If something goes wrong, you only need to change BIOS boot order to select your actual HD and voila your NAS is online again

As I am using my own scripts and shell aliases - I am not sure embedded will work for me. I used it in the past, but moved back to full release due to the above reasons and the fact that I have also zfskerneltune installed. But I will give it a try. Can I use the config.xml from the embedded install for setting up a FULL install later on?

[fallen2109]

Also SSH password-less login is not possible as /root is constantly being refreshed .

[pokeyboy]

fallen2109

I've had exactly these same issues as you, with the various N4F boxes I've built. I can't really say I have found the exact cause or solution to the problem. I've probably just found some "work around" to get things going.

I had stalling file transfers on two separate N4F units. One unit had a Pentium 4 CPU with Hyper Threading. For some reason, which I can't recall, I turned off HT in the bios, and to my surprise, no more stalling.

The second unit, which I'm currently using, has a dual core Pentium D CPU. I turned off one of the cores in the bios and it fixed that problem there as well.

My trouble shooting has left me with this configuration:
I dropped back to 9.2.0.1 - Shigawire (revision 972) from 10.x.x.x

CIFS/SMB|Settings:
Max Protocol = SMB2
Send Buffer Size = 0
Receive Buffer Size = 0
Large read/write = Enabled
Use sendfile = Disabled
Asynchronous I/O (AIO) = Disabled

I'm pretty much a novice at this NAS stuff and I'm not really sure these settings are appropriate. However, they seem to work for me and my systems are pretty fast and stable.

[fallen2109]

Thanks for the tip - I will try if I have hyper-threading (CPU is INTEL Quad Core 9650) and if it helps if I switch it off.

I don't mind falling back to 9.2.0.1, but my current configuration will be probably incompatible with it - so I am again facing
the problem of finding time making the whole config from scratch and I am worried that I will loose all my ZFS data and have to restore
it from scratch. Also fallback to earlier versions of ZFS means it will probably render my data unreadable - even if I recover
the ZFS configuration .

We will see how this goes - it requires a lot of time and I do not have it right now with Christmas approaching and my parents coming
on a visit in 2 days . At least I got it now writing across the network without aborts.

[pokeyboy]

I would like to point out an observation I've made with my testing and experimenting.

If I read a large file from the NAS (>Gigabyte), transfer speed will average about 74 Mbit/s. If I then re-read the same file from the NAS, but save to a different location on the PC, the transfer speed goes up to 90/100 Mbit/s.

If I alternately read different files, then transfer speed remains at about 74 Mbit/s. It seems like a file will remain cached in memory until new data pushes it out.

Until I had this figured out, it really had me confused as I was troubleshooting and making changes.

I think this symptom also shows up on a write to the NAS. However, my write transfer speed is always over 100 Mbit/s, so not much change.

[fallen2109]

Further testing done last night with the following settings:
- SMB - No AIO and rcv/snd buffers set to 0, la
- Network - Plain Ethernet (no LAGG) em0 as LAN and em1 as OPT1 under different IP addresses
- Write tests conducted:
a) 15.7 GB of ISO file to a Physical HDD (WD Red 6 TB which I use as a reserve copy of all my data) formatted as GPT Partition with UFS file system
b) Same as above, but writing to raidz1 ZFS pool consisting of 2 WD Reds (6 TB each) and one SEAGATE BARRACUDA (3 TB). The reason for the difference in the HDD model and size is the fact that I am in a middle of storage upgrade and WD Reds are very expensive (+300 EUR a piece)

While writing to the Single WD Red HDD went at 117 MB/s CONSTANT SPEED (very impressive) - I CAN NOT SAY THE SAME WAS TRUE for the ZFS Pool.

I am afraid I have to focus on my ZFS pool as it's performance is inconsistent (going from 112 to 0 MB/s and even completely stalling). The culprit for my problems was (and still is for the second point):
1) Usage of LAGG (LACP) instead of plain Ethernet - speed reduction from 117 MB/s (constantly) to an average speed of 70-80 MB/s (floating up and down around these values).

2) ZFS pool tuning - can someone let me know how can I measure the isolated performance of this one? I do scrub the pool regularly (at least once a week - this week I have already scrubbed 3 times). The pool is about 60-65 % full (hence the storage upgrade I am carrying out ) - can this have such detrimental effect on the performance ?

Will appreciate any advice - please let me know if I should move my topic to the ZFS related parts of this forum .

[pokeyboy]

fallen2109

It sounds like you've made some good progress and narrowed the problem down. I can't really be of any help to you, as I don't use ZFS. Hopefully, someone out there can give some tips.

Just out of curiosity, when you tested with a single Red, formatted as GPT Partition with UFS, how did write vs read speeds compare? This is more for my own information.

I'm interested to hear what your eventual solution will be, so keep us informed.

[daoyama]

Also SSH password-less login is not possible as /root is constantly being refreshed .

Personally, I don't like root access via ssh. But if you need SSH key, please write feature request on SF:
https://sourceforge.net/p/nas4free/feature-requests/

[fallen2109]

fallen2109

It sounds like you've made some good progress and narrowed the problem down. I can't really be of any help to you, as I don't use ZFS. Hopefully, someone out there can give some tips.

Just out of curiosity, when you tested with a single Red, formatted as GPT Partition with UFS, how did write vs read speeds compare? This is more for my own information.

I'm interested to hear what your eventual solution will be, so keep us informed.

I have the single WD Red HDD permanently configured as my BACKUP disk (I do not trust fully ZFS even in raidz1 mode).
Here are the test results in 4 scenarios:

1) MTU 8896, AIO ON, Send Buffer 512 KB, Rcv Buffer 1 MB

WD_RED_R_8896_1st_FLOOR.png

2) MTU 1500, AIO ON, Send Buffer 512 KB, Rcv Buffer 1 MB

WD_RED_R_1500_1st_FLOOR.png

3) MTU 8896, AIO OFF, Send Buffer 0 KB, Rcv Buffer 0 KB

WD_RED_R_8896_1st_FLOOR_NO_AIO_BUFF_0.png

4) MTU 1500, AIO OFF, Send Buffer 0 KB, Rcv Buffer 0 KB

WD_RED_R_1500_1st_FLOOR_NO_AIO_BUFF_0.png

As you can see - the best (from speed perspective) and most stable results are when MTU is at it's minimum (1500) , AIO is OFF and send/receive buffers are set to 0.

The test is done from a second switch located on my ground floor. There is a LAGG Fiber optic connection (2 x 1 GBit) between this switch and the one my NAS is attached to. Tomorrow (if time allows me) I will do the same 4 tests from the same switch that the NAS is attached (thus eliminating the influence (if any) of the LAGG connection) and will post the results.

One final question - is there a dedicated ZFS thread in this forum - I searched for it, but was not able to find it.

Thanks you so far for the information.

[pokeyboy]

Very interesting results. Your "best case" settings are the same as mine.

For Max Protocol, are you using SMB2 or SMB3? When I was troubleshooting, I never compared the two because SMB3 is not an option in 9.2.0.1 - Shigawire (revision 972, which I'm using.

It will be interesting to see if the LAGG Fiber optic connection is affecting anything.

[fallen2109]

Oops - forgot to mention. I always used SMB3 (due to the recommendation for Windows 10 compatibility). Do you want me to try SMB2 instead in all 4 scenarios above, before moving to the same switch testing?

[fallen2109]

Well here are the results I got so far (all test are from the same location on the ground floor - thus over the LAGG connection):

1) SMB2, Rcv/Snd buffers to 0, MTU 1500 - I am getting almost flat line (minor variations - +/- 1MB/s) at 99 MB/s. It seems a bit (2-3% faster than SMB3

2) SMB2, Rcv/Snd buffers to 128480 (Samba default , MTU 1500 - I am getting almost flat line (minor variations - +/- 1MB/s) at 98 MB/s. Look like buffers set to 0 really delivers the best result with single disk

3) Out of curiosity - NT1, Rcv/Snd buffers to 0, MTU 1500 - delivers almost a flat line performance at 102 MB /s - this is 6 % increase compared to SMB3. See the graphic below.

Alternative tests with AIO ON - did not deliver any significant differences - so I believe AIO should be off - at least in this scenario (UFS on a single disk GPT). Do you know by any chance what are the differences between the 3 SAMBA protocols - what I will be trading off if I stick with NT1 ?

WD_RED_R_1500_1st_FLOOR_NO_AIO_BUFF_0_NT1.png

[pokeyboy]

I'm sorry to say, I don't have the expertise to comment on the trade offs between the various SMB protocols. However, after seeing your results, using NT1, I decided to give it a try. Unfortunately, the transfer speed took a big hit. I lost about 12-15 Mbit/s. So back to SMB2.

I don't mean to hijack your thread but this is related. The system that I've been tweaking has 2 identical Seagate drives, not in a raid configuration. The log file has been showing a nag that a drive firmware update was available. So this morning I decided to go ahead and do the update. The update went fine but it looks like I lost about 8 Mbit/s in write speed. It dropped from about 104 to 96, which is still ok for me.

The challenging part of this tweaking is the fact that there are so many variables. It seems like just when I think I've got a good handle on things, something changes.

When you finally settle on your optimum settings, I'm curious to see how well your read and write performance compare. I'm seeing some differences.

[fallen2109]

I am busy with ZFS for 2 days now, but I have not made any progress (in fact - I might have even regressed). First I switched off the compression - which made the things worse. Then I decided to switch off the sync write on one of the pool data sets and I am facing now 10 MByte/s write performance. I really don't know what to do - except may be to delete and recreate the ZFS pool from scratch. May be my File system is screwed.

[fallen2109]

Finally a breakthrough:
1) After a lot of tests - I have removed the BBU unit and got decent performance (150+ MB / s ) from a single disk
2) Further investigation showed that port 2 on my 3ware 9690SA is giving CRC errors and even when it works - performs only with 20 MB/s. When did it fail - I really don't know, but I have recreated the ZFS pool using ports 0, 1 and 3 and it seems blazing fast (310+ MB/s in raidz1 and 460+ MB in stripe).

Apparently old hardware tends to break - no matter who is the vendor - a reputable company or a crappy Chinese one.

I am thinking about switching to LSI-92008e controller. I do not need the RAID function anyway. Anyone to have a better proposition?
I believe LSI-9200 is well supported by BSD 10.

Happy and Healthy New 2016 year to everyone!

[pokeyboy]

Glad to hear you made a breakthrough. Those transfer speeds look awesome! This may be a dumb question but what is the "BBU unit" and did it affect the performance? Battery Backup Unit????

Did you finalize your CIFS/SMB|Settings? If so, what did you end up with?

A Happy New Year to you as well.

[fallen2109]

Yep - BBU means Battery Backup Unit.

It seems now that with our without AIO - CIFS performance is awesome - 105 MB/s at minimum. Only on massive transfers - I am getting drops to 80 MB/s and then back to 105-110. I really can not determine what is better. Sticking with the default settings (no AIO and buffers to 102480) I believe should be the wisest thing to do.

[noclaf]

In my case turning off AIO increased the performance ~3x. So having it off is somthing I can recommend too.
Based on my experience the best thing with SAMBA setings in N4F is to change one setting at time and test the influence.

[pokeyboy]

fallen2109

Maybe I've been expecting too much from the CIFS performance. It would be nice to see a flat line at 100+ Mbits on every transfer. That just doesn't seem to be the case. There seems to be some variability and things are a bit temperamental. Maybe it's my hardware, which isn't the latest and greatest.

I have two N4F setups. One is a primary and the other is a backup, which I have set up to do a rsync. One of the boxes seems to have better read performance vs write performance. The other box is just the opposite. Also, one box has two identical Seagate HDDs. Performance from one of the drives seems slightly better than the other. As far as I can tell, there is nothing wrong with the drive.

Overall, my systems are running pretty well, so I think I will leave as is for now.

[fallen2109]

Switched to totally new hardware - Supermicro X7 board with Dual Xeon 5640L (24 cores in total) and 48 GB of memory with LSI 9211 SAS controller in IT mode (former DELL Perc H200 which I re-flashed with the latest original LSI IT firmware). I am very happy with my new platform. Much more reliable and uses only 100 Watts of power on average. Performance tests still to follow .