runaway RAM usage with cifs

[rocket]

Hey there,

I recently upgraded to 11.1.0.4 (rev 5127) and am seeing very high RAM usage with cifs every time I try reading and writing files from my macintosh (10.12)

at first I thought it was just spotlight indexing that was causing the problem by opening large numbers of files for analysis, but I have seen it when doing slow processing on 10-20gb files over the course of several hours (image processing on 10gb mp4 files, so it was long, slow reads).

here is the last day's memory usage:

Screen Shot 2018-03-10 at 2.15.40 PM.jpg

Fri 16:00 I mounted a cifs share from my mac and it started spotlight indexing. which led to maxing RAM, filling swap, and crashing the server.
Fri 22:00 is a reboot to tweak some settings
Sat 08:00 I mounted a cifs share again from my mac and it started spotlight indexing. I then turned spotlight off on the volume at 08:30 or so.
Sat 09:00 I started doing a file analysis on three 10gb files from my mac and you can see it maxed RAM, then started filling up swap around 13:00
Sat 14:00 I unmounted the drive, swap and RAM both return to normal.

the only log entries I found were:

Code: Select all

Mar 10 13:51:29	navis	kernel: swap_pager_getswapspace(1): failed
Mar 10 13:51:29	navis	kernel: swap_pager: out of swap space

is this a known issue with the latest N4F build? I never had this problem using AFP, but AFP is being discontinued by apple.
a pretty thorough search in the forums over the last year hasn't shown anything similar, anyone have any ideas? I guess I can always switch over to NFS, but then if I have a network issue, my local mac could hang.

[rocket]

to quote from the 'release build' forum:

Recently started having issues and happened to notice Samba was using 12GB+ of RAM. Found this post https://forums.freenas.org/index.php?th ... -ram.61039 and bug report https://redmine.ixsystems.com/issues/28585.

Adding

Code: Select all

unix extensions = no
wide links = yes

as additional parameters appears to have fixed it now as a workaround until they find more about the root cause.

Code: Select all

unix extensions = no
wide links = yes

appears to have fixed my problem.

think this is a bug in freeBSD, or are we sharing smbd code with the freeNas folks?

[rongway83]

Just adding on to the forum notes, I am currently having issues with CIFS transfers and swap/memory usage.

Current version 11.2.0.4 5975

24gb ram, kernel set to 24gb, zfs max 20g, zfs min 2g

Plex and OBI extensions installed

I had previously thought it was an issue with my extensions/VM's. I removed the test VM and uninstalled the nextcloud extension. Plex would still crash and the server would start slowing down after the swap started getting used. Further monitoring of the logs showed it running out of memory then shutting down. I tried tuning the ZFS to use less memory but it didn't make a difference on the swap usage. I found this post with the recommended settings for samba and have modified. Hopefully this will resolve my issue, I have also scheduled an upgrade to the latest version, i typically only touch the NAS os if there is an issue.

uses: 3 windows PC's with network mapped drives to the shares and plex. Noticed it appears to be samba related as it happened more frequently during the backup of all datasets.

[Onichan]

I don't adjust kernal RAM, but I do adjust ZFS. My personal experience with disabling swap and making sure not to run out of RAM was samba uses more than you would think as well as other system processes. Also ZFS does not release RAM quick enough for the shitty linux OOM process killer. So even if you have min set really low it doesn't really help. My usage is

VM: set to 2GB, but uses ~2.2GB
Samba: multiple processes totaling ~800MB
php-cgi: multiple processes totaling ~900MB
Then other various system processes: not sure, but I would leave 1-2GB for them

Meaning that's 5-6GB below system RAM I need to set zfs max.

I have 32GB of RAM and set vfs.zfs.arc_max to 25GB and vfs.zfs.arc_min to 20GB without any problems. That's actually lower than needed, but I was having some issues a while ago and lowered the max and just never upped it again. I think I had max set to 27GB a while ago that was working just fine until an unrelated problem.

[rongway83]

Perfect thanks for the follow up! I also found the PLEX dlna service was consuming up to 1.5gb! of ram for a single tv connected, I've disabled that which freed up some space as well. Since adding the suggested wide links and unix modification I've noticed swap usage is down to 0. I did not manually specify the kernel that was just what the variable reported running at. I've left it tuned to 18g max 2g min as it seems plex can consume up to 4gb of ram during transcodes. Stable as of now with no code upgrade so I'd consider this work around successful.