Why do i need SWAP with 32Gb of RAM?

[spy0r]

Hi Folks,

i have a simple question. I think i already read the answer at google.com, but i'm not sure:
Why do i need swap, when i have 32Gb of RAM?

I think it is because ZFS "wires" the Memory and doesn't "give it back" to other Processes, so this night, my Samba Service stopped:

Code: Select all

Sep 10 01:00:35 stef-nas4free kernel: pid 11289 (smbd), uid 0, was killed: out of swap space
Sep 10 01:00:35 stef-nas4free kernel: pid 2021 (smbd), uid 0, was killed: out of swap space
Sep 10 01:00:35 stef-nas4free kernel: pid 11291 (smbd), uid 0, was killed: out of swap space
Sep 10 01:00:35 stef-nas4free kernel: pid 2019 (nmbd), uid 0, was killed: out of swap space

Now my question: Am i right? There is no "RAM defragmentation"? Do i need to set a SWAP File to my ZFS Pool like it is described here: viewtopic.php?f=15&p=59041?

Thanks for your answers!

Regards,
Stef

[Parkcomm]

Hey Stef,

The following is oversimplified, but its what happens on my machine.

ZFS does grab all avalaible RAM, subject to a couple of tunables (so you can stop ZFS from grabbing all your RAM). It also returns RAM, but this is based on its own assessment of need rather than demand from other processes. The more memory ZFS uses (for ARC) the faster it performs.

To stop ZFS grabbing all RAM you can configure the following tuneables in loader.conf
vfs.zfs.arc_max
vm.kmem_size (don't play with this)

So if you make vfz.zfs.arc_max to 3G less than your_size, then that memory would not be available to ZFS and can be preserved for the rest of your system and applications.

I'd suggest running the following command to see what you are using now:

Code: Select all

sysctl vfs.zfs.arc_max
sysctl vm.kmem_size

Having said that - I always configure some swap because you might do something in the future that consumes more than 100% of RAM. If you have swap, the machine might slow down to a crawl. If you don't have swap you could crash the machine - and crashing is much more likely to corrupt your precious data.

Ref: https://www.freebsd.org/doc/handbook/zfs-advanced.html

Just a small note - wrt my advice on in the link you mentioned. I don't know if putting swap on your ZFS drive is best practice (obviously do it if you have no other choice) but I would put swap in order of preference:
- an alternative SSD
- an alternative rotational HDD
- my ZFS pool.
(never put swap on a flash drive).

the reason I say this is I can imagine a memory problem might adversely affect ZFS - and then prevent you from using the pool for swap to alleviate the problem.

[spy0r]

Hi,

thanks for your perfect "simple" description. At the moment, sysctl returns the following Values:

Code: Select all

stef-nas4free: ~# sysctl vfs.zfs.arc_max        
vfs.zfs.arc_max: 29984747520                     
stef-nas4free: ~# sysctl vm.kmem_size           
vm.kmem_size: 33384349696

So what would the most professional way be, what would you do in my case?

Step1) Reducing arc_max to for example 25769803776 (24Gb)?
Step2) (Optional) Adding a Small/Fast SSD for SWAP?

Thanks & Greets,
Stef

[Parkcomm]

It all depends on what applications and jails you load:

Lets say you are going to run a pretty standard nas4free with rsync, afp, cifs etc, even bittorrent but not fuppes or DAAP (they can be hogs - I don't know how much RAM they use). And lets say four jails with things like sabnzbd, or plex or owncloud.

That would all fit wthin 4G pretty easily. So 24G vfs.zfs.arc_max would leave heaps of room - no swap needed.

however - I would still configure a 4G swap partition as well, on the best drive you have available already, but I wouldn't bother buying a disk for this.

then monitor the situation

If you click on the system tab it has a line telling you how much swap is being used. I wouldn't worry if say 1G of swap gets used, but be aware as it grows you are eventually going to see a slow down.

I would also install RDDGraphs extension which will report memory used as well as ZFS use. Its really is a useful tool. Its great because it reports a day view, a week view, a month view and a year view, so you can see any long term problems before they arise.

Feel free to increase vfs.zfs.arc_max if its not being used - you do want to maximise the ZFS ARC for performance.

[spy0r]

Thank you so much mate!

I will:
- Decrease the vfs.zfs.arc_max to 24Gb (with the knowledge of increasing it again when its not used)
- Add SWAP File on my ZFS Pool (The only way if i don't want to use flash drives or buy a new disk)
- Add RDDGraphs extension (To monitor if i get in Trouble again with this setting or if a even can increase arc_max again)

I run a pretty standard NAS4FREE with CIFS/RSYNClokal/Webserver with Owncloud (no jails, no extensions [right now]).

Hope with this settings i can finally start my "endurance test".

[Parkcomm]

in that case 24G is very conservative.

[spy0r]

Okay, after step 1 and 3 and copying some data again on the NAS it looks like:

rrd-memory-detailed_daily.png

rrd-arc_daily.png

seems to be fine right now, let's see what we get in the future
I didn't forget to use swap, but i couldn't get my zfs volume to be shown in the swap page on system/advanced right now.

[Parkcomm]

The zvol wont't show - you'll need to type it in (or cut and paste) - /dev/zvol/zpool_name/swap_vol_name

[armandh]

it is [IMHO] NOT the N4F or ZFS
this runs ZFS just fine with a little over 1 Gb RAM

low ram.JPG

[spy0r]

Well, i'm pretty sure you won't have the r/w performance i have with an amount of only 1Gb of RAM. But it may surely work under specific conditions.

What is clear right now is, that no processes had to be stopped due to too less memory after reducing arc_max to 24Gb in my case. I copied about 30Gb of Data to the NAS (fast) and the RAM filled up to 85% (Status System). Then Rsync had to backup those Files to the Backup Disk. This was the usecase, where a process had to be killed. After some time ARC dropped the Memory again, while the NAS was not really used.

Now i have some "inactive" Memory in the Graphs, can somebody explain what that exactly means? If i understand it right "free" memory is the memory you "have never used" and inactive memory is the memory, that has been used, but is not used any more - right?

rrd-memory-detailed_weekly.png

rrd-arc_weekly.png

Thanks & Regards,
stef

[Parkcomm]

If i understand it right "free" memory is the memory you "have never used" and inactive memory is the memory, that has been used, but is not used any more - right?

Exactly - think of inactive like a cache, it'll be used again if its needed, or it can be overwritten by something else.

[spy0r]

Status so far:

rrd-memory-detailed_monthly.png

rrd-arc_monthly.png

Everything seems to be fine, plenty of memory free, but at the moment, there are no big read/write operations done...

[Parkcomm]

You can probably start experimenting with higher values of arc_max, the bigger the ARC the better the performance.

[spy0r]

Yeah, i think i will go to 27917287424 (26Gb) in the next Step. Is there a way to change the value without a reboot?

[Parkcomm]

Nope - it can only be set in loader.conf, which requires a reboot.

[spy0r]

Okay, thanks again.

I:
- switched to 26Gb arc_max
- added a 4Gb SWAP on my zfs pool
- upgraded to revision 1855 had, no, edit: HAVE some smb lib errors)

let's see what happens

[sokolum]

Thank you so much mate!

I will:
- Decrease the vfs.zfs.arc_max to 24Gb (with the knowledge of increasing it again when its not used)
- Add SWAP File on my ZFS Pool (The only way if i don't want to use flash drives or buy a new disk)
- Add RDDGraphs extension (To monitor if i get in Trouble again with this setting or if a even can increase arc_max again)

I run a pretty standard NAS4FREE with CIFS/RSYNClokal/Webserver with Owncloud (no jails, no extensions [right now]).

Hope with this settings i can finally start my "endurance test".

Running here with 4GB with the following services: CIFS/SMB; FTP; Dynamic DNS
And it uses only 20% of that amount of RAM.
usage: home


Mem: 23M Active, 40M Inact, 764M Wired, 24M Buf, 3088M Free

I believe 2GB would be also more than enough in my setup.
It al depends on your setup.

[spy0r]

Hi,

thanks for your reply. I think saying "my NAS uses only xx% of memory" doesn't really matter with ZFS. I have days with 20% ram usage, but also days with about 90% without reboots. Depends on NAS usage or read/write access with zfs, not on the number of processes.

Back to the topic, i removed the zvol swap and added a SSD swap - I should me more than save now - Thanks for all your replys.

Next thing i have to understand are jails

Greets, Stef