How to get RAM free from ZFS ARC

[Snufkin]

Did inplace upgrade from two independent GEOM mirrors to one ZFS pool with two mirror vdev's.
Taking into account limited hardware resources (see signature) the result is excellent, no extra CPU load, download/upload speed drop or other side effects.

The only thing I had to face was RAM size, and here is a brief report about success.

• FreeBSD kernel and related processes need at least 1 GB free RAM. Therefore have two sysctl tunables to set

  Code: Select all

  vfs.zfs.arc_max:1G
  vfs.zfs.arc_min:512M

  After a couple of days ZFS ARC balanced its actual working limit with at least 140 MB of free RAM (based on RRD graphs).
• Since vfs.zfs.arc_max is not hard limit but desirable level, expect probable lack of RAM. Got it during upgrade (WebGUI).

  Code: Select all

  nas4free lighttpd[2318]: (chunk.c.581) write() temp-file /var/tmp/lighttpd-upload-hStfpL failed:
  nas4free kernel: pid 2318 (lighttpd), uid 0: exited on signal 11

• Restarted WebGUI with

  Code: Select all

  /etc/rc.d/lighttpd restart

  and tried upgrade again

  Code: Select all

  nas4free lighttpd[6621]: (chunk.c.553) opening temp-file failed: No space left on device

  The reason is clear, in-memory /var/tmp/ is too small.
• Found another ZFS sysctl tunable vfs.zfs.arc_free_target to define how much memory should be left free from ZFS ARC for other processes. It counts in 4096 bit blocks and default NAS4Free value is 3289 (about 13 MB). Not much (140+13=152) especially if I need to get 155 MB firmware image xz file.
• Good news, vfs.zfs.arc_free_target tunable applies to live system and takes immediate effect. So simply raised it up to 128 MB.

  Code: Select all

  sysctl vfs.zfs.arc_free_target:32768

  and went through the upgrade without glitches.

This is ZFS ARC usage graph screenshot to illustrate above steps.

ZFS_ARC_usage.JPG

Legend:

1. ARC releases memory too slow and lighttpd killed
2. vfs.zfs.arc_free_target:32768 (128 MB)
3. Successful upgrade and autoreboot
4. New limit to test vfs.zfs.arc_free_target:16384 (64 MB)

[JoseMR]

Hi, you can tune your ARC setting under: System|Advanced|loader.conf but a reboot is required since this setting go to "loader.conf" unlike sysctl's, also leaving 1~2 GB free from the ARC is good troubleshooting start.

You can also increase var,tmp sizes in System|Advanced|rc.conf or either "Unionfs" to a dedicated UFS formatted disk/partition(not ZFS vol), usually the Data partition the default installer creates is a good option.

However ZFS can behave really aggressive regarding the workload as well, so this is where a small Swap partition becomes in to place to help.

Regards

[Snufkin]

JoseMR, thank you for good points to follow up and make N4F running stable.

The case I've reported is some kind of proof of concept to check whether popular beliefs about ZFS are true ot not.

• Swap is mandatory if RAM is low
• ZFS runs good on high grade hardware only
• ZFS requires a huge amount of memory

And if your server usage profile is stable and predictable you could easily fine tune NAS4Free ZFS setup and not complicate things very much.

[raulfg3]

Swap is mandatory if RAM is low

yes , and always recomended neverless what amount of RAM doi you have ( better not use swap, but must be configured)

ZFS runs good on high grade hardware only

NO , but need x64 capable system

ZFS requires a huge amount of memory

Minimun 4GB, and 8 is recommended ( perhaps someone can use ZFS with only 2GB of RAM but need a lot of tunning to make it stable),

PD: This recomendation is true for normal ZFS, if you use deduplication , you need lots and lots of RAM to work.