Kernel Panic in x86 installs (ZFS)

[ste123]

TL/DR: Out-of-the-box ZFS ARC and kmem settings on x86 installs can lead to kernel panics.

I've had issues with a NAS4Free install that was used for ZFS storage and nothing else. It is acting as a backup target.

I previously set up 2 NAS4Free storage boxes in the past, both x64 but an older version. This one was x86 due to lower RAM consumption; the system is a HP ProLiant MicroServer Gen8 with 2GB RAM.

The server would reboot on its own, with no logs to say why it rebooted. I was able to replicate the issue on the MicroServer by performing some disk I/O (rsync transfers), however was not able to catch the error right away. Fortunately, eventually, I was able to record a reboot in iLO and was able to catch the error on screen just prior to the reboot. The error returned was:

Code: Select all

panic: kmem_malloc(16384): kmem_map too small: 906522624 total allocated

Googling the problem led me to a couple of pages:

http://www.1stbyte.com/2011/01/27/tunin ... nel-panic/
https://wiki.freebsd.org/ZFSTuningGuide

From the ZFSTuningGuide:

The value for vfs.zfs.arc_max needs to be smaller than the value for vm.kmem_size (not only ZFS is using the kmem).

After much testing, I have found what I believe to be a configuration issue out of the box for x86 installs. The misconfiguration is still present in the latest release to date (9.3.0.2 rev 1310).

Test configuration (VM):

• RAM 2GB
• HDD 2GB
• Install option 1 "Install 'Embedded' OS/MBR on HDD/SSD/CF/USB (preferred)"
• SWAP size (MB) 768

When x86 install is complete, the following values can be returned from the shell.

Code: Select all

nas4free: ~ # sysctl vm.kmem_size vfs.zfs.arc_min vfs.zfs.arc_max
vm.kmem_size: 1035227136
vfs.zfs.arc_min: 1054310400
vfs.zfs.arc_max: 2108620800

Compared to x64 install.

Code: Select all

nas4free: ~ # sysctl vm.kmem_size vfs.zfs.arc_min vfs.zfs.arc_max
vm.kmem_size: 1019510784
vfs.zfs.arc_min: 89478485
vfs.zfs.arc_max: 268435456

For x86, the out-of-the-box settings vfs.zfs.arc_max is 2GB whereas vm.kmem_size is 1GB, and this is a problem.
For x64, the out-of-the-box settings vfs.zfs.arc_max is 256MB and vm.kmem_size is 1GB, so no problem.

In my case, I resolved it by setting vm.kmem_size to 1024M and the ZFS ARC min/max values to 256M, via NAS4Free's System|Advanced|loader.conf section. After a reboot, it now shows:

Code: Select all

nas4free: ~ # sysctl vm.kmem_size vfs.zfs.arc_min vfs.zfs.arc_max
vm.kmem_size: 1073741824
vfs.zfs.arc_min: 268435456
vfs.zfs.arc_max: 268435456

I also wanted to bring this to peoples' attention as I've seen other threads discussing random reboots, with troubleshooting pointing to bad hardware or bad driver support for hardware that previously worked perfectly elsewhere. It may be true for some of those installs, but it is possible if they were using an x86 install with ZFS defaults and are using ZFS it could be related to this.

My testing shows it has been a problem (based on the sysctl values of the above settings) as far back as version 9.1.0.1 rev 847, which is the earliest version I've tested.

[raulfg3]

I install to my brother N4F in a gen7 Proliant with only 2GB , and works fine for me.

1 - I use x64 install
2 - I use zfskerntune to tune system to 2GB of RAM.
3 - I do not use dedupe

viewtopic.php?f=71&t=1278

[ste123]

Thanks for the response, however:

1 - x64 is not affected, as mentioned; this only affects x86 installs
2 - I haven't used zfskerntune (fresh install), but unless it's mandatory or strongly recommended in setup guides etc it may often be overlooked
3 - I don't use dedupe either (it may cause symptoms to exhibit themselves earlier, but it is not required for this issue to occur)

I believe the hardware used is irrelevant, as the same sysctl values were seen in testing on VMs (VirtualBox on Windows 7).

What I'm trying to highlight is that x86 and x64 sysctl values are so different in fresh installs, it can cause x86 installs of NAS4Free to reboot due to kernel panics. Unfortunately, I don't know how these are configured to suggest a proper fix.

[raulfg3]

ZFs is dessinged/created to work on x64 system.
x86 is a port version that only exist on BSD ( may be true or not, not sure).
If you google a bit x64 is always recommended and x86 discouraged.

SO: if possible for you, use x64, if not possible use x86 and use ZFSkerntune, only can be good, never can be dangerous to your system because ZFSkerntune only apply predefined values to certain variables acording your memory (RAM).

[drnicolas]

May be I have the same problem:

For a longer time I see unexpected reboots in an environment where backup Exec is doing backups to a CIFS on a ZFS dataset.
For a more complete description see my posting in the ZFS forum.

It seems that my setting has also a problem: but I did not change anything. Also, Ithink I have x64

login as: root
root@hp-nas's password:
Last login: Wed Feb 11 18:03:36 2015
Welcome to NAS4Free!
hp-nas: ~ # sysctl vm.kmem_size vfs.zfs.arc_min vfs.zfs.arc_max
vm.kmem_size: 6155399168
vfs.zfs.arc_min: 4205096960
vfs.zfs.arc_max: 8410193920
hp-nas: ~ #

[ste123]

ZFs is dessinged/created to work on x64 system.
x86 is a port version that only exist on BSD ( may be true or not, not sure).
If you google a bit x64 is always recommended and x86 discouraged.

SO: if possible for you, use x64, if not possible use x86 and use ZFSkerntune, only can be good, never can be dangerous to your system because ZFSkerntune only apply predefined values to certain variables acording your memory (RAM).

My main concern is for those that use the x86 version of N4F over x64 is they assume the default sysctl settings provided should be sufficient enough to have a working ZFS setup once pools are created, like me. They will be disappointed when their systems reboot at random with no logs as to why. In my opinion, both x86 and x64 should be set up to have kmem_size larger than zfs.arc_max, as x64 currently has; they don't need to be optimized or tuned. Alternatively, they would be tweaked out of the box based on RAM to similar settings outlined in this oldish post (viewtopic.php?p=21974#p21974), or have ZFSkerntune embedded into N4F. If not, ZFS support may as well be removed, disabled or hidden in x86 installs (at least I don't think it was available for my version, 9.2.0.1.972).

If I recall, x86 installs of N4F are recommended for systems with 2GB RAM, which is why I went down this route for this install.

I have tried Googling for x86 zfs but I primarily saw references to Oracle documentation (I didn't read them, they look like user guides). Can you provide resources/links that state ZFS should not be used on x86? Unfortunately it's a bit late for my install, but maybe it'll deter others...

I currently have possession of 2 older MicroServer N54L boxes with 2GB RAM running x64^ set up for ZFS and work without an issue (apart from performance being on the slow side, which I'm not complaining about). For me, as these are primarily backup boxes, stability is more important than speed.

^ these boxes weren't set up by me; I would probably have opted for x86 then, too

[ste123]

May be I have the same problem:

For a longer time I see unexpected reboots in an environment where backup Exec is doing backups to a CIFS on a ZFS dataset.
For a more complete description see my posting in the ZFS forum.

It seems that my setting has also a problem: but I did not change anything. Also, Ithink I have x64

login as: root
root@hp-nas's password:
Last login: Wed Feb 11 18:03:36 2015
Welcome to NAS4Free!
hp-nas: ~ # sysctl vm.kmem_size vfs.zfs.arc_min vfs.zfs.arc_max
vm.kmem_size: 6155399168
vfs.zfs.arc_min: 4205096960
vfs.zfs.arc_max: 8410193920
hp-nas: ~ #

Yes, this is the same problem (vm.kmem_size less than vfs.zfs.arc_max), however I don't know if it stemmed from the install or not. I guess you could use zfskerntune to set up appropriate values, or do it yourself manually as I had done in my initial post.

You can confirm whether you're using x86 and x64 via the GUI's System Information page (Status|System); look at Platform.

[drnicolas]

Okay. Just upgraded to the latest FW 1349. System is x64.

After doing zfskerntune I get this output:
hp-nas: RRD # sysctl vm.kmem_size vfs.zfs.arc_min vfs.zfs.arc_max
vm.kmem_size: 11274289152
vfs.zfs.arc_min: 9663676416
vfs.zfs.arc_max: 9663676416

For me that sounds sufficient now.

Let's see if the crashes are gone ...


I will report tomorrow or friday ...

[drnicolas]

I am afraid, I have the same problem as before.

Sudden reboots, even after doing the tuning homework.