[SOLVED] Running on i386 with 4GB RAM (hint: don't)

[tuaris]

I know that running ZFS on a 32-bit system is not-recommended and in fact very bad. Unfortunately I have no choice but to use a 32-bit system at the moment (at least temporarily), and indeed I am getting kernel panics when the utilization on the filesystem goes up. (at least I think they are kernel panics, the system just reboots and nothing shows in remote logging).

The Tuning guide (https://wiki.freebsd.org/ZFSTuningGuide#i386) says I should increase the kmem_size, but it looks like NAS4Free has already done that:

Code: Select all

# sysctl vm.kmem_size
vm.kmem_size: 1342177280
# sysctl vm.kmem_size_max
vm.kmem_size_max: 1342177280

The guide also has a sample script that will summarize kernel memory utilization, but it does not work on NAS4Free since there is no dc command installed.

How can I verify that the server is running out of kernel memory and what else can I do to stabilize the system until I can get proper hardware?

[raulfg3]

use ZFSkerntune: viewtopic.php?f=71&t=1278


viewtopic.php?f=78&t=8338&p=50591

[tuaris]

use ZFSkerntune: viewtopic.php?f=71&t=1278


viewtopic.php?f=78&t=8338&p=50591

So far so good. I used the tuning extension, set it to a conservative 1GB left pre-fetch on. I have a 32 GB SSD as the L2ARC.
Values look okay after rebooting

Code: Select all

 sysctl vm.kmem_size vfs.zfs.arc_min vfs.zfs.arc_max
vm.kmem_size: 536870912
vfs.zfs.arc_min: 134217728
vfs.zfs.arc_max: 134217728

Stressed the server as much as I could and no kernel panic yet.
Would be great if that resolves issue, even though 64-bit hardware is on the way, this would make a good backup server.

[tuaris]

Sorry to say that tuning has not resolved the issue.

Solution: Use 64-bit hardware with ZFS because ZFS don't like 32-bit.

[raulfg3]

sorry to know,

[erico.bettoni]

It should work! Try disabling prefetch and remove ssd. I used zfs in x86 for a long time, with 768mb memory.
My guess would the the l2arc. Since you're going to use the ssd on the new box this could still be a good backup box.

[tuaris]

It should work! Try disabling prefetch and remove ssd. I used zfs in x86 for a long time, with 768mb memory.
Mu guess would the the l2arc. Since you're going to use the ssd on the new box this could still be a good backup box.

Okay, set it to 1 GB and disabled the prefetch. Also pulled out the SSD. Lets see what happens.

[tuaris]

It should work! Try disabling prefetch and remove ssd. I used zfs in x86 for a long time, with 768mb memory.
Mu guess would the the l2arc. Since you're going to use the ssd on the new box this could still be a good backup box.

Okay, set it to 1 GB and disabled the prefetch. Also pulled out the SSD. Lets see what happens.

Sorry to say it's still kernel panics.
Maybe I will try with 512 MB instead of 1 GB.

One this to keep in mind is that this is acting as an NFS server, and it's quite busy. Mostly reads. The graphs section shows a constant 40 MB/s read rate with peaks up to 50 MB/s. Writes remain in the 12 to 20 MB/s range, also constant.

Also like to point out that the kernel panic are more frequent when the NFS server is really busy.

[erico.bettoni]

Could be the memory or the mobo. Try creating on share using UFS instead of ZFS just to be sure.
Also, try memtest to see if you memory is ok.

[tuaris]

Could be the memory or the mobo. Try creating on share using UFS instead of ZFS just to be sure.
Also, try memtest to see if you memory is ok.

I used to have this hardware with UFS in NAS4Free 9.2.x (with different HBA and drives). Had issues with the drives dropping off due to me using WD Green Drives and a embedded SATA controller, but never kernel panics.

I decided to give the hardware another chance with 9.3 and put in a new HBA, ARC-1110 and drives, WD Enterprise WD2003FYPS-27Y2B R001.

Going to try UFS once I migrate this data.

[tuaris]

Here is an interesting observation... the less memory I set in the tuner, the more often the system kernel panics.
It also kernel panics less when I leave the prefetch enabled.

I currently have it set to 2GB+ and prefetch enabled. The system has been without a kernel panic for 1 hour 25 minutes. on moderate usage.
Where as if I had it set to 512 MB, and disable prefetch, the system would kernel panic roughly within 20 minutes, on moderate usage.

[tuaris]

So far no kernel panic

Code: Select all

9:19AM  up 17:38, 2 users, load averages: 0.22, 0.28, 0.30

However, I did already migrate most of the data to a different server. So it may be due to the lighter load.

[tuaris]

The server has continued to kernel panic even with a light load, just not as often as it was with high load. I have since migrated to 64-bit hardware and had no issues with that.
I took this old 32-bit server, setup hardware RAID, and reformatted it to use UFS. It's now a pretty good backups storage host.

[raulfg3]

sorry to know.