NAS4Free hangs on Mounting ZFS [1/8]

[blex]

My NAS4 Free hangs yesterday.
There was a message telling me out of swap space. No WebGui access, no shel. All frozen up.

I then rebootet the system. It bootet Up and when it was on the point of

Code: Select all

Mounting ZFS filesystems: (7/8) 

it tells me out of swap space and was frozen again.
So I rebootet the system this morging (5 houres ago) again and now it is stuck at

Code: Select all

Mounting ZFS filesystems: (1/8)

The HDD LED is always on. No flashing.

The system has 4x 8TB SATA Drives configured as ZFS with RAID-Z. One SSD with OS and a SWAP partion and a second ssd as ZIL cache.
A Quadcore CPU with 4 Cores and HT also 32GB RAM.

What will be a good recovery option. Boot the system from LiveCD do then what?

I would apreciade any hint.

[blex]

After waiting about 6h I got the following error:

Code: Select all

Mounting ZFS filesystems: pid 1200 (zfs), uid 0, was killed: out of swap space

any Idear how to fix this?

[blex]

Okay I did now an update from 10.2 to 10.3 since I have read that there is a bug in 10.2 with swap space and zfs.

I'm now back at the point where the System waits for

Code: Select all

Mounting ZFA filesystems: (1/8)

My guess is it will fail again.

What would be the best way to start the nas4free Installation, WITHOUT it mounting ZFS on boot so I can examine the SWAP space and the zpool status?

[blex]

Hi, I will update again my report.
The Upgrade tot 10.3 (last nfs4free) didn't help ether.

Code: Select all

Mounting ZFS filesystems: pid 1215 (zfs), uid 0, was killed: out of swap space
pid 1204 (sh), uid 0, was killed: out of swap space
pid 20: (sh), uid 0, was killed out of swap space

So still no booting nas4free System and no data access.

Clould please someone adivce how to get the systme booted to check the swapspace? I can also add / change the swap ssd to a bigger one but question is how. Help would be very welcome.

[blex]

Okay next day next try....

I put in a new systemdrive installed 10.3 nas4free without the zfs drives attached.
I installed with a swap partion of 64GB white the system has 32GB.

If I ran swapinfo I can see the swap /dev/ada0p3 is online and hat a size of 64GB. Usage 0kb.

In the WebGui I see the following:
Device: /dev/ada4p3 | Total: 64G | Used: 0B | Free: 64G

okay now I go to the shel an typed in:

zpool import
zpool import Pool1

this commands takes about 1.5h and fills the ram up to 100%.
Richt after that point I see:

swap_pager: I/O error - pageout failed; blkno XXXX, size 24576, error 12
swap_pager: I/O error - pageout failed; blkno XXXX, size 24576, error 12
swap_pager: I/O error - pageout failed; blkno XXXX, size 24576, error 12
swap_pager: I/O error - pageout failed; blkno XXXX, size 24576, error 12
...

So questions are
- why is the system filling up the ram -> guess of dedup. (22TB ZFS Pool, 6,1TB Used)
- why is the swap space not used?

[alexey123]

Look like you have broken pool.
Connect your disks and check its with SMART

[substr]

The error 12 is likely ENOMEM, Cannot allocate memory.

Do you have any custom tuning for ZFS or kernel parameters?

[blex]

No I have not changed anything on the zfs or kernel paremter. Clean nas4free.

Disks also look okay on smart.

Maybe the ram is not enought because of dedup.

[substr]

How much of the 6.1TB is deduped? Are these ZVOLs, or anything with small block sizes? It is possible you are very tight on RAM. Dedup uses something like 320 bytes per block.

Perhaps you can set vfs.zfs.arc_max=5000000000 to hopefully give plenty of room for the DDT table and everything else.

[blex]

The Dedup should be on 3TB max and only on some volumes.

[substr]

Also try setting vfs.zfs.dedup.prefetch=0

then import. It may be slower that way, but could also reduce memory demand.

[substr]

Also try vfs.zfs.free_max_blocks=100000

[substr]

Try using 25000000000 (25GB)for vfs.zfs.arc_max instead of just 5GB. I think the DDT is part of the ARC, not separate from it.

I think progress is being made each attempt, so do let it run as long as it wants between crashes.

[msilveira]

Hey blex, don't leave us in the dark!
Did you manage to get it back online?
( This is the sort of thread I watch to learn )