N4F resets/crashes all of the sudden. I am lost. Need help!!

[Buddy]

Hi folks,

on my side, n4f is not running stable at all and I am totally frustrated. The issue is as follows.

When writin bigger files (sometimes only some hundreds of Megs, sometimes some Gigs), n4f resets completely!
This means it reboots all of the sudden. As I have a couple of VM which are connected via iscsi and use the samba
service and also tried nfs, there could be some load on the system.

What I tried:

- Reduce MTU -> no luck
- Bought a new Intel Nic dual card -> no luck
- Switched off async in cifs. -> some luck? At least I could write at a speed of 70MB/s from windows to it (more than 8GB). No crash.
- Run memtest overnight. -> no luck
- NFS writes of bigger files seem to reliable crash the system.

The NFS transfer speed up to the reset is very erratic! It drops nearly to 0 and then up to 100MB/s of speed (graph overview). This
behaviour continues till the reset.

I have the feeling that the system is more stable when I do tests. This means that during this period there are not much clients connected.
When all clients connected, it seems to be more likely to crash.

The system:

9.2.0.1 - Shigawire (revision 972)
FreeBSD 9.2-RELEASE-p4 (kern.osreldate: 902001)
x64-embedded on Intel(R) Core(TM) i5-3470S CPU @ 2.90GHz
ASUSTeK COMPUTER INC. P8H61-I LX R2.0
American Megatrends Inc. version: 0702 10/08/2012
RAM: 8GB
ZFS based on Geli

ZFS is configured with the ZFS tune script.

Any hint is very very welcome!

[crowi]

can you post the dmesg output here?
and possibly a screenshot of your CIFS config page

[Buddy]

Hi,

dmesg shows only a ton of line like

g_dev_taste: make_dev_p() failed (gp->name=zvol/pool/.../...@auto-20131006-010000s1, error=63)

interestingly they are only happening on the zfs volumes (zvol/...)

[b0ssman]

the obvious question

have you run a memtest?

the g_dev_taste seems to be an error from the geom system.
are you running the vdev using partitions?

[Buddy]

yep, sry, updated the post...

[Buddy]

I do not know what you mean with vdev?

I am running n4f embedded and encrypted the whole disks.
Any more hints?

[Buddy]

Update:

It happened right now again. Now with access to a mounted cifs share. On this share there are ISOs which one of them has been mounted by Virtualbox for installation. And that cracked n4f!
When I do tests with nearly no client, it seems to work. But it looks like that the system becomes unstable when lots of clients and different services are in use and at the same time a heavy
read or write is happening. Very frustrating.

Any hint what I can do? I think I will try to switch of the "Enable tuning of some kernel variables ". Maybe that will help. Don't know.

Is there the possibility that the non-embedded version would behave different?

[b0ssman]

you could be running out of memory if you have configured zfs for 8gb and run a lot of services + virtualbox

[crowi]

Is there the possibility that the non-embedded version would behave different?

The embedded needs some RAM for the RAM disk it is running from, about 250 MB, the full installation therefore consumes less RAM.
That is the only difference in regards of behaviour.

[Buddy]

I have checked the settings with siftu. They shoudl be OK. RAM never goes over 60%.
Settings are

Code: Select all

vfs.zfs.arc_max  	                3072M
vfs.zfs.arc_min  	                3072M
vfs.zfs.no_write_throttle  	      0
vfs.zfs.prefetch_disable  	       1
vfs.zfs.txg.timeout  	            5
vfs.zfs.vdev.max_pending  	       10
vfs.zfs.vdev.min_pending  	       4
vfs.zfs.write_limit_override  	   0
vm.kmem_size  	                   4608M

Could it be that the embedded system somehow rans out of memory?
What I also notice is a very very erratic transfer speed with NFS. It gos up to 100MB/s and down to 0.5MB/s (nearly stalling) then up again etc....

Just to clarify. VB has not been run on n4f. I ran on another box. Just the ISO has been mounted via cifs from n4f.

[crowi]

Can you post the SMART values of your drives, please?

[Buddy]

Here they are...

Drive 1:

Code: Select all

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   181   177   021    Pre-fail  Always       -       5925
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       37
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   086   086   000    Old_age   Always       -       10676
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       36
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       14
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       22
194 Temperature_Celsius     0x0022   116   110   000    Old_age   Always       -       34
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       1
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

Drive2:

Code: Select all

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   178   174   021    Pre-fail  Always       -       6058
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       36
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   086   086   000    Old_age   Always       -       10676
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       35
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       13
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       22
194 Temperature_Celsius     0x0022   116   110   000    Old_age   Always       -       34
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

[b0ssman]

8gb ram might not be enough in your case.

the recommended amount is about 1gb per 1tb.

how much storage do you have?

also why is your prefetch turned off?
vfs.zfs.prefetch_disable 1

[Buddy]

It's only 2x3TB WD Red disks in mirror.

Regarding the disabling of the prefetch, I don't know. Must have been me a longer time ago playing with the parameters to get rid of the problem.
But good hint. I will reenable it.

I never saw after the tuning with siftu the RAM going higher than 60%-70%.
Do you really think 8GB are not enough. OK, I can try another 8GB bar.
Already have invested in the intel nic so this should be no problem.
But is the system really that sensitive to RAM that it will reboot all of the sudden? This does not sound
very nice as such systems will go to its resouce limits anyway over time and will have to be extended.
A slowing down with notification would be acceptable but a crash is definitely not.

[b0ssman]

hmm 8gb for 3 tb storage should be enough. (unless you have dedupe turned on)

there are some bios updates that say improve system stability for your motherboard.

can you state exactly what services are running.

also whatch the memory graph with the next nfs transfer.

[Buddy]

Also a good hint. The bios version 0702 is quite outdated. And between the current version and this one there
are lots of hints about improving stability. Will install latest version "P8H61-I-LX-R2 BIOS 1105" this evening.

Any other ideas?

Activities planned for this evening:

- enable prefetch
- enable pmtu discovery
- update bios

Services running are:

Code: Select all

CIFS/SMB
        6 shares
        Enable large read/write: off
        Enable use sendfile.: off
        Enable extended attribute support: off
        Enable Asynchronous I/O (AIO): off
        Auxiliary parameters:
                getwd cache = yes
                socket options = TCP_NODELAY IPTOS_LOWDELAY SO_KEEPALIVE
        
SSH

NFS
        4 shares
        Number of servers: 4

RSYNC
        3 modules

iSCSI Target
        23 extends
        23 targets
        Max. sessions: 256
        Max. connections: 4

Webserver

[Buddy]

I am really lost! I do not get a grip on this system. I have done all the settings given above, but no luck.
Just experienced a crash again while moving files in a cifs dir.

[b0ssman]

well it might be time to think about setting it up from scratch or abandoning the system.

unstable hardware could be anything. from psu, to cpu to motherboard.

[Buddy]

that is what I am up to. I have no other chance. I will buy two new disks and do a new setup. Btw will nas4free properly set
the sector size on geli devices if using 4k drives? It should be created with 4k sector size also, right?

[b0ssman]

i dont think it will from the gui,
however if you create the geli devices manually you can.

[Buddy]

I am getting the feeling that the slow performance the tons of this error messages

Code: Select all

g_dev_taste: make_dev_p() failed (gp->name=zvol/pool/.../...@auto-20131006-010000s1, error=63)

is the cause. It takes ages to import the pool and to show the snapshots. I have about 40 datasets, and 25 zvols
and about 9000 snapshots. It takes ages to import the pool and to list the snapshots. It should not, should it?
During import it fires tons of the error given above. Maybe this is also a cause of the crash?
Any clue what it could be?

[mcvillage]

I have a similar setup. One pool who shares a part with nfs to use with virtual machines, a part for cifs, rsync for backup, etc. I had problems with crashes and reboots until I made some changes. Went to full install and enabled swap. With high loads the system shuts full services when it runs out of memory. Increase your RAM and enable a swap partition at least 4 times the amount of RAM, do not use a zvol for swap but a slice. Hope this helps, my system now runs 24/7 for about 18 months now.

[Buddy]

Hi mcvillage,

thanks for that information. These are good hints to follow if the problem is happening again.

For now I got rid of the errors and it looks like (only looks, will see during next week) that the errors above also were the cause of the
crashes. I am hoping they were. I got the solution from the freenas forum as I was so frustrated that I switched the system to freenas
which showed the same behaviour. Luckily the systems are quite interchangable in their
current releases. eraser gave the solution. The reasen is that the device names of the zvols are too long! I was not
aware of any limitation there. So I renamed the vols and reconfigured iscsi: Errors gone. See thread

http://forums.freenas.org/index.php?thr ... r-6.21675/

I will not close this thread as of now as I would like to proof that is was the cause.

[Buddy]

Hi guys,

I finally solved all my problems. I do not have any resets/crashes any more and have a stable
system. I am sorry to say, but I solved it with migrating to Freenas. In this case I can
only confirm the experiences of Ivan in his blog

http://ivoras.net/blog/tree/2013-02-22. ... 4free.html

Being 2 different systems, the migration went quite smooth and I was even able to switch between the
versions after having imported it in freenas, though, I had to use command line to transfer from one pool
to another. It even worked out with encryption. The only thing you have to keep in mind is to generate
a key and ad it with password in slot 1 of geli. Then Freenas is able to unlock the encrypted n4f disks.
I also experienced a much less CPU consumption in Freenas than in n4f. When copying large data,
the CPU spiked up to 70% and very often around 40%. In Freenas I never had more than a 25% spiking
till now.

Thanks for the help so far....

[raulfg3]

no problem , the most important lesson from this post is that finally you meet a stable OS for your NAS

[STAMSTER]

Something is causing kernel panic... perhaps your hardware is not fully compatibile with this version of BSD kernel.

I had similar issue with PCI SATA/IDE controller based on VIA chipset. With IDE drives it would work fine, but with almost any SATA drive(s) I put onto the card, at some point during file transfer it would hang the whole machine, no reboot, no signs of life... that VIA chip has a long time issue with both Linux and BSD kernels.