*New 11.4 series Release:
2020-07-03: XigmaNAS 11.4.0.4.7633 - released!

*New 12.1 series Release:
2020-04-17: XigmaNAS 12.1.0.4.7542 - released


We really need "Your" help on XigmaNAS https://translations.launchpad.net/xigmanas translations. Please help today!

Producing and hosting XigmaNAS costs money. Please consider donating for our project so that we can continue to offer you the best.
We need your support! eg: PAYPAL

NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

Posts only related to Release Builds, all others will be removed!
Forum rules
Set-Up GuideFAQsForum Rules
User avatar
HenriM
Starter
Starter
Posts: 56
Joined: 15 Jun 2013 12:17
Location: Rijsbergen, The Netherlands
Contact:
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#61

Post by HenriM »

Busy with PM with zoon01 to get this issue figured out. Seems related to over-provisioned memory-usage.

User avatar
JoseMR
Hardware & Software Guru
Hardware & Software Guru
Posts: 1142
Joined: 16 Apr 2014 04:15
Location: PR
Contact:
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#62

Post by JoseMR »

HenriM wrote: EDIT 1: I have Swap *not* enabled in System|Advanced|Swap but still it shows up in System Information.
EDIT 2: When I try to add mountpoint for da4 I get:

Code: Select all

The following input errors were detected: •/dev/da4p1: Can't get UFS ID.
dumpfs: /dev/da4p1: could not read superblock to fill out disk
EDIT 3: System Information of Partition of Device /dev/da4 - Corsair Slider 3.0 000E

Code: Select all

Device /dev/da4 - Corsair Slider 3.0 000E 
=>      34  30282941  da4  GPT  (14G)
        34         6       - free -  (3.0K)
        40      1024    1  freebsd-boot  (512K)
      1064      7160       - free -  (3.5M)
      8224   1638400    2  freebsd-ufs  (800M)
   1646624  16777216    3  freebsd-swap  (8.0G)
  18423840  11853824    4  freebsd-ufs  (5.7G)
  30277664      5311       - free -  (2.6M)
EDIT 4: Even when I have not enabled Swap (see above) I see in Status|Processes:

Code: Select all

last pid: 24708;  load averages:  0.65,  0.65,  0.50  up 1+04:04:02    15:52:40
41 processes:  2 running, 39 sleeping

Mem: 10M Active, 356M Inact, 13G Wired, 98M Buf, 2262M Free
ARC: 12G Total, 2065M MFU, 9858M MRU, 16K Anon, 41M Header, 230M Other
Swap: 8192M Total, 8192M Free
Hello HenriM, NAS4Free installer will create a Swap partition by default(No Swap File), so you don't need to activate by hand in the NAS4Free WebGUI unless you need/want to override default and specify a faster device like an HDD/SSD Partition, note that an USB3.0 disk even if placed on a USB2.0 port could be used as Swap safely in a home environment, as they faster than old USB2.0 ones and NAND-Flash ready, so don't worry about the myths on wearing them out too fast as many said, though a system with enough memory hitting constant Swap need some tuning.

To sum it up, If you using NAS4Free default installer option just don't worry about setting up Swap, as a proper native Swap Partition will be automatically set equal to your system RAM, though you can just decrease this swap partition to say 2GB(2048) or even 1GB(1024) as many already do, also note than swap files or zvol-as-swap are not recommended generally.

P.S If your system keeps crashing with enough free memory, memory exhaustion is not the problem here definitely, and some system testing monitoring/debugging with multiple setups is required which unfortunately is a bit complex and also time consuming in most of the cases.

Regards
System: FreeBSD 12 RootOnZFS Mirror, MB: Supermicro X8SI6-F, Xeon X3450, 16GB DDR3 ECC RDIMMs.
XigmaNAS RootOnZFS
Addons at GitHub
BastilleBSD
Boot Environments Intro
Resources Home Page

User avatar
HenriM
Starter
Starter
Posts: 56
Joined: 15 Jun 2013 12:17
Location: Rijsbergen, The Netherlands
Contact:
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#63

Post by HenriM »

Yeah, I also think that the memory is not the issue. The Asrock E3C224D4I-14S has LSI2308 SAS.

I have the latest firmware on it (= 20.00.04.00)

Image

(https://forums.freenas.org/index.php?th ... 308.36890/)

When I check for the 10.x branch I see next driver version:

10.3.0.3.3286
zfs2: ~# sysctl -a | grep mps | grep driver_version
dev.mps.0.driver_version: 20.00.00.00-fbsd

When I check for the 11.x branch I see next driver version:

11.0.0.4.3309
zfs2: ~# sysctl -a | grep mps | grep driver_version
dev.mps.0.driver_version: 21.01.00.00-fbsd

That is thus a new driver (21 vs. 20). Normally they say that with FreeBSD the driver version should be the same as the firmware. However for LSI2308 SAS I cannot find a firmware version 21...
See also next thread: https://forums.freenas.org/index.php?th ... p21.45130/

Can this be related or the issue?
Last edited by HenriM on 15 Dec 2016 18:25, edited 1 time in total.

User avatar
JoseMR
Hardware & Software Guru
Hardware & Software Guru
Posts: 1142
Joined: 16 Apr 2014 04:15
Location: PR
Contact:
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#64

Post by JoseMR »

HenriM wrote:Can this be related or the issue?
Hello HenriM, sorry for late response, regarding the freenas link about the SAS fw/driver mismatch, seems there is no problem actually for the FW/Driver not matching regarding what the driver author/maintainer said, though many users at freenas forum where just worried about the Mismatch message alert which is rather freenas WebGUI function/feature, though I could not actually confirm this is not an issue though.

Regards
System: FreeBSD 12 RootOnZFS Mirror, MB: Supermicro X8SI6-F, Xeon X3450, 16GB DDR3 ECC RDIMMs.
XigmaNAS RootOnZFS
Addons at GitHub
BastilleBSD
Boot Environments Intro
Resources Home Page

User avatar
HenriM
Starter
Starter
Posts: 56
Joined: 15 Jun 2013 12:17
Location: Rijsbergen, The Netherlands
Contact:
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#65

Post by HenriM »

Thank you for a reaction JoseMR. I also understood that Freenas people 'think' it is not a problem. I am running 10.x branch again with exactly the same hardware and system is stable again. Thus there must be some software/configuration issue with 11.x branch.

Therefore I am looking to the differences between these two releases what potentially could cause these hang ups with 11.x branch. One other thing which I did with 11.x was enabling 'Monitoring'. As a next test I want to disable this as well.

Unfortunately all this try-and-error cost a lot of time and I also have no possibility to check any logging after the issue happened. Do you have suggestions how to pinpoint the issue? Is there e.g. some debug logging I can turn on and which survives a reboot?

I really want to help the NAS4Free team (and myself of course ;-) ) to make the software better and am "happy" to try out things.

User avatar
JoseMR
Hardware & Software Guru
Hardware & Software Guru
Posts: 1142
Joined: 16 Apr 2014 04:15
Location: PR
Contact:
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#66

Post by JoseMR »

HenriM wrote:Do you have suggestions how to pinpoint the issue? Is there e.g. some debug logging I can turn on and which survives a reboot?

I really want to help the NAS4Free team (and myself of course ;-) ) to make the software better and am "happy" to try out things.
Hello, first of all thank you for whiling to help out the NAS4Free project, really appreciated, and yes I could gave you some suggestions about keeping logs after a system crash, if the system is simply auto restarting/rebooting itself(not by hardware watchdog), then you can simply create a shutdown command under "System > Advanced > Command Scripts" to copy the "/var/log" to a permanent location under reboot events, example commands below:

Code: Select all

cp -Rf /var/log /mnt/my_data_pool
However if the system is rebooting by the watchdog due system unresponsiveness timeout, or just lock-up then the above suggestion is useless, in this case a manually executed script on a dedicated SSH section with an infinite loop for example to rsync the last logs events from "/var/log" to "/mnt/my_data_pool" is required, though on hard system crash event there is no guarantee this method will retrieve the last log events, but indeed worth a try, I will give you an example of a very simple infinite loop script below, though you can break it at any time by pressing Ctrl+C if you notice excess system resources usage, I actually tested it and bump my CPU to a steady 10~14% average though:

Code: Select all

#!/bin/sh
# Permanent data location to store logs.
SAVEPATH="/mnt/my_data_pool"

rsync_log()
{
	# Infinite rsync loop.
	rsync -aqW /var/log ${SAVEPATH}
	rsync_log
}
rsync_log
Hope this can help.

Regards
System: FreeBSD 12 RootOnZFS Mirror, MB: Supermicro X8SI6-F, Xeon X3450, 16GB DDR3 ECC RDIMMs.
XigmaNAS RootOnZFS
Addons at GitHub
BastilleBSD
Boot Environments Intro
Resources Home Page

User avatar
HenriM
Starter
Starter
Posts: 56
Joined: 15 Jun 2013 12:17
Location: Rijsbergen, The Netherlands
Contact:
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#67

Post by HenriM »

The problem is that I have a hang up and in that case I cannot ping the server anymore. I have a dedicated keyboard and monitor attached to the NAS but also this is not reacting anymore in that case. Only thing I can do then is a hard shutdown (power button 4 seconds) of the system. Problem of that is that it will create new logfiles and thus the old ones are gone again...

I have added now the above command ' cp -Rf /var/log /mnt/Pool/Data/Logs ' as PreInit script with the hope that log files are not gone already in that case.

Otherwise I will try to run the above mentioned script in command shell on the Server.

This afternoon I tried 11.0.0.4.3330 (with monitoring enabled) and after couple of hours it hanged again. Now trying with monitoring (RRD) disabled. When this is again hanging I will try with disabled 'enabled tuning of some kernel variables'. Still some testing to do...

User avatar
JoseMR
Hardware & Software Guru
Hardware & Software Guru
Posts: 1142
Joined: 16 Apr 2014 04:15
Location: PR
Contact:
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#68

Post by JoseMR »

HenriM wrote:The problem is that I have a hang up and in that case I cannot ping the server anymore. I have a dedicated keyboard and monitor attached to the NAS but also this is not reacting anymore in that case. Only thing I can do then is a hard shutdown (power button 4 seconds) of the system. Problem of that is that it will create new logfiles and thus the old ones are gone again...

I have added now the above command ' cp -Rf /var/log /mnt/Pool/Data/Logs ' as PreInit script with the hope that log files are not gone already in that case.

Otherwise I will try to run the above mentioned script in command shell on the Server.

This afternoon I tried 11.0.0.4.3330 (with monitoring enabled) and after couple of hours it hanged again. Now trying with monitoring (RRD) disabled. When this is again hanging I will try with disabled 'enabled tuning of some kernel variables'. Still some testing to do...
Hello, If a lock-up then the infinite rsync loop will do better job here then cp, since rsync will constantly update logs files on any change until the lock-up happens again, then after the hard system reset you can find logs and search for its lats events before the lock-up happens, you can also sanitize(clear any sensitive info) and zip the log folder and attach file in this thread or PM me them so development team can have a look at it too.

Thanks for your testing support, and looking forward for a solution.
Regards
System: FreeBSD 12 RootOnZFS Mirror, MB: Supermicro X8SI6-F, Xeon X3450, 16GB DDR3 ECC RDIMMs.
XigmaNAS RootOnZFS
Addons at GitHub
BastilleBSD
Boot Environments Intro
Resources Home Page

User avatar
HenriM
Starter
Starter
Posts: 56
Joined: 15 Jun 2013 12:17
Location: Rijsbergen, The Netherlands
Contact:
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#69

Post by HenriM »

Your recursive script will not run for long JoseMR as it will stop after some time with 'Segmentation fault'. This does not surprise me as recursive coding is not a friend of your stack memory in general (especially when it is infinit ;-) ).

Is there some way that I can disable the deletion of the previous logfiles when starting up the system again?

With 'Monitoring' (RRD) disabled the system also hanged after 19 hours. I am now trying with disabled 'enabled tuning of some kernel variables'. Let see what this bring. Still puzzled what causes this hangup with 11.x branch as with 10.x branch with exactly the same hardware it runs stable for months.

User avatar
JoseMR
Hardware & Software Guru
Hardware & Software Guru
Posts: 1142
Joined: 16 Apr 2014 04:15
Location: PR
Contact:
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#70

Post by JoseMR »

HenriM wrote:Is there some way that I can disable the deletion of the previous logfiles when starting up the system again?
Hello HenriM, you are right about about sudden script termination on the previous infinite loop sample, though you can adjust and play with delays and such for, however there is also a tutorial on how to Move NAS4Free Logs to a different location, you can also auto generate log folder name with "$$", so after a system restart a new name will be generated thus leaving last logs intact for debugging purposes, example:

Code: Select all

Variable: clog_logdir
Value: /mnt/data/logs$$
Description: Log files location.
If above solution still don't work for you please let me know so I can do a brief modification of the rc in charge of the "/var/log" so you could specify/edit the logs location variable and overwrite at postinit as a temp solution.

Regards
System: FreeBSD 12 RootOnZFS Mirror, MB: Supermicro X8SI6-F, Xeon X3450, 16GB DDR3 ECC RDIMMs.
XigmaNAS RootOnZFS
Addons at GitHub
BastilleBSD
Boot Environments Intro
Resources Home Page

User avatar
HenriM
Starter
Starter
Posts: 56
Joined: 15 Jun 2013 12:17
Location: Rijsbergen, The Netherlands
Contact:
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#71

Post by HenriM »

Thank you JoseMR, will try that for next run. Currently test running with disabled 'enabled tuning of some kernel variables'. Up to now stable with uptime of almost 18 hours.

PS: Also looking now to setup remote syslog server om my Debian Server.

User avatar
HenriM
Starter
Starter
Posts: 56
Joined: 15 Jun 2013 12:17
Location: Rijsbergen, The Netherlands
Contact:
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#72

Post by HenriM »

Even with disabled 'enabled tuning of some kernel variables' I encountered a hangup after 2155 minutes. I do not find any direct information in logging (which I now saved to physical location). Last lines I see in system.log before forced restart at 10:53 (power off/on) by me are:

Code: Select all

Dec 12 05:32:46 zfs2 afpd[10408]: Login by HenriMatthijssen (AFP3.4)
Dec 12 05:32:46 zfs2 afpd[8468]: Disconnected session terminating
Dec 12 05:32:46 zfs2 afpd[2354]: Terminated disconnected child[8468], client rebooted.
Dec 12 05:37:48 zfs2 afpd[10408]: read: Operation timed out
Dec 12 05:37:48 zfs2 afpd[10408]: dsi_stream_read: len:-1, Operation timed out
Dec 12 05:37:48 zfs2 afpd[10408]: dsi_disconnect: entering disconnected state
Dec 12 06:35:47 zfs2 afpd[12315]: Login by HenriMatthijssen (AFP3.4)
Dec 12 06:35:47 zfs2 afpd[10408]: Disconnected session terminating
Dec 12 06:35:47 zfs2 afpd[2354]: Terminated disconnected child[10408], client rebooted.
Dec 12 06:40:48 zfs2 afpd[12315]: afp_alarm: child timed out, entering disconnected state
Dec 12 06:40:48 zfs2 afpd[12315]: dsi_disconnect: entering disconnected state
Dec 12 06:40:48 zfs2 afpd[12315]: dsi_disconnect: entering disconnected state
Dec 12 07:12:33 zfs2 afpd[24863]: Login by HenriMatthijssen (AFP3.4)
Dec 12 07:12:33 zfs2 afpd[12315]: Disconnected session terminating
Dec 12 07:12:33 zfs2 afpd[2354]: Terminated disconnected child[12315], client rebooted.
Dec 12 07:15:58 zfs2 afpd[24863]: AFP logout by HenriMatthijssen
Dec 12 07:15:58 zfs2 afpd[24863]: AFP statistics: 109497.68 KB read, 108055.29 KB written
Dec 12 07:15:58 zfs2 afpd[24863]: done
Dec 12 10:53:12 zfs2 syslogd: kernel boot file is /boot/kernel/kernel
Looks not really related to me. As a test I can turn off AppleTalk (AFP) now (used for Timemachine of MacBookAir of my daughter) and check.

Question: It seems that still not all logs are saved to physical disk?
When I add to rc.conf:

Code: Select all

Variable: clog_logdir
Value: /mnt/Pool/Data/ZFS
I miss for example the 'FTP - File Transfer' logging on the physical disk now (it is however displayed in NAS4Free UI). Any reason for that?

Further I guess we need to safeguard some system logging of FreeBSD to find lower related issues?

User avatar
raulfg3
Site Admin
Site Admin
Posts: 5116
Joined: 22 Jun 2012 22:13
Location: Madrid (ESPAÑA)
Contact:
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#73

Post by raulfg3 »

HenriM wrote:Even with disabled 'enabled tuning of some kernel variables' I encountered a hangup after 2155 minutes.
do you use ExtendedGUI?.
12.0.0.4 (revision 6766)+OBI on SUPERMICRO X8SIL-F 8GB of ECC RAM, 12x3TB disk in 3 vdev in RaidZ1 = 32TB Raw size only 22TB usable

Wiki
Last changes

HP T510

User avatar
HenriM
Starter
Starter
Posts: 56
Joined: 15 Jun 2013 12:17
Location: Rijsbergen, The Netherlands
Contact:
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#74

Post by HenriM »

No I do not use ExtendedGUI (I did use it in the past with 10.x). I already tried to rule out all possible causes (see whole thread). Even started from scratch with configuration, but this did not help.

Difference I did found between 10.x and 11.x branch is:

10.3.0.3.3286
zfs2: ~# sysctl -a | grep mps | grep driver_version
dev.mps.0.driver_version: 20.00.00.00-fbsd

When I check for the 11.x branch I see next driver version:

11.0.0.4.3309
zfs2: ~# sysctl -a | grep mps | grep driver_version
dev.mps.0.driver_version: 21.01.00.00-fbsd

User avatar
JoseMR
Hardware & Software Guru
Hardware & Software Guru
Posts: 1142
Joined: 16 Apr 2014 04:15
Location: PR
Contact:
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#75

Post by JoseMR »

HenriM wrote:Question: It seems that still not all logs are saved to physical disk?
When I add to rc.conf:

I miss for example the 'FTP - File Transfer' logging on the physical disk now (it is however displayed in NAS4Free UI). Any reason for that?

Further I guess we need to safeguard some system logging of FreeBSD to find lower related issues?
Hello HenriM, yes you will only change common log files such daemon and system with "clog_logdir", though some others can be overridden through rc.conf and others by explicit daemon parameters, also the main rc could be modified(hardcoded) to define a global log location on embedded during boot, but requires some hacks.

Eventually if you dont mind to run Full install for debug purposes, and have at least an USB3.0 memory stick, even if plugged at USB2.0, it can run NAS4Free full install decently, and you will retrieve the logs files on every reboot without the workaround/hacking hassle like in Embedded, even this could help us to determine if the hang-up for you are not related to the mdconfig drives we set for embedded, which are initially malloc followed by compression for new writes, and indeed this is the most notorious change in N4F11 actually, though "md2" is is swap backed*.

*Note that information about Malloc/Swap RAM DISK(md) has been removed from "Diagnostics > Information > Space Used" in N4F11.x Embedded, but you could still check this by executing: mdconfig -lv" at anytime.

Please make some test with NAS4Free Full if you can, so we can discard specific hardware issues, again thank you for helping make NAS4Free better.

Regards
System: FreeBSD 12 RootOnZFS Mirror, MB: Supermicro X8SI6-F, Xeon X3450, 16GB DDR3 ECC RDIMMs.
XigmaNAS RootOnZFS
Addons at GitHub
BastilleBSD
Boot Environments Intro
Resources Home Page

User avatar
HenriM
Starter
Starter
Posts: 56
Joined: 15 Jun 2013 12:17
Location: Rijsbergen, The Netherlands
Contact:
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#76

Post by HenriM »

I tried with APF disabled, but also got a hangup after couple of hours. Then I tried with OPT LAN Interface disabled again I noticed a hangup this morning. The current available logging does not give any information. I have the feeling that the problem is more OS/driver related.

Next week I have my holidays season and can do more extensive testing. I will try to setup a full install on a 16 GB USB memory stick.

Currently the command mdconfig -lv gives:

Code: Select all

zfs2: ~# mdconfig -lv
md0     preload           128M
md1     malloc    768M
md2     swap      512M
I have a 10 disk Z2 setup. I noticed that sometimes after a reboot the order of devices in Disks - Management is not correct and I have to execute a [Clear Config & Import Disks] to correct this. Not a big thing, but I just want to mention it here.

Other change I noticed in 11.x branch is the CIFS/SMB settings. I work in a mixed environment (Windows 10 machines, MacBookAir, Linux Debian, some Raspberry PI's with Raspbian) and currently I have set:

Server Max Protocol: SMB3
Server Min Protocol: Default
Client Max Protocol: Default
Client Min Protocol: Default

I have next services enabled: CFIS/SMB, FTP, SSH, NFS, APF and RSYNC.

Just trying to give as much background information as possible. Hopefully it helps. I really want to help your team with getting stable NAS4Free again, as I guess I am not the only one who is potentially suffering from these hang-ups with 11.x branch. Interesting enough on my other NAS which runs 11.x branch NAS4Free I do not have any problems (see this thread for details for this system which does not have LSI SAS).

User avatar
HenriM
Starter
Starter
Posts: 56
Joined: 15 Jun 2013 12:17
Location: Rijsbergen, The Netherlands
Contact:
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#77

Post by HenriM »

I have now installed NAS4Free full on 16 GB USB3 drive. I keep you up to date. Please let me know which information I should provide in case a hang up occurs again (all files from /var/log, syslog stored on remote server).

Question: would it help to turn on verbose logging during startup or will this make it from USB drive to slow?

User avatar
HenriM
Starter
Starter
Posts: 56
Joined: 15 Jun 2013 12:17
Location: Rijsbergen, The Netherlands
Contact:
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#78

Post by HenriM »

Running NAS4Free Full stable now for almost 14 hours. I have next observation when I compare NAS4Free Full with NAS4Free embedded. I installed both versions with all default settings on 16 GB pendrive. Further I have 16 GB ECC memory in the system.

With NAS4Free full I now see in System Information *no* line about Swap Usage (on /dev/da0p3). This looks confirmed by 'Status > Monitoring > Memory Usage' where with Embedded I noticed 24 GB (16 + 8) where with Full I now see 16 GB.

User avatar
HenriM
Starter
Starter
Posts: 56
Joined: 15 Jun 2013 12:17
Location: Rijsbergen, The Netherlands
Contact:
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#79

Post by HenriM »

The NAS4Free Full also encountered a hang up. It ran for about 28 hours. Unfortunately the /var/log and syslog do not contain any pointer what happened when the hang up occurred. At that timeframe nothing is logged.

Now restarted NAS4Free Full and set it to 'Verbose logging' (option 4). Hopefully this brings us some more details. Feeling that I am running out of options :?

User avatar
JoseMR
Hardware & Software Guru
Hardware & Software Guru
Posts: 1142
Joined: 16 Apr 2014 04:15
Location: PR
Contact:
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#80

Post by JoseMR »

Hello HenriM, sorry for bit late, as I'm working late hours this week suddenly, regarding the NAS4Free full hang-up, I can discard for now anything related to the md ram drives, also remember that the swap in N4F full, you have to manually point it to the partition under "System > Advanced > Swap", i.e "/dev/da0p3" in the case of GPT(da0s2 for MBR), where da0 is target USB disk and p3(or s2 for MBR) the swap partition/slice created by the installer.

Indeed we are running out of options due this Hard-Lockups, I hope the verbose logging will retrieve additional and useful information about your issue, though I'm toward some specific hardware issue, unfortunately this kind of specific problems are not easy to debug nor possible to reproduce unless testing with same exact setup, I will try to do some more research bout your motherboard and FreeBSD next coming days.
System: FreeBSD 12 RootOnZFS Mirror, MB: Supermicro X8SI6-F, Xeon X3450, 16GB DDR3 ECC RDIMMs.
XigmaNAS RootOnZFS
Addons at GitHub
BastilleBSD
Boot Environments Intro
Resources Home Page

User avatar
HenriM
Starter
Starter
Posts: 56
Joined: 15 Jun 2013 12:17
Location: Rijsbergen, The Netherlands
Contact:
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#81

Post by HenriM »

Thank you for a reaction JoseMR. I have not enabled any swap drive and I also do not think it is memory related. Monitoring shows plenty of free space in the overview before the hang up.

I have installed NAS4Free with all default settings. Hopefully this verbose logging will bring more light in the dark. I will also check the internet for issues regarding this motherboard and FreeBSD. Again with 10.x branch I have no problems whatsoever.

I really need to find out what they changed in the "dev.mps.0.driver_version: 21.01.00.00-fbsd" as this is no my main suspect.

I have all latest BIOS on Motherboard and LSI controller installed.

EDIT: In order to be complete overview of my attached devices

Code: Select all

zfs2: ~# camcontrol devlist
<ATA WDC WD40EZRX-00S 0A80>        at scbus0 target 0 lun 0 (pass0,da0)
<ATA WDC WD40EZRX-00S 0A80>        at scbus0 target 1 lun 0 (pass1,da1)
<ATA WDC WD40EZRX-00S 0A80>        at scbus0 target 2 lun 0 (pass2,da2)
<ATA WDC WD40EZRX-32S 0A80>        at scbus0 target 3 lun 0 (pass3,da3)
<WDC WD40EZRX-00SPEB0 80.00A80>    at scbus1 target 0 lun 0 (ada0,pass4)
<WDC WD40EZRX-00SPEB0 80.00A80>    at scbus2 target 0 lun 0 (ada1,pass5)
<WDC WD40EZRX-00SPEB0 80.00A80>    at scbus3 target 0 lun 0 (ada2,pass6)
<WDC WD40EZRX-00SPEB0 80.00A80>    at scbus4 target 0 lun 0 (ada3,pass7)
<WDC WD40EZRX-00SPEB0 80.00A80>    at scbus5 target 0 lun 0 (ada4,pass8)
<WDC WD40EZRX-00SPEB0 80.00A80>    at scbus6 target 0 lun 0 (ada5,pass9)
<AHCI SGPIO Enclosure 1.00 0001>   at scbus7 target 0 lun 0 (ses0,pass10)
<SanDisk Ultra 1.00>               at scbus8 target 0 lun 0 (pass11,da4)
Image

Sometimes after reboot one or more of the devices da0/da1/da2/da3 seems 'swapped' (red) and I have to use [Clear Config & Import Disks] to have it OK again.

User avatar
erico.bettoni
experienced User
experienced User
Posts: 142
Joined: 25 Jun 2012 22:36
Location: São Paulo - Brasil
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#82

Post by erico.bettoni »

I struggled with an unstable system for a while. Had out of memory errors even with a lot of free memory.
Then I disabled the default swap created by the installer and created a zvol and configured swap there.
Not a single error since then.
Tried different usb sticks with the same behavior. Only got the system stable after using swap on the zfs pool. I've set it to twice my memory size.

I don't know if it's related, but you could try it.

Here is the thread on system instability:
http://forums.nas4free.org/viewtopic.php?f=66&t=9871

User avatar
HenriM
Starter
Starter
Posts: 56
Joined: 15 Jun 2013 12:17
Location: Rijsbergen, The Netherlands
Contact:
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#83

Post by HenriM »

Thank you Erico for this information. It's worth a try.

But how did you found out that for you it was related to memory? I now encounter suddenly with 11.x software hang ups with no message whatsoever in the logging about memory. Also the monitoring shows that almost half of the memory is still free when hang up occurs.

However what I find strange is that with exactly the same hardware (and same setting of swap) I did not had any single problem with the 10.x branch of software. It only started now with 11.x.

First I thought it was perhaps because of corrupt pendrive and thus bought a new one just to be sure. But also then I have exactly the same issue.

User avatar
erico.bettoni
experienced User
experienced User
Posts: 142
Joined: 25 Jun 2012 22:36
Location: São Paulo - Brasil
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#84

Post by erico.bettoni »

I was getting memory messages from FTP and SSH/SCP. Samba also crashed.
However the system remained stable until ZFS itself was memory stressed, usually when I had several VMs running (altough still with lots of free memory).
I tried setting zfs memory parameters and even reset defaults, and got nothing.
Only when I migrated to 10.3 and changed swap my system stabilized again.

I'm avoiding going to 11.x exactly because of this thread...

User avatar
HenriM
Starter
Starter
Posts: 56
Joined: 15 Jun 2013 12:17
Location: Rijsbergen, The Netherlands
Contact:
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#85

Post by HenriM »

I just got a hang up again. This time I turned on verbose logging and last messages I see on syslog before hang up are:

Code: Select all

Dec 15 21:39:31 zfs_2.eye-vision smbd[2518]: [2016/12/15 21:39:31.901101,  1] ../source3/modules/nfs4_acls.c:788(smbacl4_fill_ace4)
Dec 15 21:39:31 zfs_2.eye-vision smbd[2518]: [2016/12/15 21:39:31.901101,  1] ../source3/modules/nfs4_acls.c:788(smbacl4_fill_ace4)
Dec 15 21:39:31 zfs_2.eye-vision smbd[2518]:   nfs4_acls.c: file [Programs/Netgear GS724Tv4/Software/Smart Control Center/SCC_v1_1_3_2_setup.zip.2j129ee.partial]: could not convert S-1-5-21-849853786-2206554682-617236450-1000 to uid or gid
Dec 15 21:39:31 zfs_2.eye-vision smbd[2518]:   nfs4_acls.c: file [Programs/Netgear GS724Tv4/Software/Smart Control Center/SCC_v1_1_3_2_setup.zip.2j129ee.partial]: could not convert S-1-5-21-849853786-2206554682-617236450-1000 to uid or gid
Not really sure if this will cause a hang up. This zip-file I just downloaded from internet with my Windows 10 machine to my Samba share on my NAS4Free box. However the download itself was OK and there are no issues with the zip file. Shortly after this I got the hang up.

Below graph of the memory usage:

Image

User avatar
raulfg3
Site Admin
Site Admin
Posts: 5116
Joined: 22 Jun 2012 22:13
Location: Madrid (ESPAÑA)
Contact:
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#86

Post by raulfg3 »

try to disable System > Advanced > Monitoring Setup is new code in latest 11.
12.0.0.4 (revision 6766)+OBI on SUPERMICRO X8SIL-F 8GB of ECC RAM, 12x3TB disk in 3 vdev in RaidZ1 = 32TB Raw size only 22TB usable

Wiki
Last changes

HP T510

User avatar
HenriM
Starter
Starter
Posts: 56
Joined: 15 Jun 2013 12:17
Location: Rijsbergen, The Netherlands
Contact:
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#87

Post by HenriM »

@raulfg3: I already tried with disabled Monitoring. See Post #69.

Going back to 10.x software now (10.3.0.3 - Pilingitam (revision 3330)) and let it run for a couple of days to proof again that hardware itself is stable and not the issue.

@Developers: please let me know if I need to do some tests. I have two pendrives and can quickly switch between 10.x and 11.x in order to execute tests (as long as my family keeps accepting these hang ups as this NAS box is an important part of my network for everyone ;-) ).

slaycock
experienced User
experienced User
Posts: 89
Joined: 24 Jun 2012 17:15
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#88

Post by slaycock »

I am getting multiple errors from my Del SAS card (LSI 1068E based). Request time out. Reverting to Nas4free 10 the errors go away. I have two of these cards and the problem happens with both cards.

User avatar
JoseMR
Hardware & Software Guru
Hardware & Software Guru
Posts: 1142
Joined: 16 Apr 2014 04:15
Location: PR
Contact:
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#89

Post by JoseMR »

erico.bettoni wrote:I'm avoiding going to 11.x exactly because of this thread...
Every wise sysadmin know that being on bleeding edge is a wrong-turn regarding stability, unless you need the drivers, but no need being panicked because of a hardware specific issue IMO, and yes I'm in the conservative boat too since if it works let it work. ;)

Also too legacy hardware users(or too newer) should not jump on the first place too, after some test and confirmation is ok then, but seems to be the contrary here.
System: FreeBSD 12 RootOnZFS Mirror, MB: Supermicro X8SI6-F, Xeon X3450, 16GB DDR3 ECC RDIMMs.
XigmaNAS RootOnZFS
Addons at GitHub
BastilleBSD
Boot Environments Intro
Resources Home Page

User avatar
HenriM
Starter
Starter
Posts: 56
Joined: 15 Jun 2013 12:17
Location: Rijsbergen, The Netherlands
Contact:
Status: Offline

Re: NAS4Free 11.0.0.4.3252 released (FreeBSD 11.0-RELEASE-P3)

#90

Post by HenriM »

I totally agree on you JoseMR, when it's not broke do not fix it. However nowadays there is a bad reason for keeping up to date with latest software: security fixes. This is especially true for NAS systems which are running 24/7. When I recall correctly 10.x will only get updates until half 2017. This means that at one stage I want really want to go to the latest branch again in order to keep up to date with latest (possible security) fixes.

That's why I also want to help with latest software (by doing all these tests) now. I have 2 pendrives, thus I can easily switch between software.

Post Reply

Return to “Release Builds”