Built-in embedded firmware upgrade in NAS4Free issue

[stangri]

I've started to notice that my box seems to misbehave when I use the built-in embedded firmware upgrade in NAS4Free.

Yesterday I've updated from 175 to 188 and I've had the following to start happening shortly after I've started to access (happened first when writing then even reading) the box:

Code: Select all

Fatal trap 12: page fault while in kernel mode                                  
cpuid = 0; apic id = 00                                                         
fault virtual address   = 0x63                                                  
fault code              = supervisor write data, page not present               
instruction pointer     = 0x20:0xffffffff80bfdbd9                               
stack pointer           = 0x28:0xffffff800021fad0                               
frame pointer           = 0x28:0xffffff800021fb40                               
code segment            = base 0x0, limit 0xfffff, type 0x1b                    
                        = DPL 0, pres 1, long 1, def32 0, gran 1                
processor eflags        = resume, IOPL = 0                                      
current process         = 12 (swi1: netisr 0)                                   
trap number             = 12                                                    
panic: page fault                                                               
cpuid = 0                                                                       
KDB: stack backtrace:                                                           
#0 0xffffffff80939cce at kdb_backtrace+0x5e                                     
#1 0xffffffff80904887 at panic+0x187                                            
#2 0xffffffff80c13650 at trap_fatal+0x290                                       
#3 0xffffffff80c13d07 at trap+0x287                                             
#4 0xffffffff80bfe2ef at calltrap+0x8                                           
#5 0xffffffff8090ca04 at mi_switch+0x174                                        
#6 0xffffffff808dcb46 at ithread_loop+0x216                                     
#7 0xffffffff808d83ff at fork_exit+0x11f                                        
#8 0xffffffff80bfe81e at fork_trampoline+0xe

Code: Select all

Fatal trap 12: page fault while in kernel mode                                  
cpuid = 0; apic id = 00                                                         
fault virtual address   = 0x360                                                 
fault code              = supervisor read data, page not present                
instruction pointer     = 0x20:0xffffffff8090beaf                               
stack pointer           = 0x28:0xffffff804cb7c8d0                               
frame pointer           = 0x28:0xffffff804cb7c950                               
code segment            = base 0x0, limit 0xfffff, type 0x1b                    
                        = DPL 0, pres 1, long 1, def32 0, gran 1                
processor eflags        = interrupt enabled, resume, IOPL = 0                   
current process         = 2329 (lighttpd)                                       
trap number             = 12                                                    
panic: page fault                                                               
cpuid = 0                                                                       
KDB: stack backtrace:                                                           
#0 0xffffffff80939cce at kdb_backtrace+0x5e                                     
#1 0xffffffff80904887 at panic+0x187                                            
#2 0xffffffff80c13650 at trap_fatal+0x290                                       
#3 0xffffffff80c13999 at trap_pfault+0x1f9                                      
#4 0xffffffff80c13e5f at trap+0x3df                                             
#5 0xffffffff80bfe2ef at calltrap+0x8                                           
#6 0xffffffff8096f4a9 at soreceive_generic+0xc9                                 
#7 0xffffffff8094bc41 at dofileread+0xa1                                        
#8 0xffffffff8094bfdc at kern_readv+0x6c                                        
#9 0xffffffff8094c0c5 at sys_read+0x55                                          
#10 0xffffffff80c12f40 at amd64_syscall+0x4e0                                   
#11 0xffffffff80bfe5d7 at Xfast_syscall+0xf7                                    
Uptime: 1m45s                                                                   
Automatic reboot in 15 seconds - press a key on the console to abort

And once during the boot-up:

Code: Select all

Found configuration on device=/dev/ada4a fstype=ufs.                            
Updating rc.conf:....................................                           
pid 826 (php), uid 0: exited on signal 11 (core dumped)                         
Segmentation fault (core dumped)

Funnily enough, if I pull out the CF card and write a new image on it, I can then successfully upload the old config and everything works as it should so it's clearely not a problem with config.

Now that I look back at my past problems, I believe it's not a first time the built-in embedded firmware upgrade caused me pain. I used to blame it on the degraded health condition of the ZFS pool, but as I needed to crack the box open anyways the installation of the new HDD coincided with me putting a fresh CF card with the freshly written embedded image on it. So now that I think of it maybe the issue was that I've used the built-in firmware upgrade feature.

Anyways, as nothing bad happens during the upgrade (nothing in Web GUI and nothing in the console) I don't know if the upgrade is the culprit here, but maybe you guys would find the crash messages helpful.

[kernow]

Have you done a RAM test recently?

[lux]

sorry, but as i wrote here: viewtopic.php?f=57&t=874

i've updated all beta versions asap - there's no problem with embedded upgrade procedure...

i know this really wouldn't help you - but pls, don't use CF-Card's anymore

there are alternative's available --> http://www.transcend-info.com/industry/ ... unc2No=171

pata Flash Modules - 40Pin/44Pin

i use 40Pin PATA Flash Module without any issues

few Years ago i always used CF-Card with Pata Adapter & have sometimes very Strange Errors...

but check your RAM first as kernow mentioned - good Luck!

[stangri]

Have you done a RAM test recently?

No, but thank you, I'll start with that.

sorry, but as i wrote here: viewtopic.php?f=57&t=874

i know this really wouldn't help you - but pls, don't use CF-Card's anymore

CF card was an easier choice as my server box doesn't have CD so I have no way of installing the software other than using the cf card adaptor for my laptop to dd the image. I don't have an adaptor so I could dd the image onto DOM. Also the vertical DOM won't fit in my box.

Oh and WNY NOT CF cards?
BTW, I'm not the only one with the issues with the embedded update.

[stangri]

Created the Bootable Memtest86+ CF card and it doesn't quite work on my headless setup. I can see the SYSLINUX label and then the serial screen locks up, so I have no idea if it's running or not.

There's no memtest built into the embedded Nas4Free installation, is there?

In the meantime, I've moved to running embedded from the microSD card inside the USB adaptor. I would *really* like to hear back from the developers tho if it matters where *embedded* install runs from. AFAIK once it's loaded it doesn't touch the media as everything is in the RAM.

[lux]

Oh and WNY NOT CF cards?

do you read what i wrote???????

few Years ago i always used CF-Card with Pata Adapter & have sometimes very Strange Errors...

if you want to go your own Way with CF - it's okay - but please not blame that something must be wrong...

[zoon01]

I use CF for a very long time now,

http://interngeheugen.com/prod/49550/Ad ... black.html

I flash more than any user around as i make more builds than i upload to SF.
Never ever i had a fail at all with my CF media and flashing the .img files.

Please check your hardware, your ram, your cf media and or cables.

it seems to me you got a hardware error.

[lux]

@ zoon

not all Mainboard could handle CF-Card

my strangest Issue with CF-Card was a few month ago - i was updating my IPCop1.4.21 to 2.0.1 - no chance, installer breaks with weird errors all over the Screen (for sure i try 3-4 different CF-Card & different CF to ATA Adapter)

so i assume my Old Hardware PIII-800 with i810 Chipset was not compatible with IPCop 2.x (newer Kernel)

so i got new Hardware with itx Atom inside (not really cheap)

then i "found" old Transcend 40pin PATA Flash - so i test it with new IPCop & old i810 Chipset - wth, it works...!!! so the newer Kernel (2.6.x) broke up with my CF-Card's...?!?

the old IPCop 1.4.21 with Kernel 2.4.x works like a charm last Years...!

so i repeat do not use CF-Card's anymore - there are alternative's to buy without any issues like 40/44pin PATA Flash Module's...

...i spent alot of Money for new Hardware even i don't need...

just my experiences

I've been with freeNAS since v0.4 - i can't remind me - but i had similar problem with freeNAS after upgrade few years ago...

[stangri]

There's no memtest built into the embedded Nas4Free installation, is there?

If anyone *has* booted memtest on a headless machine, I would greatly appreciate some help. I've tried googling for info, but came up short.

Oh and WNY NOT CF cards?

do you read what i wrote???????

Did you read what I wrote? Why does the boot media matter on the embedded install when N4F runs from the memory anyways? And BTW, it crashes when running from USB sticks too.

Please check your hardware, your ram, your cf media and or cables. it seems to me you got a hardware error.

And the RAM would be my primary suspect, however I have no way of testing it on my headless machine and I have no other PC I could test that stick of RAM in. Speaking of CF media and cables -- if my *embedded* N4F install boots fine, does it ever access the CF (except the settings update) when it's running?

I've had 4 reboots of N4F build 188 today (when running from USB stick), so I'll let it run under Daisuke's FreeNAS-x64-embedded-0.7.5.9896 for a few days to see if things will pick up.

[shakky4711]

Hello,

just my few cents, I feel CF-cards are not the problem in most cases (use generally only Transcend with lifetime warranty, until now without any fail), the problems are based in many cases on the used ide-cf adapter, there is such a lot of cheap scrap available, not so easy to find a reliable adapter. But I totally agree that IDE-Flashdrives are the better choice, have three 1GB drives running from Transcend, absolute great.

About reboots, currently I have many problems at home because of the summer heat, some machines which run until now without problems reboot/power-down because of CPU emergency shutdown.
Sometimes I have had poweroff because of damaged power supply units.

Furthermore I made he experience that not all hardware is compatible with FreeBSD, I have had a Shuttle which was not able to run FreeBSD/FreeNAS, needed to install Linux.

Shakky

[HorzaSe]

Dunno if it might be related but I encountered this Error when trying to update my embedded 188 to latest 228, today.
When I press update Firmware before actually selecting any IMG.:

Code: Select all

root: UPGRADE ERROR: Failed to create in-memory file system.

Also in WebGUI I get:

Code: Select all

The following input errors were detected:
Failed to create in-memory file system.

Memory states: 33% used out of 1790MiB.

So there should be plenty right? Gonna try rebooting to see what happens.

[HorzaSe]

Tried rebooting and now it let me update, though w 9% used memory.
Maybe some thing wonky in Memory adressing?

[stangri]

Well, I did roll back to the custom Daisuke's build (9S-9896) and it's been running flawlessly, without any reboots or problems.

The problem with NAS4Free could very well be memory management related -- I'm running a box with the physical 2Gb limit.

Anyways, I guess that means no NAS4Free for me until I upgrade my hardware.

[raulfg3]

Dunno if it might be related but I encountered this Error when trying to update my embedded 188 to latest 228, today.
When I press update Firmware before actually selecting any IMG.:

Código: Seleccionar todo
root: UPGRADE ERROR: Failed to create in-memory file system.

Solved in latest 328 = http://nas4free.svn.sourceforge.net/vie ... vision=328

[jmarc]

Dunno if it might be related but I encountered this Error when trying to update my embedded 188 to latest 228, today.
When I press update Firmware before actually selecting any IMG.:

Código: Seleccionar todo
root: UPGRADE ERROR: Failed to create in-memory file system.

Solved in latest 328 = http://nas4free.svn.sourceforge.net/vie ... vision=328

Hello,
I have the same problem with the 9.1 version when i push the "Enable Firmware Update" button.
The use of the memory is less than 30 % on usable 5.6GB
My nas have 2 jails working (serviio and torrent/pyload), theBrig extension and this script at boot

Code: Select all

#!/bin/tcsh
/bin/sh cp /mnt/Raid/Jail/basejail/usr/share/syscons/keymaps/fr.iso.acc.kbd /usr/share/syscons/keymaps/
mount_unionfs /mnt/Raid/Prive/root /root
mkdir /jail
mount_nullfs /mnt/Raid/Jail /jail
mkdir /usr/local/www/serviioweb
mount_nullfs /mnt/Raid/www/serviioweb /usr/local/www/serviioweb
cp /mnt/Raid/local/etc/profile /etc

/mnt/Raid/local/etc/profile

Code: Select all

PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
export PATH
LANG=fr_FR.UTF-8
export LANG
MM_CHARSET=UTF-8
export MM_CHARSET

capture.jpg

[jmarc]

Hello, I stopped all the services, without improvement.
I put the parameters by default (parameters factory), and I can make the update.
If it is the only way of making, I am disappointed by the procedure, especially as my NAS is in the basement
I am going to continue to look, if I find what leads(infers) this error, I make you a return.

EDIT: I was not able to restore my configuration, I had to make a clean installation

[jmarc]

I stop all specific process (theBrig, Jails, personnal scripts), update don't work, same message.
I stop all services to see, no changes.
I reboot.

[raulfg3]

EDIT: I was not able to restore my configuration, I had to make a clean installation

this can be due a manual-edited config (so is not standard or is corrupted) or a bad media ( burn USB stick).

Please try do install in a new fresh boot device and restore config.

[jmarc]

I tried everything, stop of the services, no extension, I cannot update my version of NAS4Free. My NAS is in the basement, without cd-rom drive, in every update it's hellish and since the beginning of the versions 9.x.
My interface is in French, I tried in English, no change. The only way of making an update by WebUI, it is to go back in configuration manufacture, so Bad.
@raulfg3: If I have again the same problem, I shall follow your solution. Thank you.

[raulfg3]

you can do a fresh embeded install in a new USB in your desktop PC ( No need to be in your NAS), once new *.img is properly copied to new USB, you can boot your NAS from it, and restore your previosly saved config, to restore your NAS to actual state but upgraded.

[jmarc]

I found the culprit, it is about the theBrig extension. Having executed the command "/usr/local/bin/php-cgi /mnt/Raid/Jail/conf/bin/thebrig_start.php", I again had the error.
So bad, i like this extension, so easy to use jail with it.

[jmarc]

it broken configuration file, reset to default -_-
I think it is the write_rcconflocal (); in the php who's broke the config file

[raulfg3]

Thebrig works fine until N4F 804.
I detected a problem with rc.conf.local creation only in latest 847, an i do not think that afecct to upgrade of embeded system, only start of slim jails.
developers are working in a workarround, until done use Nas4Free 9.1.0.1.804 instead.

[alexey123]

jmarc
write_rcconflocal (); can not broke config.xml, this function only read values from config and write rc.conf.local and fstab.<jailname> files.
Check please permissions for /tmp, /var/tmp, and /var/ftpmp folders. Must be 1777
Some behavior I remember if you use "Install packages into embedded" with command

Code: Select all

mount_unionfs /var /mnt/your/location

In this case I removed scripts, which mount vars and after reboot make upgrade.

[jmarc]

Hi, yes it's not this script, so where did you save the nas4free config after adding this ligne "/usr/local/bin/php-cgi /mnt/Raid/Jail/conf/bin/thebrig_start.php" in System/Advanced/Command Script
I think there is a change in the XML (UTF8/ANSII ....) causing this error.
I test all change in my configuration of Nas4free, and it's after installing theBrig that i can't upgrade directly in the webUI
Upgrade with CD says me it can't read config xml (or restore can't remember)
That's all.