Is it all gone? Help me - Obi Wan...

idmah · Post by **idmah** » 01 Jun 2013 20:04

Hi all.

I've been putting off trying to fix my freeNas for a while, basically it's been off.
with great trepidation I booted it yesterday and it basically runs for about 2-4 minutes then crashes.
two of the 4 drives seem to be bad. Which I think the resilvering is killing it.

I tried to unplug one of the offending drives and but it stilled crashed.

Here is all the information I was able to get out of it before crash-freeze.
ANY Help would be greatly appreciated. Can I recover the information from the
1-2 ? remaining ZFS drives and Wipe the whole thing clean? or put them in a new
server?

thanks
Ian

ps. For some reason I can't boot from the CD either..

system info:

FreeNAS 0.7RC1 (Sardaukar):
---------------------------

Majors changes:
- Upgrade to FreeBSD 7.2.
- Include ZFS support. Thanks to Nelson Silva for doing core coding and Falk
Menzel for testing and giving some tipps and ideas.
- Upgrade iSCSI initiator to 2.1.1.
- Replace iSCSI target by istgt. Thanks to Daisuke Aoyama for the WebGUI
adaption. Please note, if you have used devices with the previous iSCSI target
software you have to recreate your target.
- Add WOL patch for nVidia(nfe(4)) and 3Com(xl(4)). Thanks to Tobias Reber.
- Upgrade mt-daapd/firefly to svn-1696.
- Refactor 'Diagnostics|Log' WebGUI.
- Add kernel patches to get ARTiGO A2000 hardware working. Thanks to
David Davis for the patches.
- Respect the modified log file location (via rc.conf for syslog, fuppes,
mt-daapd, ...) in the WebGUI (FR 2778803/2791772).
- Upgrade transmission to 1.61. Add 'Watch directory' and 'Extra options'
fields to 'Services|BitTorrent' WebGUI.
- Add entry 'FTP - Ban module' to the list of log files in 'Diagnostics|Log'
if the module is enabled (FR 2797652).
- Add 'iperf', a tool to measure maximum TCP and UDP bandwidth (FR 2785038).
- Add 'bsnmp-ucd' module that implements parts of UCD-SNMP-MIB.
- Add SNMP client tools: bsnmpget, bsnmpset and bsnmpwalk
- Add 'Auxiliary parameters' to 'Services|SNMP' that will be added to the end
of the snmpd.config file.
- Upgrade e2fsprogs to 1.41.5.
- Upgrade rsync to 3.0.6.
- Upgrade tftp-hpa to 0.49.

other info: not sure if useful.

Code: Select all

ATA channel 0:
    Master:      no device present
    Slave:       no device present
ATA channel 1:
    Master:      no device present
    Slave:       no device present
ATA channel 2:
    Master:  ad4  SATA revision 2.x
    Slave:       no device present
ATA channel 3:
    Master:  ad6  SATA revision 2.x
    Slave:       no device present
ATA channel 4:
    Master:  ad8  SATA revision 2.x
    Slave:       no device present
ATA channel 5:
    Master: ad10  SATA revision 2.x
    Slave:       no device present
Device /dev/ad4 - ST31500341AS/CC1H
******* Working on device /dev/ad4 *******
parameters extracted from in-core disklabel are:
cylinders=2907021 heads=16 sectors/track=63 (1008 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=2907021 heads=16 sectors/track=63 (1008 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
    start 63, size 2930277105 (1430799 Meg), flag 80 (active)
	beg: cyl 0/ head 1/ sector 1;
	end: cyl 908/ head 15/ sector 63
The data for partition 2 is:

The data for partition 3 is:

The data for partition 4 is:

Device /dev/ad6 - ST31500341AS/CC1H
******* Working on device /dev/ad6 *******
parameters extracted from in-core disklabel are:
cylinders=2907021 heads=16 sectors/track=63 (1008 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=2907021 heads=16 sectors/track=63 (1008 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
    start 63, size 2930277105 (1430799 Meg), flag 80 (active)
	beg: cyl 0/ head 1/ sector 1;
	end: cyl 908/ head 15/ sector 63
The data for partition 2 is:

The data for partition 3 is:

The data for partition 4 is:

Device /dev/ad8 - ST31500341AS/CC1H
******* Working on device /dev/ad8 *******
parameters extracted from in-core disklabel are:
cylinders=2907021 heads=16 sectors/track=63 (1008 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=2907021 heads=16 sectors/track=63 (1008 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 7 (0x07),(OS/2 HPFS, NTFS, QNX-2 (16 bit) or Advanced UNIX)
    start 63, size 2930272002 (1430796 Meg), flag 80 (active)
	beg: cyl 0/ head 1/ sector 1;
	end: cyl 1023/ head 254/ sector 63
The data for partition 2 is:

The data for partition 3 is:

The data for partition 4 is:

Device /dev/ad10 - WDC WD15EARX-0

Logs:

Code: Select all

Apr  4 22:22:43 bfmb syslogd: kernel boot file is /boot/kernel/kernel
Apr  4 22:22:43 bfmb kernel: Copyright (c) 1992-2009 The FreeBSD Project.
Apr  4 22:22:43 bfmb kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Apr  4 22:22:43 bfmb kernel: The Regents of the University of California. All rights reserved.
Apr  4 22:22:43 bfmb kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
Apr  4 22:22:43 bfmb kernel: FreeBSD 7.2-RELEASE-p1 #0: Fri Jun 19 20:27:15 UTC 2009
Apr  4 22:22:43 bfmb kernel: root@vmbsd72amd64:/usr/obj/freenas/usr/src/sys/FREENAS-amd64
Apr  4 22:22:43 bfmb kernel: Timecounter "i8254" frequency 1193182 Hz quality 0
Apr  4 22:22:43 bfmb kernel: CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 4200+ (2205.01-MHz K8-class CPU)
Apr  4 22:22:43 bfmb kernel: Origin = "AuthenticAMD"  Id = 0x20fb1  Stepping = 1
Apr  4 22:22:43 bfmb kernel: Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
Apr  4 22:22:43 bfmb kernel: Features2=0x1<SSE3>
Apr  4 22:22:43 bfmb kernel: AMD Features=0xe2500800<SYSCALL,NX,MMX+,FFXSR,LM,3DNow!+,3DNow!>
Apr  4 22:22:43 bfmb kernel: AMD Features2=0x3<LAHF,CMP>
Apr  4 22:22:43 bfmb kernel: Cores per package: 2
Apr  4 22:22:43 bfmb kernel: usable memory = 2039070720 (1944 MB)
Apr  4 22:22:43 bfmb kernel: avail memory  = 1963139072 (1872 MB)
Apr  4 22:22:43 bfmb kernel: ACPI APIC Table: <A M I  OEMAPIC >
Apr  4 22:22:43 bfmb kernel: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
Apr  4 22:22:43 bfmb kernel: cpu0 (BSP): APIC ID:  0
Apr  4 22:22:43 bfmb kernel: cpu1 (AP): APIC ID:  1
Apr  4 22:22:43 bfmb kernel: ioapic0 <Version 1.1> irqs 0-23 on motherboard
Apr  4 22:22:43 bfmb kernel: wlan: mac acl policy registered
Apr  4 22:22:43 bfmb kernel: kbd1 at kbdmux0
Apr  4 22:22:43 bfmb kernel: cryptosoft0: <software crypto> on motherboard
Apr  4 22:22:43 bfmb kernel: acpi0: <A M I OEMRSDT> on motherboard
Apr  4 22:22:43 bfmb kernel: acpi0: [ITHREAD]
Apr  4 22:22:43 bfmb kernel: acpi0: Power Button (fixed)
Apr  4 22:22:43 bfmb kernel: acpi0: reservation of 0, a0000 (3) failed
Apr  4 22:22:43 bfmb kernel: acpi0: reservation of 100000, 7ff00000 (3) failed
Apr  4 22:22:43 bfmb kernel: Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
Apr  4 22:22:43 bfmb kernel: acpi_timer0: <32-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
Apr  4 22:22:43 bfmb kernel: pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
Apr  4 22:22:43 bfmb kernel: pci0: <ACPI PCI bus> on pcib0
Apr  4 22:22:43 bfmb kernel: pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
Apr  4 22:22:43 bfmb kernel: pci1: <ACPI PCI bus> on pcib1
Apr  4 22:22:43 bfmb kernel: vgapci0: <VGA-compatible display> port 0xd000-0xd0ff mem 0xd0000000-0xdfffffff,0xceee0000-0xceeeffff irq 18 at device 0.0 on pci1
Apr  4 22:22:43 bfmb kernel: pci1: <multimedia, HDA> at device 0.1 (no driver attached)
Apr  4 22:22:43 bfmb kernel: pcib2: <ACPI PCI-PCI bridge> at device 26.0 on pci0
Apr  4 22:22:43 bfmb kernel: pci2: <ACPI PCI bus> on pcib2
Apr  4 22:22:43 bfmb kernel: fwohci0: <Texas Instruments TSB43AB22/A> mem 0xcefff800-0xceffffff,0xceff8000-0xceffbfff irq 20 at device 19.0 on pci2
Apr  4 22:22:43 bfmb kernel: fwohci0: [FILTER]
Apr  4 22:22:43 bfmb kernel: fwohci0: OHCI version 1.10 (ROM=1)
Apr  4 22:22:43 bfmb kernel: fwohci0: No. of Isochronous channels is 4.
Apr  4 22:22:43 bfmb kernel: fwohci0: EUI64 00:11:d8:00:00:84:17:30
Apr  4 22:22:43 bfmb kernel: fwohci0: Phy 1394a available S400, 2 ports.
Apr  4 22:22:43 bfmb kernel: fwohci0: Link S400, max_rec 2048 bytes.
Apr  4 22:22:43 bfmb kernel: firewire0: <IEEE1394(FireWire) bus> on fwohci0
Apr  4 22:22:43 bfmb kernel: fwe0: <Ethernet over FireWire> on firewire0
Apr  4 22:22:43 bfmb kernel: if_fwe0: Fake Ethernet address: 02:11:d8:84:17:30
Apr  4 22:22:43 bfmb kernel: fwe0: Ethernet address: 02:11:d8:84:17:30
Apr  4 22:22:43 bfmb kernel: fwip0: <IP over FireWire> on firewire0
Apr  4 22:22:43 bfmb kernel: fwip0: Firewire address: 00:11:d8:00:00:84:17:30 @ 0xfffe00000000, S400, maxrec 2048
Apr  4 22:22:43 bfmb kernel: sbp0: <SBP-2/SCSI over FireWire> on firewire0
Apr  4 22:22:43 bfmb kernel: fwohci0: Initiate bus reset
Apr  4 22:22:43 bfmb kernel: fwohci0: BUS reset
Apr  4 22:22:43 bfmb kernel: fwohci0: node_id=0xc800ffc0, gen=1, CYCLEMASTER mode
Apr  4 22:22:43 bfmb kernel: skc0: <Marvell Gigabit Ethernet> port 0xe800-0xe8ff mem 0xceff4000-0xceff7fff irq 21 at device 20.0 on pci2
Apr  4 22:22:43 bfmb kernel: skc0: Marvell Yukon Lite Gigabit Ethernet rev. (0x9)
Apr  4 22:22:43 bfmb kernel: sk0: <Marvell Semiconductor, Inc. Yukon> on skc0
Apr  4 22:22:43 bfmb kernel: sk0: Ethernet address: 00:15:f2:76:da:f4
Apr  4 22:22:43 bfmb kernel: miibus0: <MII bus> on sk0
Apr  4 22:22:43 bfmb kernel: e1000phy0: <Marvell 88E1011 Gigabit PHY> PHY 0 on miibus0
Apr  4 22:22:43 bfmb kernel: e1000phy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX-FDX, auto
Apr  4 22:22:43 bfmb kernel: skc0: [ITHREAD]
Apr  4 22:22:43 bfmb kernel: ohci0: <AcerLabs M5237 (Aladdin-V) USB controller> mem 0xcedfc000-0xcedfcfff irq 17 at device 28.0 on pci0
Apr  4 22:22:43 bfmb kernel: ohci0: [GIANT-LOCKED]
Apr  4 22:22:43 bfmb kernel: ohci0: [ITHREAD]
Apr  4 22:22:43 bfmb kernel: usb0: OHCI version 1.0, legacy support
Apr  4 22:22:43 bfmb kernel: usb0: SMM does not respond, resetting
Apr  4 22:22:43 bfmb kernel: usb0: <AcerLabs M5237 (Aladdin-V) USB controller> on ohci0
Apr  4 22:22:43 bfmb kernel: usb0: USB revision 1.0
Apr  4 22:22:43 bfmb kernel: uhub0: <AcerLabs OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
Apr  4 22:22:43 bfmb kernel: uhub0: 3 ports with 3 removable, self powered
Apr  4 22:22:43 bfmb kernel: ohci1: <AcerLabs M5237 (Aladdin-V) USB controller> mem 0xcedfd000-0xcedfdfff irq 18 at device 28.1 on pci0
Apr  4 22:22:43 bfmb kernel: ohci1: [GIANT-LOCKED]
Apr  4 22:22:43 bfmb kernel: ohci1: [ITHREAD]
Apr  4 22:22:43 bfmb kernel: usb1: OHCI version 1.0, legacy support
Apr  4 22:22:43 bfmb kernel: usb1: SMM does not respond, resetting
Apr  4 22:22:43 bfmb kernel: usb1: <AcerLabs M5237 (Aladdin-V) USB controller> on ohci1
Apr  4 22:22:43 bfmb kernel: usb1: USB revision 1.0
Apr  4 22:22:43 bfmb kernel: uhub1: <AcerLabs OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
Apr  4 22:22:43 bfmb kernel: uhub1: 3 ports with 3 removable, self powered
Apr  4 22:22:43 bfmb kernel: ohci2: <AcerLabs M5237 (Aladdin-V) USB controller> mem 0xcedfe000-0xcedfefff irq 19 at device 28.2 on pci0
Apr  4 22:22:43 bfmb kernel: ohci2: [GIANT-LOCKED]
Apr  4 22:22:43 bfmb kernel: ohci2: [ITHREAD]
Apr  4 22:22:43 bfmb kernel: usb2: OHCI version 1.0, legacy support
Apr  4 22:22:43 bfmb kernel: usb2: SMM does not respond, resetting
Apr  4 22:22:43 bfmb kernel: usb2: <AcerLabs M5237 (Aladdin-V) USB controller> on ohci2
Apr  4 22:22:43 bfmb kernel: usb2: USB revision 1.0
Apr  4 22:22:43 bfmb kernel: uhub2: <AcerLabs OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
Apr  4 22:22:43 bfmb kernel: uhub2: 3 ports with 3 removable, self powered
Apr  4 22:22:43 bfmb kernel: ehci0: <ALi M5239 USB 2.0 controller> mem 0xcedffc00-0xcedffcff irq 23 at device 28.3 on pci0
Apr  4 22:22:43 bfmb kernel: ehci0: [GIANT-LOCKED]
Apr  4 22:22:43 bfmb kernel: ehci0: [ITHREAD]
Apr  4 22:22:43 bfmb kernel: usb3: EHCI version 1.0
Apr  4 22:22:43 bfmb kernel: usb3: companion controllers, 3 ports each: usb0 usb1 usb2
Apr  4 22:22:43 bfmb kernel: usb3: <ALi M5239 USB 2.0 controller> on ehci0
Apr  4 22:22:43 bfmb kernel: usb3: USB revision 2.0
Apr  4 22:22:43 bfmb kernel: uhub3: <AcerLabs EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb3
Apr  4 22:22:43 bfmb kernel: uhub3: 8 ports with 8 removable, self powered
Apr  4 22:22:43 bfmb kernel: umass0: <Kingston DataTraveler 2.0, class 0/0, rev 2.00/1.00, addr 2> on uhub3
Apr  4 22:22:43 bfmb kernel: pci0: <multimedia, HDA> at device 29.0 (no driver attached)
Apr  4 22:22:43 bfmb kernel: isab0: <PCI-ISA bridge> at device 30.0 on pci0
Apr  4 22:22:43 bfmb kernel: isa0: <ISA bus> on isab0
Apr  4 22:22:43 bfmb kernel: pci0: <bridge> at device 30.1 (no driver attached)
Apr  4 22:22:43 bfmb kernel: atapci0: <AcerLabs M5229 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 31.0 on pci0
Apr  4 22:22:43 bfmb kernel: ata0: <ATA channel 0> on atapci0
Apr  4 22:22:43 bfmb kernel: ata0: [ITHREAD]
Apr  4 22:22:43 bfmb kernel: ata1: <ATA channel 1> on atapci0
Apr  4 22:22:43 bfmb kernel: ata1: [ITHREAD]
Apr  4 22:22:43 bfmb kernel: atapci1: <AcerLabs M5288 SATA300 controller> port 0xc800-0xc807,0xc400-0xc403,0xc000-0xc007,0xb800-0xb803,0xb400-0xb40f mem 0xcedff800-0xcedffbff irq 19 at device 31.1 on pci0
Apr  4 22:22:43 bfmb kernel: atapci1: [ITHREAD]
Apr  4 22:22:43 bfmb kernel: atapci1: AHCI called from vendor specific driver
Apr  4 22:22:43 bfmb kernel: atapci1: AHCI Version 01.00 controller with 4 ports detected
Apr  4 22:22:43 bfmb kernel: ata2: <ATA channel 0> on atapci1
Apr  4 22:22:43 bfmb kernel: ata2: [ITHREAD]
Apr  4 22:22:43 bfmb kernel: ata3: <ATA channel 1> on atapci1
Apr  4 22:22:43 bfmb kernel: ata3: [ITHREAD]
Apr  4 22:22:43 bfmb kernel: ata4: <ATA channel 2> on atapci1
Apr  4 22:22:43 bfmb kernel: ata4: [ITHREAD]
Apr  4 22:22:43 bfmb kernel: ata5: <ATA channel 3> on atapci1
Apr  4 22:22:43 bfmb kernel: ata5: [ITHREAD]
Apr  4 22:22:43 bfmb kernel: acpi_button0: <Power Button> on acpi0
Apr  4 22:22:43 bfmb kernel: speaker0: <PC speaker> port 0x61 on acpi0
Apr  4 22:22:43 bfmb kernel: fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
Apr  4 22:22:43 bfmb kernel: fdc0: [FILTER]
Apr  4 22:22:43 bfmb kernel: ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0
Apr  4 22:22:43 bfmb kernel: ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
Apr  4 22:22:43 bfmb kernel: ppc0: FIFO with 16/16/16 bytes threshold
Apr  4 22:22:43 bfmb kernel: ppbus0: <Parallel port bus> on ppc0
Apr  4 22:22:43 bfmb kernel: ppbus0: [ITHREAD]
Apr  4 22:22:43 bfmb kernel: ppi0: <Parallel I/O> on ppbus0
Apr  4 22:22:43 bfmb kernel: ppc0: [GIANT-LOCKED]
Apr  4 22:22:43 bfmb kernel: ppc0: [ITHREAD]
Apr  4 22:22:43 bfmb kernel: atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
Apr  4 22:22:43 bfmb kernel: atkbd0: <AT Keyboard> irq 1 on atkbdc0
Apr  4 22:22:43 bfmb kernel: kbd0 at atkbd0
Apr  4 22:22:43 bfmb kernel: atkbd0: [GIANT-LOCKED]
Apr  4 22:22:43 bfmb kernel: atkbd0: [ITHREAD]
Apr  4 22:22:43 bfmb kernel: sio0: configured irq 4 not in bitmap of probed irqs 0
Apr  4 22:22:43 bfmb kernel: sio0: port may not be enabled
Apr  4 22:22:43 bfmb kernel: sio0: configured irq 4 not in bitmap of probed irqs 0
Apr  4 22:22:43 bfmb kernel: sio0: port may not be enabled
Apr  4 22:22:43 bfmb kernel: sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
Apr  4 22:22:43 bfmb kernel: sio0: type 16550A
Apr  4 22:22:43 bfmb kernel: sio0: [FILTER]
Apr  4 22:22:43 bfmb kernel: cpu0: <ACPI CPU> on acpi0
Apr  4 22:22:43 bfmb kernel: acpi_throttle0: <ACPI CPU Throttling> on cpu0
Apr  4 22:22:43 bfmb kernel: powernow0: <Cool`n'Quiet K8> on cpu0
Apr  4 22:22:43 bfmb kernel: device_attach: powernow0 attach returned 6
Apr  4 22:22:43 bfmb kernel: cpu1: <ACPI CPU> on acpi0
Apr  4 22:22:43 bfmb kernel: powernow1: <Cool`n'Quiet K8> on cpu1
Apr  4 22:22:43 bfmb kernel: device_attach: powernow1 attach returned 6
Apr  4 22:22:43 bfmb kernel: orm0: <ISA Option ROM> at iomem 0xc0000-0xcefff on isa0
Apr  4 22:22:43 bfmb kernel: sc0: <System console> at flags 0x100 on isa0
Apr  4 22:22:43 bfmb kernel: sc0: VGA <16 virtual consoles, flags=0x300>
Apr  4 22:22:43 bfmb kernel: sio1: configured irq 3 not in bitmap of probed irqs 0
Apr  4 22:22:43 bfmb kernel: sio1: port may not be enabled
Apr  4 22:22:43 bfmb kernel: vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Apr  4 22:22:43 bfmb kernel: Timecounters tick every 1.000 msec
Apr  4 22:22:43 bfmb kernel: ipfw2 (+ipv6) initialized, divert loadable, nat loadable, rule-based forwarding disabled, default to accept, logging limited to 5 packets/entry by defaultfirewire0: 1 nodes, maxhop <= 0, cable IRM = 0 (me)
Apr  4 22:22:43 bfmb kernel: firewire0: bus manager 0 (me)
Apr  4 22:22:43 bfmb kernel:
Apr  4 22:22:43 bfmb kernel: md0: Preloaded image </mfsroot> 94371840 bytes at 0xffffffff80d64b28
Apr  4 22:22:43 bfmb kernel: GEOM_LABEL: Label for provider md0 is ufsid/4a3d467f195889fc.
Apr  4 22:22:43 bfmb kernel: ad4: 1430799MB <Seagate ST31500341AS CC1H> at ata2-master SATA300
Apr  4 22:22:43 bfmb kernel: ad6: 1430799MB <Seagate ST31500341AS CC1H> at ata3-master SATA300
Apr  4 22:22:43 bfmb kernel: ad8: 1430799MB <Seagate ST31500341AS CC1H> at ata4-master SATA300
Apr  4 22:22:43 bfmb kernel: ad10: 1430799MB <WDC WD15EARX-00PASB0 51.0AB51> at ata5-master SATA150
Apr  4 22:22:43 bfmb kernel: SMP: AP CPU #1 Launched!
Apr  4 22:22:43 bfmb kernel: da0 at umass-sim0 bus 0 target 0 lun 0
Apr  4 22:22:43 bfmb kernel: da0: <Kingston DataTraveler 2.0 PMAP> Removable Direct Access SCSI-0 device
Apr  4 22:22:43 bfmb kernel: da0: 40.000MB/s transfers
Apr  4 22:22:43 bfmb kernel: da0: 477MB (977664 512 byte sectors: 64H 32S/T 477C)
Apr  4 22:22:43 bfmb kernel: GEOM: da0: the secondary GPT table is corrupt or invalid.
Apr  4 22:22:43 bfmb kernel: GEOM: da0: using the primary only -- recovery suggested.
Apr  4 22:22:43 bfmb kernel: GEOM_LABEL: Label for provider da0s2 is ufsid/4a4964cae6af1904.
Apr  4 22:22:43 bfmb kernel: GEOM_LABEL: Label for provider da0s2 is ufs/data.
Apr  4 22:22:43 bfmb kernel: GEOM_LABEL: Label for provider da0s1a is ufsid/4a3d46a39f530e83.
Apr  4 22:22:43 bfmb kernel: Trying to mount root from ufs:/dev/md0
Apr  4 22:22:43 bfmb kernel: GEOM_LABEL: Label ufsid/4a3d467f195889fc removed.
Apr  4 22:22:43 bfmb kernel: This module (opensolaris) contains code covered by the
Apr  4 22:22:43 bfmb kernel: Common Development and Distribution License (CDDL)
Apr  4 22:22:43 bfmb kernel: see http://opensolaris.org/os/licensing/opensolaris_license/
Apr  4 22:22:43 bfmb kernel: WARNING: ZFS is considered to be an experimental feature in FreeBSD.
Apr  4 22:22:43 bfmb kernel: ZFS filesystem version 6
Apr  4 22:22:43 bfmb kernel: ZFS storage pool version 6
Apr  4 22:22:43 bfmb kernel: GEOM_LABEL: Label for provider md1 is ufsid/4614251a2c631e23.
Apr  4 22:22:43 bfmb kernel: GEOM_LABEL: Label ufsid/4614251a2c631e23 removed.
Apr  4 22:22:43 bfmb afpd[1707]: main: atp_open: Protocol not supported
Apr  4 22:22:44 bfmb smbd[1842]: [2007/04/04 22:22:44, 0] passdb/pdb_smbpasswd.c:startsmbfilepwent(241)
Apr  4 22:22:44 bfmb smbd[1842]:   startsmbfilepwent_internal: file /var/etc/private/smbpasswd did not exist. File successfully created.
Apr  4 22:22:44 bfmb smbd[1842]: [2007/04/04 22:22:44, 1] lib/account_pol.c:account_policy_get(286)
Apr  4 22:22:44 bfmb smbd[1842]:   account_policy_get: tdb_fetch_uint32 failed for field 1 (min password length), returning 0
Apr  4 22:22:44 bfmb smbd[1842]: [2007/04/04 22:22:44, 1] lib/account_pol.c:account_policy_get(286)
Apr  4 22:22:44 bfmb smbd[1842]:   account_policy_get: tdb_fetch_uint32 failed for field 2 (password history), returning 0
Apr  4 22:22:44 bfmb smbd[1842]: [2007/04/04 22:22:44, 1] lib/account_pol.c:account_policy_get(286)
Apr  4 22:22:44 bfmb smbd[1842]:   account_policy_get: tdb_fetch_uint32 failed for field 3 (user must logon to change password), returning 0
Apr  4 22:22:44 bfmb smbd[1842]: [2007/04/04 22:22:44, 1] lib/account_pol.c:account_policy_get(286)
Apr  4 22:22:44 bfmb smbd[1842]:   account_policy_get: tdb_fetch_uint32 failed for field 4 (maximum password age), returning 0
Apr  4 22:22:44 bfmb smbd[1842]: [2007/04/04 22:22:44, 1] lib/account_pol.c:account_policy_get(286)
Apr  4 22:22:44 bfmb smbd[1842]:   account_policy_get: tdb_fetch_uint32 failed for field 5 (minimum password age), returning 0
Apr  4 22:22:44 bfmb smbd[1842]: [2007/04/04 22:22:44, 1] lib/account_pol.c:account_policy_get(286)
Apr  4 22:22:44 bfmb smbd[1842]:   account_policy_get: tdb_fetch_uint32 failed for field 6 (lockout duration), returning 0
Apr  4 22:22:44 bfmb smbd[1842]: [2007/04/04 22:22:44, 1] lib/account_pol.c:account_policy_get(286)
Apr  4 22:22:44 bfmb smbd[1842]:   account_policy_get: tdb_fetch_uint32 failed for field 7 (reset count minutes), returning 0
Apr  4 22:22:44 bfmb smbd[1842]: [2007/04/04 22:22:44, 1] lib/account_pol.c:account_policy_get(286)
Apr  4 22:22:44 bfmb smbd[1842]:   account_policy_get: tdb_fetch_uint32 failed for field 8 (bad lockout attempt), returning 0
Apr  4 22:22:44 bfmb smbd[1842]: [2007/04/04 22:22:44, 1] lib/account_pol.c:account_policy_get(286)
Apr  4 22:22:44 bfmb smbd[1842]:   account_policy_get: tdb_fetch_uint32 failed for field 9 (disconnect time), returning 0
Apr  4 22:22:44 bfmb smbd[1842]: [2007/04/04 22:22:44, 1] lib/account_pol.c:account_policy_get(286)
Apr  4 22:22:44 bfmb smbd[1842]:   account_policy_get: tdb_fetch_uint32 failed for field 10 (refuse machine password change), returning 0
Apr  4 22:22:53 bfmb smartd[1899]: Device: /dev/ad6, 24 Currently unreadable (pending) sectors
Apr  4 22:22:53 bfmb smartd[1899]: Device: /dev/ad6, 24 Offline uncorrectable sectors
Apr  4 22:22:56 bfmb smartd[1899]: Device: /dev/ad10, 2 Currently unreadable (pending) sectors
Apr  4 22:22:58 bfmb lighttpd[2059]: (log.c.97) server started
Apr  4 22:22:58 bfmb login: login on ttyv0 as root
Apr  4 22:23:47 bfmb smartd[1901]: Device: /dev/ad6, 24 Currently unreadable (pending) sectors
Apr  4 22:23:47 bfmb smartd[1901]: Device: /dev/ad6, 24 Offline uncorrectable sectors
Apr  4 22:23:54 bfmb smartd[1901]: Device: /dev/ad10, 2 Currently unreadable (pending) sectors
Apr  4 22:24:38 bfmb smartd[1901]: Device: /dev/ad6, 24 Currently unreadable (pending) sectors
Apr  4 22:24:38 bfmb smartd[1901]: Device: /dev/ad6, 24 Offline uncorrectable sectors
Apr  4 22:24:44 bfmb smartd[1901]: Device: /dev/ad10, 2 Currently unreadable (pending) sectors
Apr  4 22:25:28 bfmb smartd[1901]: Device: /dev/ad6, 24 Currently unreadable (pending) sectors
Apr  4 22:25:28 bfmb smartd[1901]: Device: /dev/ad6, 24 Offline uncorrectable sectors
Apr  4 22:25:29 bfmb smartd[1901]: Device: /dev/ad10, 2 Currently unreadable (pending) sectors
Apr  4 22:26:17 bfmb smartd[1901]: Device: /dev/ad6, 24 Currently unreadable (pending) sectors
Apr  4 22:26:17 bfmb smartd[1901]: Device: /dev/ad6, 24 Offline uncorrectable sectors
Apr  4 22:26:24 bfmb smartd[1901]: Device: /dev/ad10, 2 Currently unreadable (pending) sectors
Apr  4 22:27:08 bfmb smartd[1901]: Device: /dev/ad6, 24 Currently unreadable (pending) sectors
Apr  4 22:27:08 bfmb smartd[1901]: Device: /dev/ad6, 24 Offline uncorrectable sectors
Apr  4 22:27:14 bfmb smartd[1901]: Device: /dev/ad10, 2 Currently unreadable (pending) sectors
Apr  4 22:27:58 bfmb smartd[1901]: Device: /dev/ad6, 24 Currently unreadable (pending) sectors
Apr  4 22:27:58 bfmb smartd[1901]: Device: /dev/ad6, 24 Offline uncorrectable sectors
Apr  4 22:27:59 bfmb smartd[1901]: Device: /dev/ad10, 2 Currently unreadable (pending) sectors

Post by **b0ssman** » 01 Jun 2013 20:25

first stop trying to boot that system.

buy 2 new harddrives.

then this is just an idea of mine, but lets wait for other peoples ideas.

take ad6 and ad10 out into another machine and use dd with the noerror option to copy the failing hd to a new one.
if that works put the new drives into the old system and do a scrub.

Lee Sharp · Post by **Lee Sharp** » 02 Jun 2013 06:20

Pull all the drives, and boot the system. Run memtest. Run a burn in prime test. Not booting successfully from a CD tells me that your hardware may be faulty, not your drives. But if you keep booting and crashing on faulty hardware, the drives will join it soon enough.

idmah · Post by **idmah** » 02 Jun 2013 19:46

This machine has been giving me problems for a while now even when it was "working",
YES should have backed up.. sigh...

NOW thinking I'm going to scrap this machine and replace it. with a "new one"
What's the best way to hopefully retain/recover the ZFS Pool and save my Files?

I like b0ssman's idea.. what's the best way to do this on a different machine?
put the ZFS drives in the machine. boot the Nas4Free CD and do DD to new drives?
a idiot's guide would help.. cuz well I'm an idiot... hahah!!!

** I do have another PC I can access for recovery.. which I know works..

ps. I will run a check before machine tho, scrapping feels safest tho.. but just to make sure. HIREN'S USB I guess ??

Lee Sharp · Post by **Lee Sharp** » 03 Jun 2013 23:42

First, figure out what is broke, and fix that. Do not fix what ain't broke. You have an admittedly unstable platform. That means that "bad drives" may or may not be bad.

If you have another system handy, put the drives in it, and try to see what happens.

Or, do what I mentioned first, and test your base hardware.

idmah · Post by **idmah** » 06 Jun 2013 18:23

Well put the drives in another computer, plugged in the USB-boot key.
and got error:

fatal trap 18: integer divide fault while in kernal mode
cpuid =0; apic id = 00
[a bunch of stack pointer stuff]
trap number
panic:integer divide fault
cpuid = 0
*** and then it reboots.

Going to try booting from cd next.

ANY help would be greatly appreciated!!!
*** Just want to get my stuff off these drives more than anything else. ***
thanks
Ian

ps guessing.. Is this because 0.7.1 doesn't support an i7 ? grasping at straws here..

Post by **b0ssman** » 06 Jun 2013 18:43

have you tried a recent version of nas4free? you know you can import the zfs pool in nas4free as well

idmah · Post by **idmah** » 11 Jun 2013 19:35

Building a new machine today or tomorrow...
if I pull all the drives put them in the new machine. Boot with the nas4free CD
and import for the original USB Boot key.*(how do I do this from the web GUI)
will the Scrub of the Drives continue? or should I somehow import the 2 drives that
I know are working and DD to the brand new drives then scrub? will that even work?

** Basically From the mess I have.. How do I put it all back on a new machine, and get most of the files back?
Sorry if this is nubey but well Been fighting with this for 4 months now.
thanks
Ian

Post by **b0ssman** » 11 Jun 2013 20:14

check out this tool

http://www.gnu.org/software/ddrescue/ddrescue.html

Post by **kkd** » 14 Jun 2013 21:28

download clonezilla and clone ur bad hhd to a new one. try to buy same hdd (size, manufacturer, ...)

idmah · Post by **idmah** » 22 Jun 2013 17:02

Hi Kkd.

Ok feel like a total "nube" here but ran Clonezilla
started disk to disk cloning but get an error I think "reading source."
when I get home I'll try to reproduce it. but..

The Total nube question.

There is a new unformatted drive as the destination, do I have to formate it ZFS and
how do I do that in Clonezilla ?

or is there some other settings to read ZFS ?
what am I missing?

Sorry if this is a stupid question. All this stuff in new to me.
thanks
Ian

Post by **raulfg3** » 22 Jun 2013 18:00

idmah wrote:There is a new unformatted drive as the destination, do I have to formate it ZFS and
how do I do that in Clonezilla ?

No, you didn't need to format new drive, you need to clone old disk in a new disk, you can use clonezilla or other clonning apps, like AcronisTrueImage, Ghost. or "dd" this last is included on Nas4Free and it's free, but you need to read and understand how it works:

http://linuxzone.es/dd-clona-y-graba-di ... acilmente/

http://en.wikipedia.org/wiki/Dd_%28Unix%29

idmah · Post by **idmah** » 24 Jun 2013 15:39

in CloneZillia I used:
1.Device - Device
2. Beginner mode:
3. Selected the source Drive.
4. Selected the destination Drive.

** Got: "Unknown partition table format fo file /tmp/ocs_onthefly_local.0NbQmN/src-pt.parted!"
which I have no idea what this means or how to fix it. sorry for being stupid..

thanks
Ian

Post by **raulfg3** » 24 Jun 2013 21:56

Sorry, but I'm out of ideas, perhaps is time to lose data or ask to some friends that can help you.

Do a clone & recover from a forum , without direct acces to fault machine is a hard work.

Post by **b0ssman** » 24 Jun 2013 22:00

try ddrescue or dd with noerror

idmah · Post by **idmah** » 25 Jun 2013 16:42

ok DD ing with "sudo dd bs=512 if=/dev/sda of=/dev/sdb/ conv=noerror,sync"
but with bs=512 and it's a 1.5 tb on an core-duo 2.9.
Just wondering how long that should take?

I also don't see any output .. is that good ? bad? or standard?
Again sorry all the dumb questions this stuff is new to me.

thanks

Post by **b0ssman** » 25 Jun 2013 21:54

no output is normal.

any disc errors will prob pop up in dmesg output.

something like 5-10 hours should be expected.

idmah · Post by **idmah** » 29 Jun 2013 19:25

Saga continues.

So looks like now my original key has failed also.
So in a last ditch effort to get everything/anything back.

1. I cloned one of the bad drives. ad6
2. but ad10 epic failed. hahaa!!! sigh!! dd couldn't see it.
3. So have the clone of ad6, ad4, ad8 in the new machine.

I'm planning to boot off the CD, and then create a new USB-embeded Nas4Free with the current version,
I have a year old config file from the pervious install. Which I will import to the new install. **Do I need to rename freenas to nas4free in XML file???)

and then try to "zpool import <tank>"

So now some questions.
a. Do I put ad10 old busted in or just use the three and replace the ad10 with a new drive later?
b. What order should I do the above, import XML and then zpool? or the other way around.
c. I assume I should scrub the ad6 cloned drive to fix any of the previous errors?

Thanks this has been a scary experience all your help, has given me a modicum of confidence
to survive this,and is Very much appreciated !!!

Ian

Post by **b0ssman** » 29 Jun 2013 19:32

worry about the config later.

right now connect the drives except ad10 plus you replacement.

boot from cd and resilver the pool.
http://wiki.nas4free.org/doku.php?id=faq:0149

when you data has been restored then set up your server.

idmah · Post by **idmah** » 02 Jul 2013 17:31

UGH!!!

So I was, WAS optimistic when I booted the system with the cd everything seemed to be working
it continued to replace the drive it believed to be ad8 now called ada1.. but around 87% it hits an error.

Code: Select all

ERRROR:  Permanent errors has been detected in the following files: 
/mnt/ZFS_Pool/Movies and TV  shows/TV shows Xvids/Erureka/Erureka.S04E10.avi

The error is a read error on ada0.
is there a way to abort the resilvering, and or fix or skip the offending file?
or Should could I, abort resilver and then scrub ada0 ?

** I think this is the file that caused the whole mess, in the first place.
I was copying from a laptop who's battery died while copy was happening.

Here is a log:

Code: Select all

Jun 30 23:53:26 nas4free syslogd: kernel boot file is /boot/kernel/kernel
Jun 30 23:53:26 nas4free kernel: Copyright (c) 1992-2012 The FreeBSD Project.
Jun 30 23:53:26 nas4free kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Jun 30 23:53:26 nas4free kernel: The Regents of the University of California. All rights reserved.
Jun 30 23:53:26 nas4free kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
Jun 30 23:53:26 nas4free kernel: FreeBSD 9.1-RELEASE-p3 #0 r251670M: Thu Jun 13 01:59:24 CEST 2013
Jun 30 23:53:26 nas4free kernel: root@dev.nas4free.org:/usr/obj/nas4free/usr/src/sys/NAS4FREE-amd64 amd64
Jun 30 23:53:26 nas4free kernel: CPU: Intel(R) Pentium(R) CPU G645 @ 2.90GHz (2893.48-MHz K8-class CPU)
Jun 30 23:53:26 nas4free kernel: Origin = "GenuineIntel"  Id = 0x206a7  Family = 6  Model = 2a  Stepping = 7
Jun 30 23:53:26 nas4free kernel: Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Jun 30 23:53:26 nas4free kernel: Features2=0xd9ae3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,POPCNT,TSCDLT,XSAVE,OSXSAVE>
Jun 30 23:53:26 nas4free kernel: AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
Jun 30 23:53:26 nas4free kernel: AMD Features2=0x1<LAHF>
Jun 30 23:53:26 nas4free kernel: TSC: P-state invariant, performance statistics
Jun 30 23:53:26 nas4free kernel: real memory  = 8589934592 (8192 MB)
Jun 30 23:53:26 nas4free kernel: avail memory = 7914450944 (7547 MB)
Jun 30 23:53:26 nas4free kernel: Event timer "LAPIC" quality 600
Jun 30 23:53:26 nas4free kernel: ACPI APIC Table: <INTEL  DZ68BC  >
Jun 30 23:53:26 nas4free kernel: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
Jun 30 23:53:26 nas4free kernel: FreeBSD/SMP: 1 package(s) x 2 core(s)
Jun 30 23:53:26 nas4free kernel: cpu0 (BSP): APIC ID:  0
Jun 30 23:53:26 nas4free kernel: cpu1 (AP): APIC ID:  2
Jun 30 23:53:26 nas4free kernel: ioapic0 <Version 2.0> irqs 0-23 on motherboard
Jun 30 23:53:26 nas4free kernel: kbd1 at kbdmux0
Jun 30 23:53:26 nas4free kernel: cryptosoft0: <software crypto> on motherboard
Jun 30 23:53:26 nas4free kernel: acpi0: <INTEL DZ68BC> on motherboard
Jun 30 23:53:26 nas4free kernel: acpi0: Power Button (fixed)
Jun 30 23:53:26 nas4free kernel: cpu0: <ACPI CPU> on acpi0
Jun 30 23:53:26 nas4free kernel: cpu1: <ACPI CPU> on acpi0
Jun 30 23:53:26 nas4free kernel: attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Jun 30 23:53:26 nas4free kernel: Timecounter "i8254" frequency 1193182 Hz quality 0
Jun 30 23:53:26 nas4free kernel: Event timer "i8254" frequency 1193182 Hz quality 100
Jun 30 23:53:26 nas4free kernel: atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Jun 30 23:53:26 nas4free kernel: Event timer "RTC" frequency 32768 Hz quality 0
Jun 30 23:53:26 nas4free kernel: hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Jun 30 23:53:26 nas4free kernel: Timecounter "HPET" frequency 14318180 Hz quality 950
Jun 30 23:53:26 nas4free kernel: Event timer "HPET" frequency 14318180 Hz quality 550
Jun 30 23:53:26 nas4free kernel: Event timer "HPET1" frequency 14318180 Hz quality 440
Jun 30 23:53:26 nas4free kernel: Event timer "HPET2" frequency 14318180 Hz quality 440
Jun 30 23:53:26 nas4free kernel: Event timer "HPET3" frequency 14318180 Hz quality 440
Jun 30 23:53:26 nas4free kernel: Event timer "HPET4" frequency 14318180 Hz quality 440
Jun 30 23:53:26 nas4free kernel: Event timer "HPET5" frequency 14318180 Hz quality 440
Jun 30 23:53:26 nas4free kernel: Event timer "HPET6" frequency 14318180 Hz quality 440
Jun 30 23:53:26 nas4free kernel: Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
Jun 30 23:53:26 nas4free kernel: acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
Jun 30 23:53:26 nas4free kernel: pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
Jun 30 23:53:26 nas4free kernel: pci0: <ACPI PCI bus> on pcib0
Jun 30 23:53:26 nas4free kernel: vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 0xfdc00000-0xfdffffff,0xe0000000-0xefffffff irq 16 at device 2.0 on pci0
Jun 30 23:53:26 nas4free kernel: agp0: <SandyBridge desktop GT1 IG> on vgapci0
Jun 30 23:53:26 nas4free kernel: agp0: aperture size is 256M, detected 65532k stolen memory
Jun 30 23:53:26 nas4free kernel: pci0: <simple comms> at device 22.0 (no driver attached)
Jun 30 23:53:26 nas4free kernel: em0: <Intel(R) PRO/1000 Network Connection 7.3.2> port 0xf080-0xf09f mem 0xfe500000-0xfe51ffff,0xfe528000-0xfe528fff irq 20 at device 25.0 on pci0
Jun 30 23:53:26 nas4free kernel: em0: Using an MSI interrupt
Jun 30 23:53:26 nas4free kernel: em0: Ethernet address: 38:60:77:7c:0e:a1
Jun 30 23:53:26 nas4free kernel: ehci0: <EHCI (generic) USB 2.0 controller> mem 0xfe527000-0xfe5273ff irq 16 at device 26.0 on pci0
Jun 30 23:53:26 nas4free kernel: usbus0: EHCI version 1.0
Jun 30 23:53:26 nas4free kernel: usbus0 on ehci0
Jun 30 23:53:26 nas4free kernel: hdac0: <Intel Cougar Point HDA Controller> mem 0xfe520000-0xfe523fff irq 22 at device 27.0 on pci0
Jun 30 23:53:26 nas4free kernel: pcib1: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
Jun 30 23:53:26 nas4free kernel: pci1: <ACPI PCI bus> on pcib1
Jun 30 23:53:26 nas4free kernel: pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1
Jun 30 23:53:26 nas4free kernel: pci2: <ACPI PCI bus> on pcib2
Jun 30 23:53:26 nas4free kernel: fwohci0: <Texas Instruments TSB43AB22/A> mem 0xfe404000-0xfe4047ff,0xfe400000-0xfe403fff irq 19 at device 3.0 on pci2
Jun 30 23:53:26 nas4free kernel: fwohci0: OHCI version 1.10 (ROM=1)
Jun 30 23:53:26 nas4free kernel: fwohci0: No. of Isochronous channels is 4.
Jun 30 23:53:26 nas4free kernel: fwohci0: EUI64 e0:cb:4e:00:00:05:7f:f4
Jun 30 23:53:26 nas4free kernel: fwohci0: Phy 1394a available S400, 2 ports.
Jun 30 23:53:26 nas4free kernel: fwohci0: Link S400, max_rec 2048 bytes.
Jun 30 23:53:26 nas4free kernel: firewire0: <IEEE1394(FireWire) bus> on fwohci0
Jun 30 23:53:26 nas4free kernel: fwip0: <IP over FireWire> on firewire0
Jun 30 23:53:26 nas4free kernel: fwip0: Firewire address: e0:cb:4e:00:00:05:7f:f4 @ 0xfffe00000000, S400, maxrec 2048
Jun 30 23:53:26 nas4free kernel: dcons_crom0: <dcons configuration ROM> on firewire0
Jun 30 23:53:26 nas4free kernel: dcons_crom0: bus_addr 0xda700000
Jun 30 23:53:26 nas4free kernel: fwe0: <Ethernet over FireWire> on firewire0
Jun 30 23:53:26 nas4free kernel: if_fwe0: Fake Ethernet address: e2:cb:4e:05:7f:f4
Jun 30 23:53:26 nas4free kernel: fwe0: Ethernet address: e2:cb:4e:05:7f:f4
Jun 30 23:53:26 nas4free kernel: fwohci0: Initiate bus reset
Jun 30 23:53:26 nas4free kernel: fwohci0: fwohci_intr_core: BUS reset
Jun 30 23:53:26 nas4free kernel: fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode
Jun 30 23:53:26 nas4free kernel: pcib3: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
Jun 30 23:53:26 nas4free kernel: pci3: <ACPI PCI bus> on pcib3
Jun 30 23:53:26 nas4free kernel: xhci0: <NEC uPD720200 USB 3.0 controller> mem 0xfe300000-0xfe301fff irq 19 at device 0.0 on pci3
Jun 30 23:53:26 nas4free kernel: xhci0: 32 byte context size.
Jun 30 23:53:26 nas4free kernel: usbus1 on xhci0
Jun 30 23:53:26 nas4free kernel: pcib4: <ACPI PCI-PCI bridge> irq 17 at device 28.4 on pci0
Jun 30 23:53:26 nas4free kernel: pci4: <ACPI PCI bus> on pcib4
Jun 30 23:53:26 nas4free kernel: xhci1: <NEC uPD720200 USB 3.0 controller> mem 0xfe200000-0xfe201fff irq 16 at device 0.0 on pci4
Jun 30 23:53:26 nas4free kernel: xhci1: 32 byte context size.
Jun 30 23:53:26 nas4free kernel: usbus2 on xhci1
Jun 30 23:53:26 nas4free kernel: pcib5: <ACPI PCI-PCI bridge> irq 18 at device 28.6 on pci0
Jun 30 23:53:26 nas4free kernel: pci5: <ACPI PCI bus> on pcib5
Jun 30 23:53:26 nas4free kernel: atapci0: <Marvell ATA controller> port 0xe040-0xe047,0xe030-0xe033,0xe020-0xe027,0xe010-0xe013,0xe000-0xe00f mem 0xfe110000-0xfe1107ff irq 18 at device 0.0 on pci5
Jun 30 23:53:26 nas4free kernel: ata2: <ATA channel> at channel 0 on atapci0
Jun 30 23:53:26 nas4free kernel: ata3: <ATA channel> at channel 1 on atapci0
Jun 30 23:53:26 nas4free kernel: pcib6: <ACPI PCI-PCI bridge> irq 19 at device 28.7 on pci0
Jun 30 23:53:26 nas4free kernel: pci6: <ACPI PCI bus> on pcib6
Jun 30 23:53:26 nas4free kernel: atapci1: <Marvell ATA controller> port 0xd040-0xd047,0xd030-0xd033,0xd020-0xd027,0xd010-0xd013,0xd000-0xd00f mem 0xfe010000-0xfe0107ff irq 19 at device 0.0 on pci6
Jun 30 23:53:26 nas4free kernel: ata4: <ATA channel> at channel 0 on atapci1
Jun 30 23:53:26 nas4free kernel: ata5: <ATA channel> at channel 1 on atapci1
Jun 30 23:53:26 nas4free kernel: ehci1: <EHCI (generic) USB 2.0 controller> mem 0xfe526000-0xfe5263ff irq 23 at device 29.0 on pci0
Jun 30 23:53:26 nas4free kernel: usbus3: EHCI version 1.0
Jun 30 23:53:26 nas4free kernel: usbus3 on ehci1
Jun 30 23:53:26 nas4free kernel: isab0: <PCI-ISA bridge> at device 31.0 on pci0
Jun 30 23:53:26 nas4free kernel: isa0: <ISA bus> on isab0
Jun 30 23:53:26 nas4free kernel: ahci0: <Intel Cougar Point AHCI SATA controller> port 0xf0d0-0xf0d7,0xf0c0-0xf0c3,0xf0b0-0xf0b7,0xf0a0-0xf0a3,0xf060-0xf07f mem 0xfe525000-0xfe5257ff irq 19 at device 31.2 on pci0
Jun 30 23:53:26 nas4free kernel: ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
Jun 30 23:53:26 nas4free kernel: ahcich3: <AHCI channel> at channel 3 on ahci0
Jun 30 23:53:26 nas4free kernel: ahcich4: <AHCI channel> at channel 4 on ahci0
Jun 30 23:53:26 nas4free kernel: ahcich5: <AHCI channel> at channel 5 on ahci0
Jun 30 23:53:26 nas4free kernel: pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
Jun 30 23:53:26 nas4free kernel: acpi_button0: <Power Button> on acpi0
Jun 30 23:53:26 nas4free kernel: uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
Jun 30 23:53:26 nas4free kernel: orm0: <ISA Option ROM> at iomem 0xd2800-0xd37ff on isa0
Jun 30 23:53:26 nas4free kernel: sc0: <System console> at flags 0x100 on isa0
Jun 30 23:53:26 nas4free kernel: sc0: VGA <16 virtual consoles, flags=0x300>
Jun 30 23:53:26 nas4free kernel: vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Jun 30 23:53:26 nas4free kernel: atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
Jun 30 23:53:26 nas4free kernel: atkbd0: <AT Keyboard> irq 1 on atkbdc0
Jun 30 23:53:26 nas4free kernel: kbd0 at atkbd0
Jun 30 23:53:26 nas4free kernel: atkbd0: [GIANT-LOCKED]
Jun 30 23:53:26 nas4free kernel: ppc0: cannot reserve I/O port range
Jun 30 23:53:26 nas4free kernel: ctl: CAM Target Layer loaded
Jun 30 23:53:26 nas4free kernel: coretemp0: <CPU On-Die Thermal Sensors> on cpu0
Jun 30 23:53:26 nas4free kernel: est0: <Enhanced SpeedStep Frequency Control> on cpu0
Jun 30 23:53:26 nas4free kernel: p4tcc0: <CPU Frequency Thermal Control> on cpu0
Jun 30 23:53:26 nas4free kernel: coretemp1: <CPU On-Die Thermal Sensors> on cpu1
Jun 30 23:53:26 nas4free kernel: est1: <Enhanced SpeedStep Frequency Control> on cpu1
Jun 30 23:53:26 nas4free kernel: p4tcc1: <CPU Frequency Thermal Control> on cpu1
Jun 30 23:53:26 nas4free kernel: ZFS filesystem version 5
Jun 30 23:53:26 nas4free kernel: ZFS storage pool version 28
Jun 30 23:53:26 nas4free kernel: Timecounters tick every 1.000 msec
Jun 30 23:53:26 nas4free kernel: firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0)  (me)
Jun 30 23:53:26 nas4free kernel: firewire0: bus manager 0
Jun 30 23:53:26 nas4free kernel: ipfw2 (+ipv6) initialized, divert loadable, nat loadable, rule-based forwarding disabled, default to accept, logging disabled
Jun 30 23:53:26 nas4free kernel: iSCSI boot driver version 0.2.6
Jun 30 23:53:26 nas4free kernel: hdacc0: <Realtek ALC892 HDA CODEC> at cad 0 on hdac0
Jun 30 23:53:26 nas4free kernel: hdaa0: <Realtek ALC892 Audio Function Group> at nid 1 on hdacc0
Jun 30 23:53:26 nas4free kernel: pcm0: <Realtek ALC892 (Rear Analog 5.1/2.0)> at nid 20,22,21 and 24,26 on hdaa0
Jun 30 23:53:26 nas4free kernel: pcm1: <Realtek ALC892 (Front Analog)> at nid 27 and 25 on hdaa0
Jun 30 23:53:26 nas4free kernel: pcm2: <Realtek ALC892 (Rear Digital)> at nid 30 on hdaa0
Jun 30 23:53:26 nas4free kernel: pcm3: <Realtek ALC892 (Onboard Digital)> at nid 17 on hdaa0
Jun 30 23:53:26 nas4free kernel: hdacc1: <Intel Cougar Point HDA CODEC> at cad 3 on hdac0
Jun 30 23:53:26 nas4free kernel: hdaa1: <Intel Cougar Point Audio Function Group> at nid 1 on hdacc1
Jun 30 23:53:26 nas4free kernel: pcm4: <Intel Cougar Point (HDMI/DP 8ch)> at nid 6 on hdaa1
Jun 30 23:53:26 nas4free kernel: pcm5: <Intel Cougar Point (HDMI/DP 8ch)> at nid 7 on hdaa1
Jun 30 23:53:26 nas4free kernel: md0: Preloaded image </mfsroot> 232783872 bytes at 0xffffffff819a3c58
Jun 30 23:53:26 nas4free kernel: usbus0: 480Mbps High Speed USB v2.0
Jun 30 23:53:26 nas4free kernel: usbus1: 5.0Gbps Super Speed USB v3.0
Jun 30 23:53:26 nas4free kernel: usbus2: 5.0Gbps Super Speed USB v3.0
Jun 30 23:53:26 nas4free kernel: usbus3: 480Mbps High Speed USB v2.0
Jun 30 23:53:26 nas4free kernel: ugen0.1: <Intel> at usbus0
Jun 30 23:53:26 nas4free kernel: uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
Jun 30 23:53:26 nas4free kernel: ugen1.1: <0x1033> at usbus1
Jun 30 23:53:26 nas4free kernel: uhub1: <0x1033 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1
Jun 30 23:53:26 nas4free kernel: ugen2.1: <0x1033> at usbus2
Jun 30 23:53:26 nas4free kernel: uhub2: <0x1033 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus2
Jun 30 23:53:26 nas4free kernel: ugen3.1: <Intel> at usbus3
Jun 30 23:53:26 nas4free kernel: uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
Jun 30 23:53:26 nas4free kernel: uhub1: 4 ports with 4 removable, self powered
Jun 30 23:53:26 nas4free kernel: uhub2: 4 ports with 4 removable, self powered
Jun 30 23:53:26 nas4free kernel: uhub0: 2 ports with 2 removable, self powered
Jun 30 23:53:26 nas4free kernel: uhub3: 2 ports with 2 removable, self powered
Jun 30 23:53:26 nas4free kernel: ugen0.2: <vendor 0x8087> at usbus0
Jun 30 23:53:26 nas4free kernel: uhub4: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus0
Jun 30 23:53:26 nas4free kernel: ugen3.2: <vendor 0x8087> at usbus3
Jun 30 23:53:26 nas4free kernel: uhub5: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus3
Jun 30 23:53:26 nas4free kernel: uhub4: 6 ports with 6 removable, self powered
Jun 30 23:53:26 nas4free kernel: uhub5: 8 ports with 8 removable, self powered
Jun 30 23:53:26 nas4free kernel: ugen0.3: <vendor 0x0510> at usbus0
Jun 30 23:53:26 nas4free kernel: ukbd0: <USB Mac Keyboard> on usbus0
Jun 30 23:53:26 nas4free kernel: kbd2 at ukbd0
Jun 30 23:53:26 nas4free kernel: uhid0: <vendor 0x0510 Joint Mac Keyboard, class 0/0, rev 1.10/1.00, addr 3> on usbus0
Jun 30 23:53:26 nas4free kernel: run_interrupt_driven_hooks: still waiting after 60 seconds for xpt_config
Jun 30 23:53:26 nas4free kernel: ada0 at ahcich3 bus 0 scbus4 target 0 lun 0
Jun 30 23:53:26 nas4free kernel: ada0: <ST31500341AS CC1H> ATA-8 SATA 2.x device
Jun 30 23:53:26 nas4free kernel: ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
Jun 30 23:53:26 nas4free kernel: ada0: Command Queueing enabled
Jun 30 23:53:26 nas4free kernel: ada0: 1430799MB (2930277168 512 byte sectors: 16H 63S/T 16383C)
Jun 30 23:53:26 nas4free kernel: ada0: Previously was known as ad12
Jun 30 23:53:26 nas4free kernel: ada1 at ahcich4 bus 0 scbus5 target 0 lun 0
Jun 30 23:53:26 nas4free kernel: ada1: <ST31500341AS CC1H> ATA-8 SATA 2.x device
Jun 30 23:53:26 nas4free kernel: ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
Jun 30 23:53:26 nas4free kernel: ada1: Command Queueing enabled
Jun 30 23:53:26 nas4free kernel: ada1: 1430799MB (2930277168 512 byte sectors: 16H 63S/T 16383C)
Jun 30 23:53:26 nas4free kernel: ada1: Previously was known as ad14
Jun 30 23:53:26 nas4free kernel: ada2 at ahcich5 bus 0 scbus6 target 0 lun 0
Jun 30 23:53:26 nas4free kernel: ada2: <WDC WD15EADS-00W4B0 01.00A01> ATA-8 SATA 2.x device
Jun 30 23:53:26 nas4free kernel: ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
Jun 30 23:53:26 nas4free kernel: ada2: Command Queueing enabled
Jun 30 23:53:26 nas4free kernel: ada2: 1430799MB (2930277168 512 byte sectors: 16H 63S/T 16383C)
Jun 30 23:53:26 nas4free kernel: ada2: Previously was known as ad16
Jun 30 23:53:26 nas4free kernel: SMP: AP CPU #1 Launched!
Jun 30 23:53:26 nas4free kernel: Timecounter "TSC-low" frequency 11302668 Hz quality 1000
Jun 30 23:53:26 nas4free kernel: Trying to mount root from ufs:/dev/md0 []...
Jun 30 23:53:27 nas4free lighttpd[1923]: (log.c.166) server started
Jun 30 23:53:27 nas4free login: login on ttyv0 as root
Jul  1 02:02:11 nas4free kernel: ahcich3: Timeout on slot 26 port 0
Jul  1 02:02:11 nas4free kernel: ahcich3: is 00000000 cs 04000000 ss 00000000 rs 04000000 tfd c0 serr 00000000 cmd 0000da17
Jul  1 02:02:11 nas4free kernel: (ada0:ahcich3:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
Jul  1 02:02:11 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: Command timeout
Jul  1 02:02:11 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 02:02:42 nas4free kernel: ahcich3: AHCI reset: device not ready after 31000ms (tfd = 00000080)
Jul  1 02:03:12 nas4free kernel: ahcich3: Timeout on slot 26 port 0
Jul  1 02:03:12 nas4free kernel: ahcich3: is 00000000 cs 00000000 ss 00000000 rs 04000000 tfd 58 serr 00000000 cmd 0000da17
Jul  1 02:03:12 nas4free kernel: (aprobe0:ahcich3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jul  1 02:03:12 nas4free kernel: (aprobe0:ahcich3:0:0:0): CAM status: Command timeout
Jul  1 02:03:12 nas4free kernel: (aprobe0:ahcich3:0:0:0): Error 5, Retry was blocked
Jul  1 02:18:53 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 77 b9 1a 40 ab 00 00 01 00 00
Jul  1 02:18:53 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 02:18:53 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 02:18:53 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 1c ba 1a 00 ab 00 00 00 01
Jul  1 02:18:53 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 02:18:58 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 77 b9 1a 40 ab 00 00 01 00 00
Jul  1 02:18:58 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 02:18:58 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 02:18:58 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 1c ba 1a 00 ab 00 00 00 01
Jul  1 02:18:58 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 02:19:02 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 77 b9 1a 40 ab 00 00 01 00 00
Jul  1 02:19:02 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 02:19:02 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 02:19:02 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 1c ba 1a 00 ab 00 00 00 01
Jul  1 02:19:02 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 02:19:06 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 77 b9 1a 40 ab 00 00 01 00 00
Jul  1 02:19:06 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 02:19:06 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 02:19:06 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 1c ba 1a 00 ab 00 00 00 01
Jul  1 02:19:06 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 02:19:10 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 77 b9 1a 40 ab 00 00 01 00 00
Jul  1 02:19:10 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 02:19:10 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 02:19:10 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 1c ba 1a 00 ab 00 00 00 01
Jul  1 02:19:10 nas4free kernel: (ada0:ahcich3:0:0:0): Error 5, Retries exhausted
Jul  1 02:19:15 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 82 cf 1a 40 ab 00 00 01 00 00
Jul  1 02:19:15 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 02:19:15 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 02:19:15 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 de cf 1a 00 ab 00 00 00 01
Jul  1 02:19:15 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 02:19:18 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 82 cf 1a 40 ab 00 00 01 00 00
Jul  1 02:19:18 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 02:19:18 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 02:19:18 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 de cf 1a 00 ab 00 00 00 01
Jul  1 02:19:18 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 02:19:20 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 82 cf 1a 40 ab 00 00 01 00 00
Jul  1 02:19:20 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 02:19:20 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 02:19:20 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 de cf 1a 00 ab 00 00 00 01
Jul  1 02:19:20 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 02:19:24 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 82 cf 1a 40 ab 00 00 01 00 00
Jul  1 02:19:24 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 02:19:24 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 02:19:24 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 de cf 1a 00 ab 00 00 00 01
Jul  1 02:19:24 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 02:19:27 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 82 cf 1a 40 ab 00 00 01 00 00
Jul  1 02:19:27 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 02:19:27 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 02:19:27 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 de cf 1a 00 ab 00 00 00 01
Jul  1 02:19:27 nas4free kernel: (ada0:ahcich3:0:0:0): Error 5, Retries exhausted
Jul  1 07:13:37 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 44 64 ec 40 ad 00 00 01 00 00
Jul  1 07:13:37 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:13:37 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:13:37 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 f6 64 ec 00 ad 00 00 00 01
Jul  1 07:13:37 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:13:40 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 44 64 ec 40 ad 00 00 01 00 00
Jul  1 07:13:40 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:13:40 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:13:40 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 f6 64 ec 00 ad 00 00 00 01
Jul  1 07:13:40 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:13:43 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 44 64 ec 40 ad 00 00 01 00 00
Jul  1 07:13:43 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:13:43 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:13:43 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 f6 64 ec 00 ad 00 00 00 01
Jul  1 07:13:43 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:13:46 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 44 64 ec 40 ad 00 00 01 00 00
Jul  1 07:13:46 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:13:46 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:13:46 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 f6 64 ec 00 ad 00 00 00 01
Jul  1 07:13:46 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:13:49 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 44 64 ec 40 ad 00 00 01 00 00
Jul  1 07:13:49 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:13:49 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:13:49 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 f6 64 ec 00 ad 00 00 00 01
Jul  1 07:13:49 nas4free kernel: (ada0:ahcich3:0:0:0): Error 5, Retries exhausted
Jul  1 07:13:54 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 52 80 ec 40 ad 00 00 01 00 00
Jul  1 07:13:54 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:13:54 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:13:54 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 65 80 ec 00 ad 00 00 00 01
Jul  1 07:13:54 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:13:57 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 52 80 ec 40 ad 00 00 01 00 00
Jul  1 07:13:57 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:13:57 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:13:57 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 65 80 ec 00 ad 00 00 00 01
Jul  1 07:13:57 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:13:59 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 52 80 ec 40 ad 00 00 01 00 00
Jul  1 07:13:59 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:13:59 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:13:59 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 65 80 ec 00 ad 00 00 00 01
Jul  1 07:13:59 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:14:02 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 52 80 ec 40 ad 00 00 01 00 00
Jul  1 07:14:02 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:14:02 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:14:02 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 65 80 ec 00 ad 00 00 00 01
Jul  1 07:14:02 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:14:05 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 52 80 ec 40 ad 00 00 01 00 00
Jul  1 07:14:05 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:14:05 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:14:05 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 65 80 ec 00 ad 00 00 00 01
Jul  1 07:14:05 nas4free kernel: (ada0:ahcich3:0:0:0): Error 5, Retries exhausted
Jul  1 07:14:10 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 43 d1 ec 40 ad 00 00 01 00 00
Jul  1 07:14:10 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:14:10 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:14:10 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 2e d2 ec 00 ad 00 00 00 01
Jul  1 07:14:10 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:14:13 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 43 d1 ec 40 ad 00 00 01 00 00
Jul  1 07:14:13 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:14:13 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:14:13 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 2e d2 ec 00 ad 00 00 00 01
Jul  1 07:14:13 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:14:16 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 43 d1 ec 40 ad 00 00 01 00 00
Jul  1 07:14:16 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:14:16 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:14:16 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 2e d2 ec 00 ad 00 00 00 01
Jul  1 07:14:16 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:14:19 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 43 d1 ec 40 ad 00 00 01 00 00
Jul  1 07:14:19 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:14:19 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:14:19 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 2e d2 ec 00 ad 00 00 00 01
Jul  1 07:14:19 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:14:21 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 43 d1 ec 40 ad 00 00 01 00 00
Jul  1 07:14:21 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:14:21 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:14:21 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 2e d2 ec 00 ad 00 00 00 01
Jul  1 07:14:21 nas4free kernel: (ada0:ahcich3:0:0:0): Error 5, Retries exhausted
Jul  1 07:14:24 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 44 d2 ec 40 ad 00 00 01 00 00
Jul  1 07:14:24 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:14:24 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:14:24 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 22 d3 ec 00 ad 00 00 00 01
Jul  1 07:14:24 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:14:27 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 44 d2 ec 40 ad 00 00 01 00 00
Jul  1 07:14:27 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:14:27 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:14:27 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 22 d3 ec 00 ad 00 00 00 01
Jul  1 07:14:27 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:14:30 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 44 d2 ec 40 ad 00 00 01 00 00
Jul  1 07:14:30 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:14:30 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:14:30 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 22 d3 ec 00 ad 00 00 00 01
Jul  1 07:14:30 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:14:35 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 53 f0 ec 40 ad 00 00 01 00 00
Jul  1 07:14:35 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:14:35 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:14:35 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 a8 f0 ec 00 ad 00 00 00 01
Jul  1 07:14:35 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:14:38 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 53 f0 ec 40 ad 00 00 01 00 00
Jul  1 07:14:38 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:14:38 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:14:38 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 a8 f0 ec 00 ad 00 00 00 01
Jul  1 07:14:38 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:14:42 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 53 f0 ec 40 ad 00 00 01 00 00
Jul  1 07:14:42 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:14:42 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:14:42 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 41 f1 ec 00 ad 00 00 00 01
Jul  1 07:14:42 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:14:45 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 53 f0 ec 40 ad 00 00 01 00 00
Jul  1 07:14:45 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:14:45 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:14:45 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 41 f1 ec 00 ad 00 00 00 01
Jul  1 07:14:45 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:14:48 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 53 f0 ec 40 ad 00 00 01 00 00
Jul  1 07:14:48 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:14:48 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:14:48 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 41 f1 ec 00 ad 00 00 00 01
Jul  1 07:14:48 nas4free kernel: (ada0:ahcich3:0:0:0): Error 5, Retries exhausted
Jul  1 07:14:52 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 53 f1 ec 40 ad 00 00 01 00 00
Jul  1 07:14:52 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:14:52 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:14:52 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 9a f1 ec 00 ad 00 00 00 01
Jul  1 07:14:52 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:14:54 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 53 f1 ec 40 ad 00 00 01 00 00
Jul  1 07:14:54 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:14:54 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:14:54 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 9a f1 ec 00 ad 00 00 00 01
Jul  1 07:14:54 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:14:57 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 53 f1 ec 40 ad 00 00 01 00 00
Jul  1 07:14:57 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:14:57 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:14:57 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 9a f1 ec 00 ad 00 00 00 01
Jul  1 07:14:57 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:15:00 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 53 f1 ec 40 ad 00 00 01 00 00
Jul  1 07:15:00 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:15:00 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:15:00 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 9a f1 ec 00 ad 00 00 00 01
Jul  1 07:15:00 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:15:03 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 53 f1 ec 40 ad 00 00 01 00 00
Jul  1 07:15:03 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:15:03 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:15:03 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 9a f1 ec 00 ad 00 00 00 01
Jul  1 07:15:03 nas4free kernel: (ada0:ahcich3:0:0:0): Error 5, Retries exhausted
Jul  1 07:15:09 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 29 1e ed 40 ad 00 00 01 00 00
Jul  1 07:15:09 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:15:09 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:15:09 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 d6 1e ed 00 ad 00 00 00 01
Jul  1 07:15:09 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:15:12 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 29 1e ed 40 ad 00 00 01 00 00
Jul  1 07:15:12 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:15:12 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:15:12 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 d6 1e ed 00 ad 00 00 00 01
Jul  1 07:15:12 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:15:15 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 29 1e ed 40 ad 00 00 01 00 00
Jul  1 07:15:15 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:15:15 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:15:15 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 d6 1e ed 00 ad 00 00 00 01
Jul  1 07:15:15 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:15:18 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 29 1e ed 40 ad 00 00 01 00 00
Jul  1 07:15:18 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:15:18 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:15:18 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 d6 1e ed 00 ad 00 00 00 01
Jul  1 07:15:18 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:15:21 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 29 1e ed 40 ad 00 00 01 00 00
Jul  1 07:15:21 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:15:21 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:15:21 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 d6 1e ed 00 ad 00 00 00 01
Jul  1 07:15:21 nas4free kernel: (ada0:ahcich3:0:0:0): Error 5, Retries exhausted
Jul  1 07:15:36 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 bf 75 f6 40 ad 00 00 01 00 00
Jul  1 07:15:36 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:15:36 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:15:36 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 f7 75 f6 00 ad 00 00 00 01
Jul  1 07:15:36 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:15:40 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 c2 7b f6 40 ad 00 00 01 00 00
Jul  1 07:15:40 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:15:40 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:15:40 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 df 7b f6 00 ad 00 00 00 01
Jul  1 07:15:40 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:15:43 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 c2 7b f6 40 ad 00 00 01 00 00
Jul  1 07:15:43 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:15:43 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:15:43 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 df 7b f6 00 ad 00 00 00 01
Jul  1 07:15:43 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:15:46 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 c2 7b f6 40 ad 00 00 01 00 00
Jul  1 07:15:46 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:15:46 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:15:46 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 df 7b f6 00 ad 00 00 00 01
Jul  1 07:15:46 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:15:51 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 c6 83 f6 40 ad 00 00 01 00 00
Jul  1 07:15:51 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:15:51 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:15:51 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 d6 83 f6 00 ad 00 00 00 01
Jul  1 07:15:51 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:15:54 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 c6 83 f6 40 ad 00 00 01 00 00
Jul  1 07:15:54 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:15:54 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:15:54 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 d6 83 f6 00 ad 00 00 00 01
Jul  1 07:15:54 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:15:57 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 c6 83 f6 40 ad 00 00 01 00 00
Jul  1 07:15:57 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:15:57 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:15:57 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 d6 83 f6 00 ad 00 00 00 01
Jul  1 07:15:57 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:16:00 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 c6 83 f6 40 ad 00 00 01 00 00
Jul  1 07:16:00 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:16:00 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:16:00 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 d6 83 f6 00 ad 00 00 00 01
Jul  1 07:16:00 nas4free kernel: (ada0:ahcich3:0:0:0): Retrying command
Jul  1 07:16:03 nas4free kernel: (ada0:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 c6 83 f6 40 ad 00 00 01 00 00
Jul  1 07:16:03 nas4free kernel: (ada0:ahcich3:0:0:0): CAM status: ATA Status Error
Jul  1 07:16:03 nas4free kernel: (ada0:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul  1 07:16:03 nas4free kernel: (ada0:ahcich3:0:0:0): RES: 41 40 d6 83 f6 00 ad 00 00 00 01
Jul  1 07:16:03 nas4free kernel: (ada0:ahcich3:0:0:0): Error 5, Retries exhausted
Jul  1 07:36:08 nas4free login: login on ttyv0 as root
Jul  1 07:37:13 nas4free kernel: em0: link state changed to UP
Jul  1 07:38:14 nas4free root: rcconf service executed
Jul  1 07:38:14 nas4free kernel: em0: link state changed to DOWN
Jul  1 07:38:16 nas4free kernel: em0: link state changed to UP
Jul  1 07:38:33 nas4free dhclient: New IP Address (em0): 192.168.1.104
Jul  1 07:38:33 nas4free dhclient: New Subnet Mask (em0): 255.255.255.0
Jul  1 07:38:33 nas4free dhclient: New Broadcast Address (em0): 192.168.1.255
Jul  1 07:38:33 nas4free dhclient: New Routers (em0): 192.168.1.1
Jul  1 07:38:33 nas4free root: netif service restarted
Jul  1 07:38:33 nas4free root: routing service restarted
Jul  1 07:38:33 nas4free dhclient[3252]: connection closed
Jul  1 07:38:33 nas4free dhclient[3252]: exiting.
Jul  1 07:38:33 nas4free kernel: em0: link state changed to DOWN
Jul  1 07:38:35 nas4free kernel: em0: link state changed to UP
Jul  1 07:38:36 nas4free dhclient: New IP Address (em0): 192.168.1.104
Jul  1 07:38:36 nas4free dhclient: New Subnet Mask (em0): 255.255.255.0
Jul  1 07:38:36 nas4free dhclient: New Broadcast Address (em0): 192.168.1.255
Jul  1 07:38:36 nas4free dhclient: New Routers (em0): 192.168.1.1
Jul  1 07:38:36 nas4free root: resolv service restarted
Jul  1 07:38:36 nas4free lighttpd[1923]: (server.c.1546) server stopped by UID = 0 PID = 3630
Jul  1 07:38:36 nas4free lighttpd[3703]: (log.c.166) server started
Jul  1 07:38:36 nas4free root: lighttpd service restarted
Jul  1 07:40:35 nas4free root: Authentication error for illegal user admin from 192.168.1.111
Jul  1 07:40:42 nas4free root: Authentication error for illegal user admin from 192.168.1.111

Thanks for any help
Ian

Post by **b0ssman** » 02 Jul 2013 20:09

this is bad.

pls post the smart output for ada0.

but it looks like your ada0 is dying as well.
i suspect you need to clone ada0 onto a new drive as well.
with the unrecoverable error it could be that you might loose some data.

so that mean 3 out of 4 drives have gone bad. what have you done to your discs?

idmah · Post by **idmah** » 03 Jul 2013 05:52

I think this is what you want..

Code: Select all

$ smartctl -a /dev/ada0
smartctl 6.1 2013-03-16 r3800 [FreeBSD 9.1-RELEASE-p3 amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.11
Device Model:     ST31500341AS
Serial Number:    9VS1ZKR6
LU WWN Device Id: 5 000c50 014e2ef6d
Firmware Version: CC1H
User Capacity:    1,500,301,910,016 bytes [1.50 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 2.6, 3.0 Gb/s
Local Time is:    Wed Jul  3 11:43:22 2013 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status:  (0x82)	Offline data collection activity
					was completed without error.
					Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(  609) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   1) minutes.
Extended self-test routine
recommended polling time: 	 ( 290) minutes.
Conveyance self-test routine
recommended polling time: 	 (   2) minutes.
SCT capabilities: 	       (0x103f)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   106   099   006    Pre-fail  Always       -       12187261
  3 Spin_Up_Time            0x0003   100   099   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       295
  5 Reallocated_Sector_Ct   0x0033   082   082   036    Pre-fail  Always       -       742
  7 Seek_Error_Rate         0x000f   065   060   030    Pre-fail  Always       -       111763248048
  9 Power_On_Hours          0x0032   074   074   000    Old_age   Always       -       23071
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       294
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   001   001   000    Old_age   Always       -       107
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       17180131332
189 High_Fly_Writes         0x003a   028   028   000    Old_age   Always       -       72
190 Airflow_Temperature_Cel 0x0022   065   041   045    Old_age   Always   In_the_past 35 (34 113 35 28 0)
194 Temperature_Celsius     0x0022   035   059   000    Old_age   Always       -       35 (0 16 0 0 0)
195 Hardware_ECC_Recovered  0x001a   049   015   000    Old_age   Always       -       12187261
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       3
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       3
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       222419176413717
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       4290029007
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       295692493

SMART Error Log Version: 1
ATA Error Count: 109 (device log contains only the most recent five errors)
	CR = Command Register [HEX]
	FR = Features Register [HEX]
	SC = Sector Count Register [HEX]
	SN = Sector Number Register [HEX]
	CL = Cylinder Low Register [HEX]
	CH = Cylinder High Register [HEX]
	DH = Device/Head Register [HEX]
	DC = Device Command Register [HEX]
	ER = Error register [HEX]
	ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 109 occurred at disk power-on lifetime: 23067 hours (961 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 80 ff ff ff 4f 00   2d+07:33:02.401  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00   2d+07:33:02.309  READ LOG EXT
  60 00 80 ff ff ff 4f 00   2d+07:32:59.602  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00   2d+07:32:59.502  READ LOG EXT
  60 00 80 ff ff ff 4f 00   2d+07:32:56.781  READ FPDMA QUEUED

Error 108 occurred at disk power-on lifetime: 23067 hours (961 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 80 ff ff ff 4f 00   2d+07:32:59.602  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00   2d+07:32:59.502  READ LOG EXT
  60 00 80 ff ff ff 4f 00   2d+07:32:56.781  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00   2d+07:32:56.754  READ LOG EXT
  60 00 80 ff ff ff 4f 00   2d+07:32:53.976  READ FPDMA QUEUED

Error 107 occurred at disk power-on lifetime: 23067 hours (961 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 80 ff ff ff 4f 00   2d+07:32:56.781  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00   2d+07:32:56.754  READ LOG EXT
  60 00 80 ff ff ff 4f 00   2d+07:32:53.976  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00   2d+07:32:53.940  READ LOG EXT
  60 00 80 ff ff ff 4f 00   2d+07:32:51.196  READ FPDMA QUEUED

Error 106 occurred at disk power-on lifetime: 23067 hours (961 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 80 ff ff ff 4f 00   2d+07:32:53.976  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00   2d+07:32:53.940  READ LOG EXT
  60 00 80 ff ff ff 4f 00   2d+07:32:51.196  READ FPDMA QUEUED
  61 00 10 10 02 00 40 00   2d+07:32:51.183  WRITE FPDMA QUEUED
  61 00 10 ff ff ff 4f 00   2d+07:32:51.152  WRITE FPDMA QUEUED

Error 105 occurred at disk power-on lifetime: 23067 hours (961 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 80 ff ff ff 4f 00   2d+07:32:51.196  READ FPDMA QUEUED
  61 00 10 10 02 00 40 00   2d+07:32:51.183  WRITE FPDMA QUEUED
  61 00 10 ff ff ff 4f 00   2d+07:32:51.152  WRITE FPDMA QUEUED
  61 00 10 ff ff ff 4f 00   2d+07:32:51.145  WRITE FPDMA QUEUED
  60 00 02 ff ff ff 4f 00   2d+07:32:51.123  READ FPDMA QUEUED

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     16834         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Could I scrub ada0 ?

What started it.
Basically I had an error in one of the drives, probably the laptop shutting down during copy.
So then a drive, let's call it ad8 was giving me problems, so i replaced it.
but the server running the freenas 0.7.1 was crashing so I think that exacerbated all the problems
and I ended up where I am now: 2-3 drives kaput!

I don't mind loosing a few files, just not all of them ..

Isn't there just a way to skip the offending filer? Well you guys are the experts.
I guess I'll buy another 1.5 and clone ada0 on to it.. if that's what you think.
getting depressed.

thanks
Ian

Post by **b0ssman** » 03 Jul 2013 06:50

yes your ada0 is dying.
basically once you see a non zero value for Reallocated_Sector_Ct its time to replace the drive.
your drive is also reporting 107 unrecoverable errors.

try rescuing it the same way.

btw i never asked how the 4 drives where set up. i assume 4 drive raidz1?
raidz1 can normally handle 1 failed drive.

what about the 4th drive. just for safety post the smart values as well.

Post by **b0ssman** » 03 Jul 2013 07:50

oh i didnt see the information that it failed the "Airflow_Temperature_Cel" in the past.

are you cooling your drives properly? in a nas environment you want a constant stream of air across your drives.

idmah · Post by **idmah** » 03 Jul 2013 17:03

Could have happened on old machine. The new one has 3 fans, One top, side and back. (do I need more?)
maybe that is what caused the original error?

So I'm going to try your DD noerror suggestion for ada0 to a new drive.
fingers crossed. gawd I now have 8 X 1.5 TB drives.. hahaha.. sigh..

Thanks so much for the help really appreciated.

idmah · Post by **idmah** » 08 Jul 2013 18:40

Well that didn't help..

Pretty close to giving up on everything.
cloning ada0 to new drive seemed to work but had 10M checksum errors. no ReadWrite.
So when resilvering it does the same thing and just keeps looping/retrying once it hits. 75% - 87% but huge number of checksum errors. see above.

Is it worth trying to pull what I can from the NAS drives and "format it all With extreme prejudice?"
What's the chances I can pull anything off the drives?
Well going to try and then NUKE it all.

*UNLESS ANY/ someone has a last ditch/ desperate ideas? .. I'll try anything!!!

thanks for the help.
Ian

Post by **b0ssman** » 08 Jul 2013 19:38

the files with the errors are gone.

i dont know why it doesnt finish the resilvering.

so yes just move the data from the array and recreate the array.

idmah · Post by **idmah** » 17 Aug 2013 18:22

Ok So let's call this solved !!

Here's what I did which 'sort of worked'

1. built a new non-crashy machine.
2. put the drives in the machine.
3. (this didn't help) - copied the drives bad with to new 'fresh' drives - and but nothing, came back.. so...
4. Imported config file. *after renaming freenas to nas4free in text file.
5. put the 'bad drives' back in the machine.
6. imported ZFS with: zpool import.
7. Allowed Scrub of drives to finish, with a gazillion errors.
8. Grabbed as much of the files that survived.
9. Now I'm going to wipe the Whole thing and Create a new ZFS Pool. and put the recovered files back in..

thanks everyone for your help !!!

ps. One weird thing was that the system rebooted it's self. When I was trying to copy the files.

XigmaNAS

Is it all gone? Help me - Obi Wan...

Is it all gone? Help me - Obi Wan...

Re: Is it all gone? Help me - Obi Wan...

Re: Is it all gone? Help me - Obi Wan...

Re: Is it all gone? Help me - Obi Wan...

Re: Is it all gone? Help me - Obi Wan...

Re: Is it all gone? Help me - Obi Wan...

Re: Is it all gone? Help me - Obi Wan...

Re: Is it all gone? Help me - Obi Wan...

Re: Is it all gone? Help me - Obi Wan...

Re: Is it all gone? Help me - Obi Wan...

Re: Is it all gone? Help me - Obi Wan...

Re: Is it all gone? Help me - Obi Wan...

Re: Is it all gone? Help me - Obi Wan...

Re: Is it all gone? Help me - Obi Wan...

Re: Is it all gone? Help me - Obi Wan...

Re: Is it all gone? Help me - Obi Wan...

Re: Is it all gone? Help me - Obi Wan...

Re: Is it all gone? Help me - Obi Wan...

Re: Is it all gone? Help me - Obi Wan...

Re: Is it all gone? Help me - Obi Wan...

Re: Is it all gone? Help me - Obi Wan...

Re: Is it all gone? Help me - Obi Wan...

Re: Is it all gone? Help me - Obi Wan...

Re: Is it all gone? Help me - Obi Wan...

Re: Is it all gone? Help me - Obi Wan...

Re: Is it all gone? Help me - Obi Wan...

Re: Is it all gone? Help me - Obi Wan...

Re: Is it all gone? Help me - Obi Wan...