Page 1 of 1

ZFS pool status UNAVAIL

Posted: 08 Dec 2015 00:23
by simoncs
Hi

v9.3
Three 2tb drives in zfs pool

I had to relocate the NAS. It had become unresponsive in the webgui, so i did a hard turn-oss rather than shutdown. (i don't have a video card and monitor permently attached, so no access to cmd line) I presume hard turn off is unadisable? - I seem to go through disks more often than i would like - typically at least one a year - is that normal?

When booting back up i was faced with zpool status UNAVAIL. It seems two of the disks were unavailabale, but I have not seen this error before so am not sure if this means the disks are bad or not? I have only seen it say Degraded before.
Is there some cmd i can run to check the quality of the disks, or can the pool just be bought back online somehow?
Can i try to just reimport the last config file?

I did a search and ran these cmds, but they didn't seem to indicate whether the disks were bad or not....
pool: NASBOX
state: UNAVAIL
status: One or more devices could not be opened. There are insufficient
replicas for the pool to continue functioning.
action: Attach the missing device and online it using 'zpool online'.
see: http://illumos.org/msg/ZFS-8000-3C
scan: none requested
config:

NAME STATE READ WRITE CKSUM
NASBOX UNAVAIL 0 0 0
raidz1-0 UNAVAIL 0 0 0
ada0 ONLINE 0 0 0 block size: 512B configured, 4096B native
5607581269443893222 UNAVAIL 0 0 0 was /dev/ada2
2872910672160556070 UNAVAIL 0 0 0 was /dev/ada3


$ camcontrol devlist
<ST2000DM001-1ER164 CC25> at scbus3 target 0 lun 0 (ada0,pass0)
<WDC WD10EZEX-08M2NA0 01.01A01> at scbus3 target 1 lun 0 (ada1,pass1)
<SanDisk Cruzer Switch 1.27> at scbus7 target 0 lun 0 (da0,pass2)


$ gpart show
=> 63 31266753 da0 MBR (14G)
63 1654632 1 freebsd [active] (807M)
1654695 16787925 2 freebsd (8.0G)
18442620 12819870 3 freebsd (6.1G)
31262490 4326 - free - (2.1M)

=> 0 1654632 da0s1 BSD (807M)
0 8129 - free - (4M)
8129 1638400 1 freebsd-ufs (800M)
1646529 8103 - free - (4M)

=> 0 16787925 da0s2 BSD (8.0G)
0 89 - free - (44k)
89 16777216 2 freebsd-swap (8.0G)
16777305 10620 - free - (5.2M)

=> 0 12819870 da0s3 BSD (6.1G)
0 5764 - free - (2.8M)
5764 12812288 1 freebsd-ufs (6.1G)
12818052 1818 - free - (909k)



thanks

Simon

Re: ZFS pool status UNAVAIL

Posted: 08 Dec 2015 12:18
by crowi
can you post the dmesg log? maybe your controller has gone...

Re: ZFS pool status UNAVAIL

Posted: 08 Dec 2015 22:52
by simoncs
sure - i just ran the cmd in the webgui - is this the whole output, or has it truncated?

$ dmesg
Copyright (c) 1992-2014 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.3-RELEASE-p22 #0 r286663M: Wed Aug 12 14:28:48 CEST 2015
root@dev.nas4free.org:/usr/obj/nas4free/usr/src/sys/NAS4FREE-amd64 amd64
gcc version 4.2.1 20070831 patched [FreeBSD]
CPU: Intel(R) Core(TM)2 Duo CPU E6750 @ 2.66GHz (2666.73-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x6fb Family = 0x6 Model = 0xf Stepping = 11
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0xe3fd<SSE3,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant, performance statistics
real memory = 8589934592 (8192 MB)
avail memory = 8236453888 (7854 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <GBT GBTUACPI>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ioapic0: Changing APIC ID to 2
ioapic0 <Version 2.0> irqs 0-23 on motherboard
module_register_init: MOD_LOAD (vesa, 0xffffffff80e94380, 0) error 19
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
acpi0: <GBT GBTUACPI> on motherboard
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, f3de0000 (3) failed
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Event timer "HPET3" frequency 14318180 Hz quality 440
atrtc0: <AT realtime clock> port 0x70-0x73 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
uhci0: <Intel 82801I (ICH9) USB controller> port 0xe100-0xe11f irq 16 at device 26.0 on pci0
uhci0: LegSup = 0x2f00
usbus0 on uhci0
uhci1: <Intel 82801I (ICH9) USB controller> port 0xe200-0xe21f irq 21 at device 26.1 on pci0
uhci1: LegSup = 0x2f00
usbus1 on uhci1
uhci2: <Intel 82801I (ICH9) USB controller> port 0xe000-0xe01f irq 18 at device 26.2 on pci0
uhci2: LegSup = 0x2f00
usbus2 on uhci2
ehci0: <Intel 82801I (ICH9) USB 2.0 controller> mem 0xfa104000-0xfa1043ff irq 18 at device 26.7 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci0
hdac0: <Intel 82801I HDA Controller> mem 0xfa100000-0xfa103fff irq 22 at device 27.0 on pci0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
pci2: <ACPI PCI bus> on pcib2
atapci0: <JMicron JMB363 UDMA133 controller> port 0xc000-0xc007,0xc100-0xc103,0xc200-0xc207,0xc300-0xc303,0xc400-0xc40f mem 0xfa000000-0xfa001fff irq 19 at device 0.0 on pci2
ahci0: <JMicron JMB363 AHCI SATA controller> at channel -1 on atapci0
ahci0: AHCI v1.00 with 2 3Gbps ports, Port Multiplier supported
ahci0: quirks=0x1<NOFORCE>
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ata2: <ATA channel> at channel 0 on atapci0
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0
pci3: <ACPI PCI bus> on pcib3
re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xd000-0xd0ff mem 0xf9000000-0xf9000fff irq 16 at device 0.0 on pci3
re0: Using 1 MSI message
re0: Chip rev. 0x38000000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus0
rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Ethernet address: 00:1a:4d:5a:5c:b1
uhci3: <Intel 82801I (ICH9) USB controller> port 0xe300-0xe31f irq 23 at device 29.0 on pci0
usbus4 on uhci3
uhci4: <Intel 82801I (ICH9) USB controller> port 0xe400-0xe41f irq 19 at device 29.1 on pci0
usbus5 on uhci4
uhci5: <Intel 82801I (ICH9) USB controller> port 0xe500-0xe51f irq 18 at device 29.2 on pci0
usbus6 on uhci5
ehci1: <Intel 82801I (ICH9) USB 2.0 controller> mem 0xfa105000-0xfa1053ff irq 23 at device 29.7 on pci0
usbus7: EHCI version 1.0
usbus7 on ehci1
pcib4: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci4: <ACPI PCI bus> on pcib4
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci1: <Intel ICH9 SATA300 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xf000-0xf00f,0xfc00-0xfc0f at device 31.2 on pci0
ata0: <ATA channel> at channel 0 on atapci1
ata1: <ATA channel> at channel 1 on atapci1
atapci2: <Intel ICH9 SATA300 controller> port 0xe700-0xe707,0xe800-0xe803,0xe900-0xe907,0xea00-0xea03,0xeb00-0xeb0f,0xec00-0xec0f irq 19 at device 31.5 on pci0
ata3: <ATA channel> at channel 0 on atapci2
ata4: <ATA channel> at channel 1 on atapci2
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (1355,n,8,1)
ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
acpi_perf0: <ACPI CPU Frequency Control> on cpu0
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 82a082a0600082a
device_attach: est1 attach returned 6
p4tcc1: <CPU Frequency Thermal Control> on cpu1
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert loadable, nat loadable, default to accept, logging disabled
iSCSI boot driver version 0.2.8
hdacc0: <Realtek ALC889A HDA CODEC> at cad 2 on hdac0
hdaa0: <Realtek ALC889A Audio Function Group> at nid 1 on hdacc0
pcm0: <Realtek ALC889A (Rear Analog 7.1/2.0)> at nid 20,22,21,23 and 24,26 on hdaa0
pcm1: <Realtek ALC889A (Front Analog)> at nid 27 and 25 on hdaa0
pcm2: <Realtek ALC889A (Rear Digital)> at nid 30 and 31 on hdaa0
xmd version 0.4.4 by NAS4Free Project (experimental)
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
uhub0: 2 ports with 2 removable, self powered
ugen3.1: <Intel> at usbus3
uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
uhub1: 2 ports with 2 removable, self powered
usbus4: 12Mbps Full Speed USB v1.0
uhub2: 2 ports with 2 removable, self powered
ugen4.1: <Intel> at usbus4
uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
usbus5: 12Mbps Full Speed USB v1.0
usbus6: 12Mbps Full Speed USB v1.0
ugen5.1: <Intel> at usbus5
uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
ugen6.1: <Intel> at usbus6
uhub6: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
uhub4: 2 ports with 2 removable, self powered
usbus7: 480Mbps High Speed USB v2.0
ugen7.1: <Intel> at usbus7
uhub7: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus7
uhub5: 2 ports with 2 removable, self powered
uhub6: 2 ports with 2 removable, self powered
uhub3: 6 ports with 6 removable, self powered
uhub7: 6 ports with 6 removable, self powered
xmd: default blocklen 32KB, compression LZ4/LZ4HC, level 6
xmd0: Loaded preload cloop image </mfsroot.uzip> 14216704 bytes at 0xffffffff81eacee0 (128MB disk w/LZ4)
ugen3.2: <SanDisk> at usbus3
umass0: <SanDisk Cruzer Switch, class 0/0, rev 2.00/1.27, addr 2> on usbus3
umass0: SCSI over Bulk-Only; quirks = 0x0100
umass0:7:0:-1: Attached to scbus7
ada0 at ata0 bus 0 scbus3 target 0 lun 0
ada0: <ST2000DM001-1ER164 CC25> ATA-9 SATA 3.x device
ada0: Serial Number Z4Z1QAPB
ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada0: 1907728MB (3907027055 512 byte sectors: 16H 63S/T 16383C)
ada0: quirks=0x1<4K>
ada0: Previously was known as ad0
ada1 at ata0 bus 0 scbus3 target 1 lun 0
ada1: <WDC WD10EZEX-08M2NA0 01.01A01> ATA-9 SATA 3.x device
ada1: Serial Number WD-WMC3F0E7E64H
ada1: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada1: 953868MB (1953523055 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad1
cam_periph_alloc: attempt to re-allocate valid device pass2 rejected flags 0 refcount 2
passasync: Unable to attach new device due to status 0x6: CCB request was invalid
da0 at umass-sim0 bus 0 scbus7 target 0 lun 0
SMP: AP CPU #1 Launched!
da0: <SanDisk Cruzer Switch 1.27> Removable Direct Access SCSI-6 device
da0: Serial Number 4C530006840519101250
da0: 40.000MB/s transfers
da0: 15267MB (31266816 512 byte sectors: 255H 63S/T 1946C)
da0: quirks=0x2<NO_6_BYTE>
Timecounter "TSC-low" frequency 1333362920 Hz quality 1000
Trying to mount root from ufs:/dev/xmd0 []...

Re: ZFS pool status UNAVAIL

Posted: 09 Dec 2015 09:53
by crowi
so, your system just detects two SATA and one USB disk
ada1: <WDC WD10EZEX-08M2NA0 01.01A01> ATA-9 SATA 3.x device
ada0: <ST2000DM001-1ER164 CC25> ATA-9 SATA 3.x device
da0: <SanDisk Cruzer Switch 1.27> Removable Direct Access SCSI-6 device
but your controllers seem to work.
can you reconnect your drives on other SATA ports? you have 6 ports, so this should be no problem.

Re: ZFS pool status UNAVAIL

Posted: 09 Dec 2015 12:12
by simoncs
thanks crowi
i tried that, and in the process of doing so noticed the cables weren't a tight fit to the drives. so replaced two of the cables.
now at least one of the drives has come back, and the pool now says degraded as opposed to UNAVAIL, so i presume i just have to accept that i have lost one disk, replace it and resilver?

pool: NASBOX
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://illumos.org/msg/ZFS-8000-2Q
scan: resilvered 1.02T in 7h2m with 0 errors on Tue Dec 30 10:24:26 2014
config:

NAME STATE READ WRITE CKSUM
NASBOX DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
ada0 ONLINE 0 0 0 block size: 512B configured, 4096B native
ada2 ONLINE 0 0 0 block size: 512B configured, 4096B native
2872910672160556070 UNAVAIL 0 0 0 was /dev/ada3

errors: No known data errors

pool: NAS_BACKUP
state: ONLINE
status: One or more devices are configured to use a non-native block size.
Expect reduced performance.
action: Replace affected devices with devices that support the
configured block size, or migrate data to a properly configured
pool.
scan: none requested
config:

NAME STATE READ WRITE CKSUM
NAS_BACKUP ONLINE 0 0 0
ada1 ONLINE 0 0 0 block size: 512B configured, 4096B native

errors: No known data errors


was planning on just following this guide to replace the bad disk?
http://prefetch.net/blog/index.php/2007 ... -zfs-pool/