- 3 x 3tb _raidz1-0_
- Created with Nas4Free
- Migrated to Linux/Debian and zol a few month ago.
- Now in FAULTED state (The pool metadata is corrupted) http://zfsonlinux.org/msg/ZFS-8000-72
I'd love to be able to recover some data. I think I've reached my knowledge limit. =) Please help!
# History
1. ~4 weeks ago: `zpool scrub` reports checksum errors on sdb.
2. ~2 weeks ago: `zpool scrub` says sdb has too many errors and should be replaced. I order a new drive.
3. ~2 days ago: I install the new drive. (note that sdb got renamed to sdd after I added the new drive). I attempt to replace the bad drive (sdd) with the new drive (sda)
Code: Select all
402 zpool replace bigbubba0 sdd sda
403 zpool replace bigbubba0 sdd sda -f
404 zpool replace bigbubba0 sdd sda -f -o ashift=12
405 zpool replace bigbubba0 sdd sda -f -o ashift=4
5. After 8h only 500mb has been completed. I give it another 8h and it hasn’t made any progress. I figure the bad drive has totally bit it and reboot the system.
6. Reboot doesn’t help. I attempt to abort:
Code: Select all
422 zpool detach bigbubba0 sda
430 zpool clear -F bigbubba0
434 zpool detach bigbubba0 sda
7. After this step I am left with error http://zfsonlinux.org/msg/ZFS-8000-72 — I tried several things to recover:
Code: Select all
440 zpool import -V
441 zpool import -fX
442 zpool import -fX bigbubba0
456 zpool import -fD bigbubba0
457 zpool import -f bigbubba0
463 zpool import -fFX -o ro -o failmode=continue
464 zpool import -f -N -d /dev/disk/by-id/ata-Hitachi_HDS722020ALA330_JK1130YAGKS1VT bigbubba0
465 zpool import -f -N -d /dev/disk/by-id/ata-Hitachi_HDS722020ALA330_JK1130YAGKSW6T bigbubba0
466 zpool import -f -N -d /dev/sda bigbubba0
467 zpool import -f -N -d /dev/sdb bigbubba0
468 zpool import -f -N -d /dev/disk/by-id/bigbubba0
469 zpool import -f -N -d /dev/disk/by-id/ bigbubba0
477 zpool import -d /dev/disk/by-id
478 zpool import -o readonly=on bigbubba0
479 zpool import -d /dev/sdd
481 zpool import -d /dev/sda
482 zpool import -d /dev/disk/by-uuid/8140ce08-86d4-49ca-81df-a39eedcb1cab
483 zpool import -d /dev/disk/by-uuid/d42b4883-8800-4ce4-a9be-868709fbf952
None of this helped…. And I have reached the point where I realize I don’t know enough to fix the problem. Please help!
# Output
**# zpool import**
Code: Select all
pool: bigbubba0
id: 16328167560584328470
state: FAULTED
status: The pool metadata is corrupted.
action: The pool cannot be imported due to damaged devices or data.
The pool may be active on another system, but can be imported using
the '-f' flag.
see: http://zfsonlinux.org/msg/ZFS-8000-72
config:
bigbubba0 FAULTED corrupted data
raidz1-0 ONLINE
sdc ONLINE
sdb ONLINE
replacing-2 ONLINE
sdd ONLINE
sda ONLINE
**# uname -a**
Code: Select all
Linux bb2 3.16.0-4-amd64 #1 SMP Debian 3.16.39-1+deb8u2 (2017-03-07) x86_64 GNU/Linux
**# dkms status**
Code: Select all
spl, 0.6.5.9, 3.16.0-4-amd64, x86_64: installed
zfs, 0.6.5.9, 3.16.0-4-amd64, x86_64: installed
**$ zdb**
Code: Select all
bigbubba0:
version: 5000
name: 'bigbubba0'
state: 0
txg: 41072035
pool_guid: 16328167560584328470
errata: 0
hostid: 8323329
hostname: 'bb2'
vdev_children: 1
vdev_tree:
type: 'root'
id: 0
guid: 16328167560584328470
children[0]:
type: 'raidz'
id: 0
guid: 4280100286181462697
nparity: 1
metaslab_array: 14
metaslab_shift: 32
ashift: 9
asize: 6001182375936
is_log: 0
children[0]:
type: 'disk'
id: 0
guid: 714237933490165329
path: '/dev/sdc'
phys_path: '/dev/ada0'
whole_disk: 0
children[1]:
type: 'disk'
id: 1
guid: 13042370445854942928
path: '/dev/sdb'
phys_path: '/dev/ada1'
whole_disk: 0
children[2]:
type: 'replacing'
id: 2
guid: 9626083375216793335
whole_disk: 0
children[0]:
type: 'disk'
id: 0
guid: 14340659031728992743
path: '/dev/sdd'
phys_path: '/dev/ada2'
whole_disk: 0
children[1]:
type: 'disk'
id: 1
guid: 13403179909814545905
path: '/dev/sda1'
whole_disk: 1
DTL: 65990
resilver_txg: 41072032
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
**# zdb -s bigbubba0**
Code: Select all
zdb: can't open 'bigbubba0': Input/output error
**# ls -l /dev/disk/by-id**
Code: Select all
total 0
lrwxrwxrwx 1 root root 9 Mar 10 19:31 ata-Hitachi_HDS722020ALA330_JK1130YAGKG37T -> ../../sdd
lrwxrwxrwx 1 root root 9 Mar 10 19:31 ata-Hitachi_HDS722020ALA330_JK1130YAGKS1VT -> ../../sdb
lrwxrwxrwx 1 root root 9 Mar 10 19:31 ata-Hitachi_HDS722020ALA330_JK1130YAGKSW6T -> ../../sdc
lrwxrwxrwx 1 root root 9 Mar 10 19:31 ata-TOSHIBA_HDWD130_Z6DP2RVAS -> ../../sda
lrwxrwxrwx 1 root root 10 Mar 10 19:31 ata-TOSHIBA_HDWD130_Z6DP2RVAS-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Mar 10 19:31 ata-TOSHIBA_HDWD130_Z6DP2RVAS-part9 -> ../../sda9
lrwxrwxrwx 1 root root 9 Mar 10 19:31 wwn-0x5000039fe6c99611 -> ../../sda
lrwxrwxrwx 1 root root 10 Mar 10 19:31 wwn-0x5000039fe6c99611-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Mar 10 19:31 wwn-0x5000039fe6c99611-part9 -> ../../sda9
lrwxrwxrwx 1 root root 9 Mar 10 19:31 wwn-0x5000cca221c7ef41 -> ../../sdd
lrwxrwxrwx 1 root root 9 Mar 10 19:31 wwn-0x5000cca221c810df -> ../../sdb
lrwxrwxrwx 1 root root 9 Mar 10 19:31 wwn-0x5000cca221c813f1 -> ../../sdc
