XigmaNAS

Posted: **12 Mar 2017 04:39**

# Summary

- 3 x 3tb _raidz1-0_
- Created with Nas4Free
- Migrated to Linux/Debian and zol a few month ago.
- Now in FAULTED state (The pool metadata is corrupted) http://zfsonlinux.org/msg/ZFS-8000-72

I'd love to be able to recover some data. I think I've reached my knowledge limit. =) Please help!

# History

1. ~4 weeks ago: `zpool scrub` reports checksum errors on sdb.
2. ~2 weeks ago: `zpool scrub` says sdb has too many errors and should be replaced. I order a new drive.
3. ~2 days ago: I install the new drive. (note that sdb got renamed to sdd after I added the new drive). I attempt to replace the bad drive (sdd) with the new drive (sda)

Code: Select all

     
        402  zpool replace bigbubba0 sdd sda
        403  zpool replace bigbubba0 sdd sda -f
        404  zpool replace bigbubba0 sdd sda -f -o ashift=12
        405  zpool replace bigbubba0 sdd sda -f -o ashift=4

4. zpool complains about sector alignment differing. I attempt 12 and that fails. It eventually accepts 4 and resilvering starts.
5. After 8h only 500mb has been completed. I give it another 8h and it hasn’t made any progress. I figure the bad drive has totally bit it and reboot the system.
6. Reboot doesn’t help. I attempt to abort:

Code: Select all

    
        422  zpool detach bigbubba0 sda
        430  zpool clear -F bigbubba0
        434  zpool detach bigbubba0 sda

7. After this step I am left with error http://zfsonlinux.org/msg/ZFS-8000-72 — I tried several things to recover:

Code: Select all

     
        440  zpool import -V
        441  zpool import -fX
        442  zpool import -fX bigbubba0
        456  zpool import -fD bigbubba0
        457  zpool import -f bigbubba0
        463  zpool import -fFX -o ro -o failmode=continue
        464  zpool import -f -N -d /dev/disk/by-id/ata-Hitachi_HDS722020ALA330_JK1130YAGKS1VT bigbubba0
        465  zpool import -f -N -d /dev/disk/by-id/ata-Hitachi_HDS722020ALA330_JK1130YAGKSW6T bigbubba0
        466  zpool import -f -N -d /dev/sda bigbubba0
        467  zpool import -f -N -d /dev/sdb bigbubba0
        468  zpool import -f -N -d /dev/disk/by-id/bigbubba0
        469  zpool import -f -N -d /dev/disk/by-id/ bigbubba0
        477  zpool import -d /dev/disk/by-id
        478  zpool import -o readonly=on bigbubba0
        479  zpool import -d /dev/sdd
        481  zpool import -d /dev/sda
        482  zpool import -d /dev/disk/by-uuid/8140ce08-86d4-49ca-81df-a39eedcb1cab
        483  zpool import -d /dev/disk/by-uuid/d42b4883-8800-4ce4-a9be-868709fbf952

None of this helped…. And I have reached the point where I realize I don’t know enough to fix the problem. Please help!

# Output

**# zpool import**

Code: Select all

     
     pool: bigbubba0
       id: 16328167560584328470
    state: FAULTED
    status: The pool metadata is corrupted.
    action: The pool cannot be imported due to damaged devices or data.
    The pool may be active on another system, but can be imported using
    the '-f' flag.
     see: http://zfsonlinux.org/msg/ZFS-8000-72
    config:

    bigbubba0        FAULTED  corrupted data
      raidz1-0       ONLINE
        sdc          ONLINE
        sdb          ONLINE
        replacing-2  ONLINE
          sdd        ONLINE
          sda        ONLINE

**# uname -a**

Code: Select all

     
    Linux bb2 3.16.0-4-amd64 #1 SMP Debian 3.16.39-1+deb8u2 (2017-03-07) x86_64 GNU/Linux

**# dkms status**

Code: Select all

     
    spl, 0.6.5.9, 3.16.0-4-amd64, x86_64: installed
    zfs, 0.6.5.9, 3.16.0-4-amd64, x86_64: installed

**$ zdb**

Code: Select all

     
    bigbubba0:
        version: 5000
        name: 'bigbubba0'
        state: 0
        txg: 41072035
        pool_guid: 16328167560584328470
        errata: 0
        hostid: 8323329
        hostname: 'bb2'
        vdev_children: 1
        vdev_tree:
            type: 'root'
            id: 0
            guid: 16328167560584328470
            children[0]:
                type: 'raidz'
                id: 0
                guid: 4280100286181462697
                nparity: 1
                metaslab_array: 14
                metaslab_shift: 32
                ashift: 9
                asize: 6001182375936
                is_log: 0
                children[0]:
                    type: 'disk'
                    id: 0
                    guid: 714237933490165329
                    path: '/dev/sdc'
                    phys_path: '/dev/ada0'
                    whole_disk: 0
                children[1]:
                    type: 'disk'
                    id: 1
                    guid: 13042370445854942928
                    path: '/dev/sdb'
                    phys_path: '/dev/ada1'
                    whole_disk: 0
                children[2]:
                    type: 'replacing'
                    id: 2
                    guid: 9626083375216793335
                    whole_disk: 0
                    children[0]:
                        type: 'disk'
                        id: 0
                        guid: 14340659031728992743
                        path: '/dev/sdd'
                        phys_path: '/dev/ada2'
                        whole_disk: 0
                    children[1]:
                        type: 'disk'
                        id: 1
                        guid: 13403179909814545905
                        path: '/dev/sda1'
                        whole_disk: 1
                        DTL: 65990
                        resilver_txg: 41072032
        features_for_read:
            com.delphix:hole_birth
            com.delphix:embedded_data

**# zdb -s bigbubba0**

Code: Select all

     
    zdb: can't open 'bigbubba0': Input/output error

**# ls -l /dev/disk/by-id**

Code: Select all

     
    total 0
    lrwxrwxrwx 1 root root  9 Mar 10 19:31 ata-Hitachi_HDS722020ALA330_JK1130YAGKG37T -> ../../sdd
    lrwxrwxrwx 1 root root  9 Mar 10 19:31 ata-Hitachi_HDS722020ALA330_JK1130YAGKS1VT -> ../../sdb
    lrwxrwxrwx 1 root root  9 Mar 10 19:31 ata-Hitachi_HDS722020ALA330_JK1130YAGKSW6T -> ../../sdc
    lrwxrwxrwx 1 root root  9 Mar 10 19:31 ata-TOSHIBA_HDWD130_Z6DP2RVAS -> ../../sda
    lrwxrwxrwx 1 root root 10 Mar 10 19:31 ata-TOSHIBA_HDWD130_Z6DP2RVAS-part1 -> ../../sda1
    lrwxrwxrwx 1 root root 10 Mar 10 19:31 ata-TOSHIBA_HDWD130_Z6DP2RVAS-part9 -> ../../sda9
    lrwxrwxrwx 1 root root  9 Mar 10 19:31 wwn-0x5000039fe6c99611 -> ../../sda
    lrwxrwxrwx 1 root root 10 Mar 10 19:31 wwn-0x5000039fe6c99611-part1 -> ../../sda1
    lrwxrwxrwx 1 root root 10 Mar 10 19:31 wwn-0x5000039fe6c99611-part9 -> ../../sda9
    lrwxrwxrwx 1 root root  9 Mar 10 19:31 wwn-0x5000cca221c7ef41 -> ../../sdd
    lrwxrwxrwx 1 root root  9 Mar 10 19:31 wwn-0x5000cca221c810df -> ../../sdb
    lrwxrwxrwx 1 root root  9 Mar 10 19:31 wwn-0x5000cca221c813f1 -> ../../sdc

Posted: **12 Mar 2017 11:15**

sorry, ZFS guru is needed for your problem.

for me "The pool metadata is corrupted." is a lost of all data, but perhaps someone guru can help you.

Posted: **12 Mar 2017 13:11**

Created with Nas4Free
ZFS v28 or 5000 ?

Posted: **12 Mar 2017 15:26**

It was created with v28, upgraded to 5000, both with Nas4Free. On zol a `zpool upgrade` was done to add a few feature flags from zol.

Posted: **12 Mar 2017 19:13**

After v28 there are plenty of examples of metadata loss after migrations to zol or even to FreeNas, these problems sometimes come after months of use without apparent explanations and in any case the data is lost.

Posted: **12 Mar 2017 19:51**

Wow that really stinks. I am taking disk images of the disks now just in case there is hope to restore... and then will start over. =(

XigmaNAS

Resilvering failed, now faulted state with corrupted metadata. Help!

Resilvering failed, now faulted state with corrupted metadata. Help!

Re: Resilvering failed, now faulted state with corrupted metadata. Help!

Re: Resilvering failed, now faulted state with corrupted metadata. Help!

Re: Resilvering failed, now faulted state with corrupted metadata. Help!

Re: Resilvering failed, now faulted state with corrupted metadata. Help!

Re: Resilvering failed, now faulted state with corrupted metadata. Help!