Can't import zfs mirror

[asherlazarus]

I had created a simple zfs mirror a long time ago when this was still FreeNAS 7.x

One of my drives died. When I brought up my server it would no longer boot.

I had a spare box I loaded up the latest version of NAS4Free brought over both drives and tried to import the zfs pool to recover my data.

I get the following error:

zpool import -f
pool: files
id: 7904906901605944398
state: FAULTED
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
devices and try again.
The pool may be active on another system, but can be imported using
the '-f' flag.
see: http://illumos.org/msg/ZFS-8000-3C
config:

files FAULTED corrupted data
mirror-0 FAULTED corrupted data
1721085329635625914 FAULTED corrupted data
5177942592732901939 UNAVAIL cannot open

It says to use the -f flag however when I do I get the same error.

There is a hardware fault in the UNAVAIL drive the bios won't recognize it. Is there anyway to get past this error?

I have a replacement drive. Could I copy the working drive and give it the ID of the failed drive and then do the import?

Since I was doing a Mirror all or at least most of my data is on the good drive. How can I get to it? Can I convert it to a non mirror and just access it?

[Parkcomm]

Nope - ZFS is telling you both drives have problems, so you do not have one good copy. If you copy the drive you will be copying the fault.

If your mirrored disk is OK you do not need the other drive, one drice will import and work if it OK.

Here a suggestion

ZFS might not be able to find 5177942592732901939 because the drive number has changed. (for instance this drive was in /dev/ada0 on the old system and is now in /dev/ada1). If you can remedy this ZFS might find it.

If you don't know what the device number name is run:

zdb -C -e yourZFSfileSystemName

The output sill include (the numbers you have above) followed by the path. Copy the output here if you need a hand reading it.

Obviously this is assuming the drives are OK - check you S.M.A.R.T status, cables etc.

[asherlazarus]

I have one drive with corrupted data (1721085329635625914) and I had one drive that was completely dead (5177942592732901939). Is it possible to recover some data from the corrupted drive?

I used dd to create a copy of the corrupted drive and edited the GUIDs in the 4 vdev labels to the missing drive value(5177942592732901939)

When I do a zpool import it still says UNAVAIL but it now says corrupted data vs cannot open after that see below:

Is there another value I have to edit in the vdev labels to make it recognize the drive? I don't expect miracles, but is it possible to recover any of my data?

I found the hidden -V switch in zpool import which will allow me to import the pool but, it's marked as unavail so I can't split or remove the dead drive.

$ zpool import
pool: files
id: 7904906901605944398
state: FAULTED
status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.
The pool may be active on another system, but can be imported using
the '-f' flag.
see: http://illumos.org/msg/ZFS-8000-5E
config:

files FAULTED corrupted data
mirror-0 FAULTED corrupted data
1721085329635625914 FAULTED corrupted data
5177942592732901939 UNAVAIL corrupted data

Here is the entire vdev lavel and the first uberblock

$ zdb -l -u /dev/ada1
--------------------------------------------
LABEL 0
--------------------------------------------
version: 13
name: 'files'
state: 0
txg: 7975053
pool_guid: 7904906901605944398
hostid: 2180312168
hostname: 'unset'
top_guid: 14069723223106526422
guid: 1721085329635625914
vdev_tree:
type: 'mirror'
id: 0
guid: 14069723223106526422
metaslab_array: 14
metaslab_shift: 32
ashift: 9
asize: 1500297035776
is_log: 0
children[0]:
type: 'disk'
id: 0
guid: 1721085329635625914
path: '/dev/ad4'
whole_disk: 0
DTL: 372
children[1]:
type: 'disk'
id: 1
guid: 5177942592732901939
path: '/dev/ad6'
whole_disk: 0
DTL: 371
Uberblock[0]
magic = 0000000000bab10c
version = 13
txg = 7988224
guid_sum = 10426913973371447057
timestamp = 1440682419 UTC = Thu Aug 27 13:33:39 2015

[Parkcomm]

The first thing to look at is the dev numbers:

are the disks /dev/ada4 and /dev/ada6

If not can you move the disks so that you get these numbers?

[raulfg3]

really difficult. if your datais in valuable, please consider a proffessional recover entreprise

[asherlazarus]

So I started back on freebsd 7.x and the zfs version is 13. The drives are now actually ada1 & ada2 but there are links in /dev ad4->ada1 and ad6->ada2 which is the correct mapping I have the drives inn the correct locations.

Is there anyone on the boards that is a zfs or zfs metadata guru? I've been reading around online it looks like mirror are supposed to be two drives with identical data. Since I used dd to replicate the drive and then updated the GUIDs. I double checked all 4 labels they are correct. I don't understand why the replacement drive is listed as unavail.

[Parkcomm]

There is no point in dd-ing the corrupted drive other than making a backup. If one copy in the mirror is working then pool import will work.

Pool import can be confused by changing drive number, so one very common reason disks are UNAVAIL when moving between machines, it is looking in the old slot number.

if you want to dd a drive (as i said it probably won't help) then the 12th line of each label (4 times - 2 at the start of the disk and 2 at the end) on /dev/ada4 needs to be

Code: Select all

guid: 1721085329635625914

and on /dev/ada6 ends to be

Code: Select all

guid: 5177942592732901939

That is the only difference between the two halves of a mirror.

I know I am focussing on something you have ruled out but the fix is so simple and perfect I just want to check. Are you sure the link to /dev/ada0 and /dev/ada1 are working? Also if they are not working you could edit to equivalent lines in the metadata to point to the new device.

[asherlazarus]

Here are the pertinant parts of:
$ ls -l /dev
lrwxr-xr-x 1 root wheel 4 Feb 21 18:45 ad4 -> ada1
lrwxr-xr-x 1 root wheel 4 Feb 21 18:45 ad6 -> ada2
crw-r----- 1 root operator 0x65 Feb 21 18:45 ada1
crw-r----- 1 root operator 0x6d Feb 21 18:45 ada1p1
crw-r----- 1 root operator 0x6f Feb 21 18:45 ada2
crw-r----- 1 root operator 0x78 Feb 21 18:45 ada2p1

the ad4 & ad6 links work with the zdb command:

$ zdb -l /dev/ad4
--------------------------------------------
LABEL 0
--------------------------------------------
version: 13
name: 'files'
state: 0
txg: 7975053
pool_guid: 7904906901605944398
hostid: 2180312168
hostname: 'unset'
top_guid: 14069723223106526422
guid: 1721085329635625914
vdev_tree:
type: 'mirror'
id: 0
guid: 14069723223106526422
metaslab_array: 14
metaslab_shift: 32
ashift: 9
asize: 1500297035776
is_log: 0
children[0]:
type: 'disk'
id: 0
guid: 1721085329635625914
path: '/dev/ad4'
whole_disk: 0
DTL: 372
children[1]:
type: 'disk'
id: 1
guid: 5177942592732901939
path: '/dev/ad6'
whole_disk: 0
DTL: 371
--------------------------------------------

$ zdb -l /dev/ad6
--------------------------------------------
LABEL 0
--------------------------------------------
version: 13
name: 'files'
state: 0
txg: 7975053
pool_guid: 7904906901605944398
hostid: 2180312168
hostname: 'unset'
top_guid: 14069723223106526422
guid: 5177942592732901939
vdev_tree:
type: 'mirror'
id: 0
guid: 14069723223106526422
metaslab_array: 14
metaslab_shift: 32
ashift: 9
asize: 1500297035776
is_log: 0
children[0]:
type: 'disk'
id: 0
guid: 1721085329635625914
path: '/dev/ad4'
whole_disk: 0
DTL: 372
children[1]:
type: 'disk'
id: 1
guid: 5177942592732901939
path: '/dev/ad6'
whole_disk: 0
DTL: 371
--------------------------------------------

I am only including label 0 for each drive but the other 3 are identical on their respective drives.

As you can see between drives the labels are identical except for the "guid:" and as you can see from the vdev_tree: and the zdb commands I have the correct "guid:" in the correct locations.

When you say the 12th line of each label your not talking about the formatted output from zdb -l command. What did you mean? Is documentation anywhere that would tell me how to edit a nvlist with a hex editor? Does the vdev_label metadata have a checksum?

One thing I was considering is there a way to change the meta data to just have a single drive with no mirror.

[asherlazarus]

So I have gotten a little bit further in this and I know why it is refusing to recognize my replacement drive. All metadata has a checksum usually SHA-256 this is stored at the end of the field so in the case of the vdev label 0 at hex address 0x1FFE0 and it 4 64 bit words composing the 256 bits of the label checksum an offset location is added to the checksum calculation. The net result of which is the checksums are different at the 4 vdev label locations. Is there any easy way to compute corrected checksums for the vdev labels? Is there anyway to tell zfs to ignore the vdev label checksums?

[Parkcomm]

As you can see between drives the labels are identical except for the "guid:" and as you can see from the vdev_tree: and the zdb commands I have the correct "guid:" in the correct locations.

You have it correct - those are the lines I meant (12th line)

[asherlazarus]

so I have been quite busy:

actually used a system restore cd and zfsonlinux and a hack of zdb to correct one of the labels on the replacement drive.

When I do the import now I get:

root@sysresccd /root % zpool import
pool: files
id: 7904906901605944398
state: DEGRADED
status: The pool was last accessed by another system.
action: The pool can be imported despite missing or damaged devices. The
fault tolerance of the pool may be compromised if imported.
see: http://zfsonlinux.org/msg/ZFS-8000-EY
config:

files DEGRADED
mirror-0 DEGRADED
sdb FAULTED corrupted data
sdc ONLINE

root@sysresccd /root % zpool import files
cannot import 'files': pool may be in use from other system, it was last accessed by unset (hostid: 0x81f4ec68) on Thu Aug 27 21:09:06 2015
use '-f' to import anyway
root@sysresccd /root % zpool import -f files
cannot import 'files': one or more devices is currently unavailable
root@sysresccd /root % zpool import -fm files

I tried the oldest uberblock:

Uberblock[122]
magic = 0000000000bab10c
version = 13
txg = 7987194
guid_sum = 10426913973371447057
timestamp = 1440651520 UTC = Thu Aug 27 04:58:40 2015
Uberblock[123]
magic = 0000000000bab10c
version = 13
txg = 7989115
guid_sum = 10426913973371447057
timestamp = 1440709149 UTC = Thu Aug 27 20:59:09 2015
Uberblock[124]
magic = 0000000000bab10c
version = 13
txg = 7988220
guid_sum = 10426913973371447057
timestamp = 1440682299 UTC = Thu Aug 27 13:31:39 2015
Uberblock[125]
magic = 0000000000bab10c
version = 13
txg = 7989117
guid_sum = 10426913973371447057
timestamp = 1440709209 UTC = Thu Aug 27 21:00:09 2015
Uberblock[126]
magic = 0000000000bab10c
version = 13
txg = 7987198
guid_sum = 10426913973371447057
timestamp = 1440651639 UTC = Thu Aug 27 05:00:39 2015
Uberblock[127]
magic = 0000000000bab10c
version = 13
txg = 7989119
guid_sum = 10426913973371447057
timestamp = 1440709269 UTC = Thu Aug 27 21:01:09 2015
root@sysresccd /root % zpool import -fmFXT7987194 files
cannot import 'files': one or more devices is currently unavailable
root@sysresccd /root % zpool import -fmFXVT7987194 files
pool: files
state: UNAVAIL
status: One or more devices could not be used because the label is missing
or invalid. There are insufficient replicas for the pool to continue
functioning.
action: Destroy and re-create the pool from
a backup source.
see: http://zfsonlinux.org/msg/ZFS-8000-5E
scan: none requested
config:

NAME STATE READ WRITE CKSUM
files UNAVAIL 0 0 0 insufficient replicas
mirror-0 UNAVAIL 0 0 0 insufficient replicas
1721085329635625914 UNAVAIL 0 0 0 was /dev/ad4
5177942592732901939 UNAVAIL 0 0 0 was /dev/ad6

so zfsonlinux is yielding slightly different results than nas4free.

I started scanning the disk with various file recovery softwares. One was able to pull all the file names out but it had the wrong sizes for them.

Any advice on what I should try next?

[Parkcomm]

you could start with this - http://www.joyent.com/blog/zfs-forensic ... oyed-zpool

The main thing it taught me is how bloody had file recovery is on ifs