ZFS configuration corrupted??

[n8n]

Hello.
I originally thought I had the same issue as here viewtopic.php?f=66&t=9182&p=56700#p56700
But my issue seems to be deeper. Or at least I'm sure I no longer know what to do.
I have 2 mirrored pools, and decided I was going to correct the advanced format message.
I copied all my data onto one pool, leaving one set of disks empty. I thought I had removed the pool via the gui, and I learned about the faq where one needs to write the script to remove the bit at the beginning and end of the disk after a reformat, then export the pool and reimport in order to get the ashift correct. At one point I actually received a response to the ashift query that made me believe all was good -12.
Now my situation is that every time I reboot my system, the pool comes up as faulted. I had thought my issue was that I used the same name as previous, and I used a new name, rebooting produced the same message that the drives weren't there, and named the old pool name, despite my having destroyed the pool name.
I've gone around in many circles using clear, import, export, destroy, detach. Every time I either think I have success to destroy the pool name, or I have the pool online and stable, after a reboot I get the following message after <zpool status>
pool: set1
state: UNAVAIL
status: One or more devices could not be opened. There are insufficient
replicas for the pool to continue functioning.
action: Attach the missing device and online it using 'zpool online'.
see: http://illumos.org/msg/ZFS-8000-3C
scan: none requested
config:

NAME STATE READ WRITE CKSUM
set1 UNAVAIL 0 0 0
mirror-0 UNAVAIL 0 0 0
9336664799176194296 UNAVAIL 0 0 0 was /dev/ada2
10676280146457818980 UNAVAIL 0 0 0 was /dev/ada3

I tried to do a synchronize regardless, and received this message from the gui.
Warning: Invalid argument supplied for foreach() in /usr/local/www/disks_zfs_config_sync.php on line 346 Warning: Cannot modify header information - headers already sent by (output started at /usr/local/www/disks_zfs_config_sync.php:346) in /usr/local/www/disks_zfs_config_sync.php on line 468

I actually at one point had one disk detached, and wasn't able to reattach it, but rebooting seems to clear the healthy pool and replace it with the faulted one.

This issue is way beyond my skill level. I'm hoping there's some quick modification I can do that I haven't found in the forums yet.
Bossman's link to the Oracle zpool admin database was a good read http://docs.oracle.com/cd/E19253-01/819 ... index.html
But all I could decipher from it was that I should find the zpool.cache file and delete it. I think that would be a mistake before I ask for help.

Any suggestions? I'm all ears.
Thanks

[Parkcomm]

Have a look here viewtopic.php?f=66&t=10133 - if this is the fault (stale /cf/boot/zfs/zpool.cache) your timing is great, this was a b*tch to diagnose.

[n8n]

ok, thanks for helping!
So my post has changed a lot while I was creating it. because I've found with a clearer head after reading your posts, I was able to get the pool to show up with the /dev names, as opposed to labels.
I'm sure I have a similar issue with the .cache file(s) being stale.
To make the post more readable, I've not included alot of the the text of the PuTTY session...

First zdb -l /dev/ada2 shows me that zfs thinks I'm using labels....I've never set up labels.
Next zpool import tells me that I have one pool available to import.
zpool export allows me to export
so I try to reimport

Code: Select all

 zpool import
   pool: set1
     id: 13401760060581711676
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

        set1        ONLINE
          ada3      ONLINE
nas4free: ~ # zpool import set1
nas4free: ~ # zpool list
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
set1  1.81T   816K  1.81T     0%  1.00x  ONLINE  -
vd00  1.82T  1.60T   227G    87%  1.00x  ONLINE  -

Cool, so now I try to reattach the "missing" drive. In fact, I tried this the other day, and must have gotten the syntax wrong, or misunderstood the man pages, 'cause today, it reattached.
I'm going to reboot, but before I do, I'll take note from the post (and sub-posts) mentioned.

Code: Select all

 zdb -C
set1:
    version: 5000
    name: 'set1'
    state: 0
    txg: 259
    pool_guid: 13401760060581711676
    hostid: 2583737530
    hostname: 'nas4free.local'
    vdev_children: 1
    vdev_tree:
        type: 'root'
        id: 0
        guid: 13401760060581711676
        children[0]:
            type: 'mirror'
            id: 0
            guid: 825851342103262817
            whole_disk: 0
            metaslab_array: 34
            metaslab_shift: 34
            ashift: 12
            asize: 2000394125312
            is_log: 0
            create_txg: 4
            children[0]:
                type: 'disk'
                id: 0
                guid: 8168463133462030097
                path: '/dev/ada3'
                phys_path: '/dev/ada3'
                whole_disk: 1
                DTL: 39
                create_txg: 4
            children[1]:
                type: 'disk'
                id: 1
                guid: 16322115279715544611
                path: '/dev/ada2'
                phys_path: '/dev/ada2'
                whole_disk: 1
                DTL: 38
                create_txg: 4
                resilver_txg: 256
    features_for_read:
        com.delphix:hole_birth

Code: Select all

nas4free: ~ # ls -la /boot/zfs/zpool.cache
-rw-r--r--  1 root  wheel  3296 Dec 26 21:19 /boot/zfs/zpool.cache
nas4free: ~ # ls -la /cf/boot/zfs/zpool.cache
-rw-r--r--  1 root  wheel  3068 Apr 12  2015 /cf/boot/zfs/zpool.cache

This looks promising - last night I receive the same error as fletch, that the file .cache file didn't exist.
Just for fun I'll take the advice from parkcomm

Re: My raidz2 array comes up faulted when disk is replaced
Quote
Postby Parkcomm » December 23rd, 2015, 3:31 pm

you can just delete /cf/boot/zfs/zpool.cache and reboot

So having gotten this far, this is where I get an unexpected surprise. Logged in as root, I am unable to delete the file, even after finding it and trying to use chmod on it.
<gaaak> what now??? I'm not going to reboot my system until I can figure out how I can delete this file even though I'm logged in as root, as I believe that when I reboot, I'll start over with the faulted error message.
The webGUI won't allow me to delete the file either.

Code: Select all

nas4free: boot # chmod ug+w zfs
chmod: zfs: Read-only file system
nas4free: boot # ls -al
total 366
drwxr-xr-x  5 root  wheel     512 Feb  9  2015 .
drwxr-xr-x  5 root  wheel     512 Feb  9  2015 ..
-r--r--r--  1 root  wheel    8192 Feb  9  2015 boot
-r--r--r--  1 root  wheel    2937 Feb  9  2015 brand.4th
-r--r--r--  1 root  wheel    5664 Feb  9  2015 check-password.4th
-r--r--r--  1 root  wheel    1869 Feb  9  2015 color.4th
drwxr-xr-x  2 root  wheel     512 Feb  9  2015 defaults
-r--r--r--  1 root  wheel    3914 Feb  9  2015 delay.4th
-r--r--r--  1 root  wheel     788 Feb  9  2015 device.hints
-r--r--r--  1 root  wheel    2620 Feb  9  2015 frames.4th
drwxr-xr-x  2 root  wheel   16384 Feb  9  2015 kernel
-r-xr-xr-x  1 root  wheel  229376 Feb  9  2015 loader
-r--r--r--  1 root  wheel    6675 Feb  9  2015 loader.4th
-rw-r--r--  1 root  wheel     339 Feb  9  2015 loader.conf
-rw-r--r--  1 root  wheel      80 Feb  9  2015 loader.rc
-r--r--r--  1 root  wheel    7291 Feb  9  2015 menu-commands.4th
-rw-r--r--  1 root  wheel    4236 Feb  9  2015 menu.4th
-r--r--r--  1 root  wheel     697 Feb  9  2015 screen.4th
-r--r--r--  1 root  wheel    2615 Feb  9  2015 shortcuts.4th
-r--r--r--  1 root  wheel   35598 Feb  9  2015 support.4th
-r--r--r--  1 root  wheel    2846 Feb  9  2015 version.4th
drwxr-xr-x  2 root  wheel     512 Apr 12  2015 zfs
nas4free: boot # cd zfs
nas4free: zfs # rm zpool.cache
override rw-r--r--  root/wheel for zpool.cache? y
rm: zpool.cache: Read-only file system

[n8n]

<Learning How to Fish>
So reading the post again about 15x, and figuring out that cf actually does pertain to me.....
Following the command set that fletchowns used, I noticed that when
umount -v was used, my output was

Code: Select all

nas4free: ~ # mount -v
/dev/xmd0 on / (ufs, local, noatime, acls, writes: sync 386 async 25, reads: sync 670 async 13, fsid 1c69d95431e3c6ef)
devfs on /dev (devfs, local, multilabel, fsid 00ff007171000000)
/dev/xmd1 on /usr/local (ufs, local, noatime, soft-updates, acls, writes: sync 2 async 21, reads: sync 1024 async 0, fsid 1d69d9544a0d12a0)
/dev/da0s1a on /cf (ufs, local, read-only, reads: sync 9 async 0, fsid 2d6bd954b21f39d0)
procfs on /proc (procfs, local, fsid 01ff000202000000)
vd00 on /mnt (zfs, local, nfsv4acls, fsid e066419dde55ac02)
/dev/xmd2 on /var (ufs, local, noatime, soft-updates, acls, writes: sync 130 async 299, reads: sync 9 async 0, fsid b8d881561b0a4555)
tmpfs on /var/tmp (tmpfs, local, fsid 02ff008787000000)

so I figured that I should be making reference to /dev/da0s1a (just as Parkcomm wrote), as opposed to /dev/da0a that fletchowns used.
umount -f /cf
mount -w /dev/da0s1a /cf

Now I see that my drives are back online without errors!! <Party......Party.......>

Big Thanks to Parkcomm for the troubleshooting,
Big Thanks to fletchowns for being thorough in the post.
Mark as SOLVED.

[Parkcomm]

You are welcome