Degraded ZFS... may have blundered

[codad]

NOTE: since I am using the older version of FreeNAS, I was told to come here for help... I will likely upgrade as soon as I get the zfs back and fully functional

First let me state that my FreeNAS server was assembled 2+ years ago and has been working great with no meddling on my part (also why I am still on the old version: FreeNAS 0.7.1 (Shere))

Regardless, my problem started a few days ago, and it basically indicated that the da2 drive had changed label to 15159316559305955361 (not sure why that happened, as it used to be dev/da2)

Then I spent quite some time reading various posts (and the FAQ here) and it seemed I had arrived at a solution on how to fix it, the last was to reformat the drive since it was still functional (so I did, using one of my Windows machines), and put the drive back in my server.

However, the steps definitely did not give me the expected result, and I am in quite a pickle as to what to do next. Please see the information readout from my pools/information listing

Code: Select all

  pool: tank
 state: DEGRADED
 scrub resilver completed with 0 errors on Tue Aug 14 10:56:33 2012
config:	NAME                        STATE     READ WRITE CKSUM
	tank                        DEGRADED     0     0     0
	  raidz2                    DEGRADED     0     0     0
	    da0                     ONLINE       0     0     0
	    da1                     ONLINE       0     0     0
	    replacing               UNAVAIL      0     0     0  insufficient replicas
	      da2/old               UNAVAIL      0     0     0  cannot open
	      15159316559305955361  FAULTED      0     0     0  was /dev/da2
	    da3                     ONLINE       0     0     0
	    da4                     ONLINE       0     0     0
	    da5                     ONLINE       0     0     0
	    da6                     ONLINE       0     0     0
errors: No known data errors

I can still access my server (thank goodness), but what should my next steps be? Given that I am on the older version of FreeNAS and using raidz2, I was hoping to wait to upgrade the OS until 8.3 came out... bad idea?

If I need to reformat the 2TB drive again, is there a simpler way than using my Windows machine (it took all night!)?

Thank you for your patience and also being as specific as possible with instructions to help.

EDIT1:
Note that I reformatted the drive using NTFS
And, on the normal disk/management readout, the drive shows as online:

Code: Select all

da0 1907730MB WDC WD2001FASS-00U0B R001 n/a 60 ZFS storage pool device ONLINE 
da1 1907730MB WDC WD2001FASS-00U0B R001 n/a 60 ZFS storage pool device ONLINE 
da2 1907730MB WDC WD2001FASS-00U0B R001 n/a 60 ZFS storage pool device ONLINE 
da3 1907730MB WDC WD2001FASS-00U0B R001 n/a 60 ZFS storage pool device ONLINE 
da4 1907730MB WDC WD2001FASS-00U0B R001 n/a 60 ZFS storage pool device ONLINE 
da5 1907730MB WDC WD2001FASS-00U0B R001 n/a 60 ZFS storage pool device ONLINE 
da6 1907730MB WDC WD2001FASS-00U0B R001 n/a 60 ZFS storage pool device ONLINE 
da7 1907730MB WDC WD2001FASS-00U0B R001 n/a 60 ZFS storage pool device ONLINE

EDIT2
I just noticed on the GUI that I can go into a Disks / Format section, which does show two drives available for formatting: da2 and da7... should I give that a go, since da2 in the zpool is faulted/unavailable anyway?

[codad]

My gut tells me that my immediate next step should be to format the drive using the GUI for da2 (and set the file system dropdown to ZFS Storage Pool Device)... Any experts out there that can give me the thumbs up or down on this move?

[Onichan]

Well considering that you have formatted it my guess is you need to actually remove the old drive from the pool and readd it and resilver it. I am no expert though.

[codad]

The strange thing is that (from the web GUI) there is nothing for me to remove - when I pick that tool command, no da2 is listed... there is a checkbox available that just has a comma next to it (unlike the option next to all the functional drives, where the drive label is listed), but when I try to remove it, it gives me a "missing device" syntax error

[Onichan]

I think you have to do that through cli. According to oracle you are first suppose to offline the old disk, power down the server, physically replace the disk, then run the zpool replace command using the new disk http://docs.oracle.com/cd/E19253-01/819 ... index.html but you have already replaced the disk so I am not sure.

[codad]

It would be great to get the exact process in light of the Information status I am getting - I cannot replace da2 (doesn't work), and I am not sure what else I can enter on the cli. Also, I think I need to somehow STOP the "replacing" effort already underway (but that isn't actually doing anything).

[codad]

I was able to stop the replacement by using the cli command
zpool detach tank 15159316559305955361

And now I am at this point in pools/information

Code: Select all

  pool: tank
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
	the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-D3
 scrub resilver completed with 0 errors on Tue Aug 14 14:15:09 2012
config:
	NAME         STATE     READ WRITE CKSUM
	tank         DEGRADED     0     0     0
	  raidz2     DEGRADED     0     0     0
	    da0      ONLINE       0     0     0
	    da1      ONLINE       0     0     0
	    da2/old  UNAVAIL      0     0     0  cannot open
	    da3      ONLINE       0     0     0
	    da4      ONLINE       0     0     0
	    da5      ONLINE       0     0     0
	    da6      ONLINE       0     0     0
errors: No known data errors

What should my next step be? Do I need to reboot the server? Then reformat the da2 drive again? How do I get it added back into the pool?

NOTE: I have tried to use the command: zpool online tank da2 (but I get the message "no such device in pool")... This DESPITE that the normal disks/mgmt listing shows da2 as being there, online, and formatted for zfs.... HELP!!!

[codad]

I have done a few extra things that I discovered from various posts, summarized below:
Sometimes a format command doesn't do anything unless you make sure the first and last sectors are cleared... so I did that using the following two commands

Code: Select all

dd if=/dev/zero of=/dev/da2 bs=1m count=4
dd if=/dev/zero of=/dev/da2 bs=1m oseek=`diskinfo da2 | awk '{print int($3 / (1024*1024)) - 4;}'`

Both commands executed fine, and very quickly
After rebooting, the status of the zpool has still not changed...

Code: Select all

        NAME         STATE     READ WRITE CKSUM
        tank         DEGRADED     0     0     0
          raidz2     DEGRADED     0     0     0
            da0      ONLINE       0     0     0
            da1      ONLINE       0     0     0
            da2/old  UNAVAIL      0     0     0  cannot open
            da3      ONLINE       0     0     0
            da4      ONLINE       0     0     0
            da5      ONLINE       0     0     0
            da6      ONLINE       0     0     0

Synchronization didn't do anything. I have also tried to simply follow that with various versions of 'online' or 'replace' commands, and basically all commands come back telling me "no such device in pool" (for da2)

However, the drive IS there, and SMART says working perfectly! I need help figuring out what to do to repair this zpool - is it perhaps a config file that is not being updated right? Or, should I do something else to reformat the drive itself? I am truly baffled. And, why does it keep listing "da2/old" in the status?

[Onichan]

I am sorry I am no expert and haven't had the chance to test this scenario yet. It actually is on my todo list sometime in the next week. Only thing I can think of is you need to get the pool to remove that da2/old somehow then hopefully add the new da2 back.

[codad]

Anyone out there have some suggestions? I really have no idea what to do at this point

[codad]

I have noticed something odd... when I enter the pools/tools area and try to select any of the various commands (e.g. offline), the list of devices includes all of the working drives AND one other that simply has a comma (there is no da2)... but when I check the box next to the comma and execute the command, it tells me there is "no device selected".

Is it possible that something on the USB drive / GUI has become out of sync with what is actually happening in the server itself?

NOTE: I have also tried to use a different drive completely (physically replacing the working drive with another working drive)

I am getting super desperate here... any experts with zfs checking this thread?

[Onichan]

You can resync what the pool physically has configured with the GUI. In the ZFS secion in the GUI there is a Synchronize section to do that. I am pretty sure you are going to need to use cli to fix this I am just not sure what. Have you tried removing da2 physically from the server then trying to remove it from the pool? I think that is the best way. Once it doesn't see the old/new da2 or any faulty disks then try inserting da2 back into the server and try adding it back to the pool.

[codad]

I have tried resyncing using the GUI... but I have not yet tried turning off the machine, removing the HD, and booting it up WITHOUT the drive in there - I will give that a shot and see what happens

[codad]

Tried what you suggested & I mentioned above... and nothing really changed. When the HD was out, the zpool did not change (even though the actual disk status obviously showed one less drive online). When I check the zfs status online, it says the same as before. Should I try to delete the drive from the disks/mgmt gui (while it is actually online)?

Not sure if that would really help the zpool any, but SOMETHING is causing the da2/old thing to keep hanging around, AND something is causing zpool to NOT detect the functional drive when it is online... this is crazy frustrating

[Onichan]

I think while the drive is physically removed from the server you need to remove it from the pool. Like I said earlier it might not work by the gui and you might need to use the cli.

Then try plugging it back in and readding it.

[stangri]

codad -- before you physically disconnected the drive from the machine, did you "offline" it via CLI or GUI first?

I've started having problems with one of the drives a while ago, so I offlined it, disconnected it then had to run the box without the replacement for a few months but once I plugged the new one in and put it online I didn't have any problems (well, once the scrub was completed).

[Onichan]

So I just now tested something similar to what it sounds like you have done codad. I removed a disk then connected it via MoBo SATA instead of RAID card so it was assigned a different device name. So now my zfs shows degraded with some number where the da1 was and says was /dev/da1 at the end. I first had to go into disk management and add the new ada1 disk. Then I went into the gui and formatted the now ada1 to zfs. Then I just ran "zpool replace pool da1 ada1" from the command line and it now says its re-silvering.
So I would first check that you have added the formatted disk via disk management. Then go to the Disks/Format and format it as ZFS. Then try doing a "zpool replace tank da2 newdiskyoujustadded".

[misterredman]

I had a similar situation in the past using FreeNAS when replacing a failed disk and the new disk failed too: I found myself unable to replace the disk and I was stuck with the "insufficient replicas" message.
The solution was booting from Solaris 11 express liveCD (there is probably a newer version now; a NAS4Free livecd may also do the trick since it has a newer ZFS version than FreeNAS), importing the array (without upgrading) and detaching one of the "stuck" disks. At that point, exporting and then reimporting in FreeNAS solved the issue and I was able to replace the disk.

Not sure if it still applies since you made many attempts already, but it may be worth a try to use a livecd with a newer version of zfs without upgrading to offline, detach and replace.

[codad]

After all the attempts, I decided to offload all the important files to various PCs/laptops, and doing a fresh install of the latest NAS4Free package... everything went fine, and ofcourse, the drives all showed up fine. Quite a frustrating experience overall, but in fairness, it could easily have been due to me having blundered somewhere along the way.

My two cents: don't do anything in haste trying to fix stuff that you aren't 100% sure about - it seems the OS is pretty unforgiving, and you may end up in the same pickle I was in. Hopefully, these forums will become more active too =)

[Onichan]

Well I am glad in the end you are good, but I rather know of a fix as I think it was fixable.

[codad]

I agree... it would be nice to see an actual fix that worked. I tried everything under the sun EXCEPT the method suggested by misterredman (had already reinstalled by then). Somewhat related to his solution, how do we export a pool and autoimport from the GUI? I suspect, from reading some of the FreeNAS forums, that this may be the solution for getting my HD serial/model #s to show up in the GUI as well.

[Onichan]

I don't know how to do it from the gui, but form cli to export the pool you just do "zpool export" and "zpool import"

[codad]

Before I go ahead and use the CLI for that, what exactly does that command do to the server? Anything risky? I suspect it won't update the GUI listing for the HDs, but I am willing to try it

[Onichan]

I actually think it is the same thing as the synchronize in the webgui, but the zpool export just ensure any pending writes are executed and unmounts the pools from the system. Now if it is in use it wont actually unmount it. It also prepare the pool/drives to be migrated which I don't know exactly what that does, but I guess it just marks them as properly dismounted from a system so the next system knows they are clean and ready. Then you can just run "zpool import pool" on the next system or whenever to import it. Again I think the synchronize actually just does this.

[elliotpl]

This is interesting. When I set up my new NAS, I did some testing. One of them was to disconnect 2 drives and replace them. One did fine, but the other behaved similarly to yours. It gave me OFFLINE or FAULTED state and said to replace. Bu when tried to replace I got something like you. So I hope someone answers you, or at least you will find out how to fix this and share...