Can't import zfs-stripe, zpool import hangs indefinetely

[madgabz]

Hi all,

As some of you know, I've been having problems with my zfs-based stripe-array of 2 disks. Yes, I've been living life dangerously, but the data are not THAT important, mostly backups and movies/TV shows! But, it would be a pain to recreate.
So, after suddenly having the array/pool drop out on my old hardware setup (it was starting to do insane amounts of errors on scrubs, so i decided this most likely had to be a hardware fault!), i bought a new mobo+ram+cpu and looked forward to a brand new experience!

But, same things happen when i try to import the pool in my new hardware setup. The pool consists of 2 striped 1 TB disk, for a total of 2 TB storage.
When i do a 'zpool import' it correctly sees the pool as online, but needs to be imported w -f flag:

Code: Select all

nas4free:~# zpool import
   pool: DataRaid1
     id: 5339075205090693951
  state: ONLINE
 status: The pool was last accessed by another system.
 action: The pool can be imported using its name or numeric identifier and
        the '-f' flag.
   see: http://illumos.org/msg/ZFS-8000-EY
 config:

        DataRaid1   ONLINE
          ada1      ONLINE
          ada0      ONLINE

If I issue a 'zpool import -f DataRaid1' the system just hangs forever (left it overnight for a total 8 hours)! I can hear the disks working for app. 5 seconds. then silence!
If I import an nas4free-config from previously, it says under 'Disks:ZFS' that the 'ZFS system is in an unhealthy state' or something, later if i try to clear and re-import pools from web-frontend, it hangs forever again...
FWIW, I tried replacing all SATA-cables, as they've been known to regularly be culprits of many problems!

So, I google'd for this particular hang-problem, the only relevant (to my knowledge) thing that came up was this from the freenas.org forums: zpool import hangs forever, help please!!!
So, after reading this, i tried to import using 'zpool import -fFX DataRaid1', but am getting the same result - it hangs indefinitely!

Now, I'm a fiddler and have absolutely no in-depth knowledge of ZFS, so I'm very reluctant to start fiddling too much with the pool
If any ZFS Zealot could help out here, I'd very much appreciate it!
Or, should I just be done with it, and restart the fileserver from scratch?

Thank You so much in advance!
Cheers, Gabriel S. Hansen

[madgabz]

I might add:
gstat reveals no activity
and also doing this:

Code: Select all

[root@nas4free ~]# ps axl |grep 'zpool import'
    0 9338 7078   0  22  0 16280  2108 piperd   S+    2    0:00.02 grep zpool import
    0 9295 7081   0  21  0 35476  3352 tx->tx_s D+    3    0:01.12 zpool import -f 5339075205090693951

what is going on?
I can't kill or terminate any zpool process, even if I'm root....?

[sleid]

Hello!
Have you tried zpool import-f only

[madgabz]

YEs, same thing
tried: -f, -fF, -F, -fFX, -X

[sleid]

How do you use zpool import-f ?
from web gui, from Putty or from console ?

[madgabz]

I've tried from console and from remote ssh, even using tmux because of all the stalling. killall nor top/htops kill/terminate can get rid of the stalled zpool processes.....
The webgui either stalls ('waiting for webserver....' - i'm on firefox here), or gives a weird php error message, regarding some 'name' variable... I'll try reproduce and post it, but its hardly relevant!
EDIT:
Forgot, Thanks, sleid

[madgabz]

ok, no ZFS zealots out there to the rescue....
I'll take that as a start-over-thumbs-up!

[raulfg3]

sorry, import a ZFS pool is normally an easy operation, only if you have problems, must need to use -f operator, but you case is really extrange and do not have experience on others simmilar.

[raulfg3]

What can I do in your situation ( webare: you can do what you want).

I'll try to change SATA ports for others, ( perhaps you mix SATA controllers and one is not detected), revise SATA Cables and use SATA cables with metalic Clip to avoid false contact.

Do a new install of latest Nas4Free ( full or embeded ), once sucessfully booted, try one more time to import pool ( from shell, use monitor and keyboard atached to your NAS), probably , first time fail, try -f , if fail, normally ZFS say why fail, and post here the message.

PD: Post all you write in shell , and output, is easy to us to detect some write fault or so... (ok; probably you write correctly how to import your pool but is good to be sure)

[madgabz]

First: Big Thank You, raulfg3!
I've already done all you said, even ran nas4free directly from a bootup cd, now running from new install on external USB3-HDD! WHen I get the time, I will wipe and reinstall to USB3-hdd, then post my findings!
Regarding cables, i already said i swapped out the cables 2 or 3 times, i'll check if i can get those cables w clips in the local tech-shop! You have a valid point in that the sata cables are rather flimsy
I also started out using the console via my kbd and monitor. So, that was actually where i began....

The strange thing here is, a simple 'zpool import' reveals the array nicely, but suggests I use the -f option to import, what the exact reason for having to do this is, is beyond me! I was hoping that there were some kind of way to get ZFS to more verbosely shell out some detailed info about what is really going on. So far, I've only come across a few ZFS-related posts somewhere suggesting to use 'procstat -kk -a', but I have no idea what that info is or how to interpret it!

fwiw,
Gabriel

[madgabz]

OK, so I finally got around to download .698beta adn installed it freshly on my USB disk.
I rebooted, configured IP address, went to webgui, enabled SSH, and putty'ed in, here's what the output was:

Code: Select all

nas4free:~# bash
[root@nas4free ~]# zpool import
   pool: DataRaid1
     id: 5339075205090693951
  state: ONLINE
 status: The pool was last accessed by another system.
 action: The pool can be imported using its name or numeric identifier and
        the '-f' flag.
   see: http://illumos.org/msg/ZFS-8000-EY
 config:

        DataRaid1   ONLINE
          ada0      ONLINE
          ada1      ONLINE
[root@nas4free ~]# zpool import -f 5339075205090693951

...after app. 20 secs of disk activity from the implicated disks, nothing happens and zpool just hangs there
I cannot issue ANY other zpool commands, not from webgui and not from shell (i tried new putty session, as well as tmux'ing). I cannot kill any zpool process, not with kill -9 nor killall

[fsbruva]

What about booting a Debian Live CD - and installing zfs-fuse. You might be able to import and backup the data. Or, you might be able to import and then export it in a way that Nas4Free can import.

It is obvious that Nas4Free (and FreeBSD) are not dealing well with the zfs format on those disks.

[madgabz]

Thanks a lot, fsbruva, I already tried ZFSGuru distribution, but that is more or less the same as nas4free, ill try debian now... Will be fun, i guess
What options are actually available for import-function? - it seems, the 'X' option s totally undocumented.... just wondering if there were other?

[madgabz]

OK, now sitting in Debian 7 'wheezy' KDE LiveCD - Exact same thing happens, after installing zfs on linux. doing a 'zpool import' straight up reveals the stripe to be online, but -f ID makes the disks active for app 20 secs, then hangs zpool entirely. Starting up a new terminal, i try zpool import again, and zpool itself is totally hung - I cant even do a zpool status
Oh well.... I guess, i should try some illumos dist now.... even if chances are slim!

[madgabz]

oh well... sitting in OpenIndiana LiveCD 151a .... x86 though... same thing:

Code: Select all

root@openindiana:/jack# zpool import
  pool: DataRaid1
    id: 5339075205090693951
 state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier and
	the '-f' flag.
   see: http://www.sun.com/msg/ZFS-8000-EY
config:

	DataRaid1   ONLINE
	  c2t0d0p0  ONLINE
	  c5t0d0p0  ONLINE
root@openindiana:/jack# zpool import -f DataRaid1

[raulfg3]

I have a question:

When you do a Zpool import ( without -f flag), and ZFS answer status: The pool was last accessed by another system

Your pool is mark as online,

Do you try to use it , and do not force import?.

[madgabz]

No, I'm not trying anything! Only those two disks are attached, and i can't do anything with them, except start all over i guess.... All i'm trying is to import them! When you issue a 'zpool import', that is a regular normal import, obviously it won't work, so I try the -f (and also tried -fFX) forced approach, which just hangs the system!

Maybe I should try and post this to the zfs-support/developer mailing list, if there is any...?

[raulfg3]

ll i'm trying is to import them! When you issue a 'zpool import', that is a regular normal import, obviously it won't work,

well it's not obvious for me, I see that your pool is online, this first time that you do import ( prior to force), so please try to acces to your data if possible.

PD: perhaps the problem can be the mount point, use modificator to define new mount point like describe here:
viewtopic.php?f=66&t=1182#p4346

[madgabz]

Thanks, Raulfg3, that's the first thing I've heard making sense I'll try to re-mount the pool, but - you have to admit - zpool stalling/hanging on a regular import, that IS a bit weird....?
I'm off to family stuff, will try later....

[madgabz]

There's something rotten in the state of Denmark....

Trying to mount still yields a hanging zpool command :/

I tried 'zpool import -R /mnt -f DataRaid1' and it just hangs!

Seeing that this is still importing the pool, isn't it just necessary to mount the pool, or is mounting the same as import?
fwiw, i cannot use the pool in any way, as it is not present in the filesystem at all. How can I use the pool or even just salvage data off of it, without importing/mounting it?

[raulfg3]

mounting it's not necessary, only import.

But you can't import, so sorry I think that your data is lost, and your only option is to reformat your disk, and recreate your pool ( or try to boot from your Old boot disk if still works).

In the first post says that your data is not so important (movies, etc.. that you can re-download), so sorry I can't help you more

[madgabz]

Anyway, thanks for Your time, Raulfg3, I'm in the proces of evaluating my situation over at the zfs-illumos mailing list, those are deffo the guys in the know
If I come up with something interesting, I'll let you guys in on the secrets

[couper101]

One trick you could try is only importing one disk! Physically disconnect one of your disks and try to import the other. All going well, one will import the other will hang. Once the good disk is imported, attach the second disk to the system. With a bit of luck the second disk can be repaired by ZFS.

This can work for raidz that crash or hang at system at boot. It might work in your case. I think it works because the problem disk is now out of sync with the other disks. So its treated as suspect by the system.

[raulfg3]

I find this usefull info in a new thread, perhaps can help you: http://docs.oracle.com/cd/E19082-01/817 ... index.html