Page 1 of 1
Can't import zfs-stripe, zpool import hangs indefinetely
Posted: 14 May 2013 10:19
by madgabz
Hi all,
As some of you know,
I've been having problems with my zfs-based stripe-array of 2 disks. Yes, I've been living life dangerously, but the data are not THAT important, mostly backups and movies/TV shows! But, it would be a pain to recreate.
So, after suddenly having the array/pool drop out on my old hardware setup (it was starting to do insane amounts of errors on scrubs, so i decided this most likely had to be a hardware fault!), i bought a new mobo+ram+cpu and looked forward to a brand new experience!
But, same things happen when i try to import the pool in my new hardware setup. The pool consists of 2 striped 1 TB disk, for a total of 2 TB storage.
When i do a 'zpool import' it correctly sees the pool as online, but needs to be imported w -f flag:
Code: Select all
nas4free:~# zpool import
pool: DataRaid1
id: 5339075205090693951
state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier and
the '-f' flag.
see: http://illumos.org/msg/ZFS-8000-EY
config:
DataRaid1 ONLINE
ada1 ONLINE
ada0 ONLINE
If I issue a 'zpool import -f DataRaid1' the system just hangs forever (left it overnight for a total 8 hours)! I can hear the disks working for app. 5 seconds. then silence!
If I import an nas4free-config from previously, it says under 'Disks:ZFS' that the 'ZFS system is in an unhealthy state' or something, later if i try to clear and re-import pools from web-frontend, it hangs forever again...
FWIW, I tried replacing all SATA-cables, as they've been known to regularly be culprits of many problems!
So, I google'd for this particular hang-problem, the only relevant (to my knowledge) thing that came up was this from the freenas.org forums:
zpool import hangs forever, help please!!!
So, after reading this, i tried to import using 'zpool import -fFX DataRaid1', but am getting the same result - it hangs indefinitely!
Now, I'm a fiddler and have absolutely no in-depth knowledge of ZFS, so I'm very reluctant to start fiddling too much with the pool
If any ZFS Zealot could help out here, I'd very much appreciate it!
Or, should I just be done with it, and restart the fileserver from scratch?
Thank You so much in advance!
Cheers, Gabriel S. Hansen
Re: Can't import zfs-stripe, zpool import hangs indefinetely
Posted: 14 May 2013 10:38
by madgabz
I might add:
gstat reveals no activity
and also doing this:
Code: Select all
[root@nas4free ~]# ps axl |grep 'zpool import'
0 9338 7078 0 22 0 16280 2108 piperd S+ 2 0:00.02 grep zpool import
0 9295 7081 0 21 0 35476 3352 tx->tx_s D+ 3 0:01.12 zpool import -f 5339075205090693951
what is going on?
I can't kill or terminate any zpool process, even if I'm root....?
Re: Can't import zfs-stripe, zpool import hangs indefinetely
Posted: 14 May 2013 11:10
by sleid
Hello!
Have you tried zpool import-f only
Re: Can't import zfs-stripe, zpool import hangs indefinetely
Posted: 14 May 2013 15:42
by madgabz
YEs, same thing
tried: -f, -fF, -F, -fFX, -X
Re: Can't import zfs-stripe, zpool import hangs indefinetely
Posted: 14 May 2013 22:38
by sleid
How do you use zpool import-f ?
from web gui, from Putty or from console ?
Re: Can't import zfs-stripe, zpool import hangs indefinetely
Posted: 15 May 2013 07:36
by madgabz
I've tried from console and from remote ssh, even using tmux because of all the stalling. killall nor top/htops kill/terminate can get rid of the stalled zpool processes.....
The webgui either stalls ('waiting for webserver....' - i'm on firefox here), or gives a weird php error message, regarding some 'name' variable... I'll try reproduce and post it, but its hardly relevant!
EDIT:

Forgot, Thanks, sleid

Re: Can't import zfs-stripe, zpool import hangs indefinetely
Posted: 17 May 2013 11:53
by madgabz
ok, no ZFS zealots out there to the rescue....
I'll take that as a start-over-thumbs-up!
Re: Can't import zfs-stripe, zpool import hangs indefinetely
Posted: 17 May 2013 13:31
by raulfg3
sorry, import a ZFS pool is normally an easy operation, only if you have problems, must need to use -f operator, but you case is really extrange and do not have experience on others simmilar.
Re: Can't import zfs-stripe, zpool import hangs indefinetely
Posted: 17 May 2013 13:36
by raulfg3
What can I do in your situation ( webare: you can do what you want).
I'll try to change SATA ports for others, ( perhaps you mix SATA controllers and one is not detected), revise SATA Cables and use SATA cables with metalic Clip to avoid false contact.
Do a new install of latest Nas4Free ( full or embeded ), once sucessfully booted, try one more time to import pool ( from shell, use monitor and keyboard atached to your NAS), probably , first time fail, try -f , if fail, normally ZFS say why fail, and post here the message.
PD: Post all you write in shell , and output, is easy to us to detect some write fault or so... (ok; probably you write correctly how to import your pool but is good to be sure)
Re: Can't import zfs-stripe, zpool import hangs indefinetely
Posted: 17 May 2013 17:39
by madgabz
First: Big Thank You, raulfg3!
I've already done all you said, even ran nas4free directly from a bootup cd, now running from new install on external USB3-HDD! WHen I get the time, I will wipe and reinstall to USB3-hdd, then post my findings!
Regarding cables, i already said i swapped out the cables 2 or 3 times, i'll check if i can get those cables w clips in the local tech-shop! You have a valid point in that the sata cables are rather flimsy
I also started out using the console via my kbd and monitor. So, that was actually where i began....
The strange thing here is, a simple 'zpool import' reveals the array nicely, but suggests I use the -f option to import, what the exact reason for having to do this is, is beyond me! I was hoping that there were some kind of way to get ZFS to more verbosely shell out some detailed info about what is really going on. So far, I've only come across a few ZFS-related posts somewhere suggesting to use 'procstat -kk -a', but I have no idea what that info is or how to interpret it!
fwiw,
Gabriel
Re: Can't import zfs-stripe, zpool import hangs indefinetely
Posted: 19 May 2013 00:27
by madgabz
OK, so I finally got around to download .698beta adn installed it freshly on my USB disk.
I rebooted, configured IP address, went to webgui, enabled SSH, and putty'ed in, here's what the output was:
Code: Select all
nas4free:~# bash
[root@nas4free ~]# zpool import
pool: DataRaid1
id: 5339075205090693951
state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier and
the '-f' flag.
see: http://illumos.org/msg/ZFS-8000-EY
config:
DataRaid1 ONLINE
ada0 ONLINE
ada1 ONLINE
[root@nas4free ~]# zpool import -f 5339075205090693951
...after app. 20 secs of disk activity from the implicated disks, nothing happens and zpool just hangs there
I cannot issue ANY other zpool commands, not from webgui and not from shell (i tried new putty session, as well as tmux'ing). I cannot kill any zpool process, not with kill -9 nor killall
Re: Can't import zfs-stripe, zpool import hangs indefinetely
Posted: 19 May 2013 04:28
by fsbruva
What about booting a Debian Live CD - and installing zfs-fuse. You might be able to import and backup the data. Or, you might be able to import and then export it in a way that Nas4Free can import.
It is obvious that Nas4Free (and FreeBSD) are not dealing well with the zfs format on those disks.
Re: Can't import zfs-stripe, zpool import hangs indefinetely
Posted: 19 May 2013 11:50
by madgabz
Thanks a lot, fsbruva, I already tried ZFSGuru distribution, but that is more or less the same as nas4free, ill try debian now... Will be fun, i guess

What options are actually available for import-function? - it seems, the 'X' option s totally undocumented.... just wondering if there were other?
Re: Can't import zfs-stripe, zpool import hangs indefinetely
Posted: 19 May 2013 12:56
by madgabz
OK, now sitting in Debian 7 'wheezy' KDE LiveCD - Exact same thing happens, after installing zfs on linux. doing a 'zpool import' straight up reveals the stripe to be online, but -f ID makes the disks active for app 20 secs, then hangs zpool entirely. Starting up a new terminal, i try zpool import again, and zpool itself is totally hung - I cant even do a zpool status
Oh well.... I guess, i should try some illumos dist now.... even if chances are slim!
Re: Can't import zfs-stripe, zpool import hangs indefinetely
Posted: 19 May 2013 15:32
by madgabz
oh well... sitting in OpenIndiana LiveCD 151a .... x86 though... same thing:
Code: Select all
root@openindiana:/jack# zpool import
pool: DataRaid1
id: 5339075205090693951
state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier and
the '-f' flag.
see: http://www.sun.com/msg/ZFS-8000-EY
config:
DataRaid1 ONLINE
c2t0d0p0 ONLINE
c5t0d0p0 ONLINE
root@openindiana:/jack# zpool import -f DataRaid1
Re: Can't import zfs-stripe, zpool import hangs indefinetely
Posted: 19 May 2013 17:25
by raulfg3
I have a question:
When you do a Zpool import ( without -f flag), and ZFS answer status: The pool was last accessed by another system
Your pool is mark as online,
Do you try to use it , and do not force import?.
Re: Can't import zfs-stripe, zpool import hangs indefinetely
Posted: 19 May 2013 17:46
by madgabz
No, I'm not trying anything! Only those two disks are attached, and i can't do anything with them, except start all over i guess.... All i'm trying is to import them! When you issue a 'zpool import', that is a regular normal import, obviously it won't work, so I try the -f (and also tried -fFX) forced approach, which just hangs the system!
Maybe I should try and post this to the zfs-support/developer mailing list, if there is any...?
Re: Can't import zfs-stripe, zpool import hangs indefinetely
Posted: 19 May 2013 21:57
by raulfg3
madgabz wrote:ll i'm trying is to import them! When you issue a 'zpool import', that is a regular normal import, obviously it won't work,
well it's not obvious for me, I see that your pool is online, this first time that you do import ( prior to force), so please try to acces to your data if possible.
PD: perhaps the problem can be the mount point, use modificator to define new mount point like describe here:
viewtopic.php?f=66&t=1182#p4346
Re: Can't import zfs-stripe, zpool import hangs indefinetely
Posted: 20 May 2013 10:15
by madgabz
Thanks, Raulfg3, that's the first thing I've heard making sense

I'll try to re-mount the pool, but - you have to admit - zpool stalling/hanging on a regular import, that IS a bit weird....?
I'm off to family stuff, will try later....
Re: Can't import zfs-stripe, zpool import hangs indefinetely
Posted: 20 May 2013 19:57
by madgabz
There's something rotten in the state of Denmark....
Trying to mount still yields a hanging zpool command :/
I tried 'zpool import -R /mnt -f DataRaid1' and it just hangs!
Seeing that this is still importing the pool, isn't it just necessary to mount the pool, or is mounting the same as import?
fwiw, i cannot use the pool in any way, as it is not present in the filesystem at all. How can I use the pool or even just salvage data off of it, without importing/mounting it?
Re: Can't import zfs-stripe, zpool import hangs indefinetely
Posted: 20 May 2013 20:35
by raulfg3
mounting it's not necessary, only import.
But you can't import, so sorry I think that your data is lost, and your only option is to reformat your disk, and recreate your pool ( or try to boot from your Old boot disk if still works).
In the first post says that your data is not so important (movies, etc.. that you can re-download), so sorry I can't help you more
Re: Can't import zfs-stripe, zpool import hangs indefinetely
Posted: 22 May 2013 10:20
by madgabz
Anyway, thanks for Your time, Raulfg3, I'm in the proces of evaluating my situation over at the zfs-illumos mailing list, those are deffo the guys in the know

If I come up with something interesting, I'll let you guys in on the secrets

Re: Can't import zfs-stripe, zpool import hangs indefinetely
Posted: 25 May 2013 14:56
by couper101
One trick you could try is only importing one disk! Physically disconnect one of your disks and try to import the other. All going well, one will import the other will hang. Once the good disk is imported, attach the second disk to the system. With a bit of luck the second disk can be repaired by ZFS.
This can work for raidz that crash or hang at system at boot. It might work in your case. I think it works because the problem disk is now out of sync with the other disks. So its treated as suspect by the system.
Re: Can't import zfs-stripe, zpool import hangs indefinetely
Posted: 27 May 2013 11:45
by raulfg3
I find this usefull info in a new thread, perhaps can help you:
http://docs.oracle.com/cd/E19082-01/817 ... index.html