2 drive of 4, down and don't know what to do.

[idmah]

Hi gang.
After being Yelled at in official FreeNas forum, I'm here..
Hello. Help please...

I was replacing a bad drive when another one started to give read and write errors.
Which is killing the "zpool replace" gets stuck at 25.7% for days.

I tried restarting it and same.
disconnected the bad drive ad8 and zpool replace continued but machine keeps crashing.
not sure if it's the machine or the drives or the both. not sure if an error has propagated
through to the other drives.

and worse I'm running 0.7 rc 1 I was going to update after fixing the drives.

I should be able to zpool replace with just two drives right?
If not what should I do ? the system is unstable with just 2 drives.
thanks
Ian

ps. I tried to copy files off the freenas 0.7 rc but it crashed. Is there a way to safely abort the zpool replace, so I can backup?
or is there a way to mount the 2 'working' drives into something else and recover the files?

[Onichan]

What RAIDZ are you running? It sounds like you only have RAIDZ1 which means if you loose two drives you're screwed. Now I don't know if its possible to recover some of the data, but there is a good chance you can't.

[idmah]

How do I find out ?
I'm new to all this stuff. Is there a way to copy/mirror the console output to a another screen? or maybe a file.
but the system is crashing, a lot !

thanks


ps. as I remember, the problem started when "the girlfriend" turned off her computer in the middle of a file copy, but I can't for the life of me remember
what file that was. is there any utility to track down the file? Could that be the problem? grasping at straws.

So tried to zpool scrub because well ... I'm desperate.

here's what's coming up..

Code: Select all

root: ZFS: vdev I/O failure, zpool=ZFS_Pool path=/dev/ad10 offset=172779273216 size=65536 error=5
Jan 14 00:06:56 	bfmb 	root: ZFS: vdev I/O failure, zpool=ZFS_Pool path=/dev/ad10 offset=172779207680 size=65536 error=5
Jan 14 00:07:15 	bfmb 	kernel: ad10: WARNING - SETFEATURES SET TRANSFER MODE taskqueue timeout - completing request directly
Jan 14 00:07:19 	bfmb 	kernel: ad10: WARNING - SETFEATURES SET TRANSFER MODE taskqueue timeout - completing request directly
Jan 14 00:07:23 	bfmb 	kernel: ad10: WARNING - SETFEATURES ENABLE RCACHE taskqueue timeout - completing request directly
Jan 14 00:07:27 	bfmb 	kernel: ad10: WARNING - SETFEATURES ENABLE WCACHE taskqueue timeout - completing request directly
Jan 14 00:07:31 	bfmb 	kernel: ad10: WARNING - SET_MULTI taskqueue timeout - completing request directly
Jan 14 00:07:31 	bfmb 	kernel: ad10: TIMEOUT - READ_DMA48 retrying (1 retry left) LBA=337459900
Jan 14 00:07:50 	bfmb 	kernel: ad10: WARNING - SETFEATURES SET TRANSFER MODE taskqueue timeout - completing request directly
Jan 14 00:07:54 	bfmb 	kernel: ad10: WARNING - SETFEATURES SET TRANSFER MODE taskqueue timeout - completing request directly
Jan 14 00:07:58 	bfmb 	kernel: ad10: WARNING - SETFEATURES ENABLE RCACHE taskqueue timeout - completing request directly
Jan 14 00:08:02 	bfmb 	kernel: ad10: WARNING - SETFEATURES ENABLE WCACHE taskqueue timeout - completing request directly
Jan 14 00:08:06 	bfmb 	kernel: ad10: WARNING - SET_MULTI taskqueue timeout - completing request directly
Jan 14 00:08:06 	bfmb 	kernel: ad10: TIMEOUT - READ_DMA48 retrying (1 retry left) LBA=337460029
Jan 14 00:08:25 	bfmb 	kernel: ad10: WARNING - SETFEATURES SET TRANSFER MODE taskqueue timeout - completing request directly
Jan 14 00:08:29 	bfmb 	kernel: ad10: WARNING - SETFEATURES SET TRANSFER MODE taskqueue timeout - completing request directly
Jan 14 00:08:33 	bfmb 	kernel: ad10: WARNING - SETFEATURES ENABLE RCACHE taskqueue timeout - completing request directly
Jan 14 00:08:37 	bfmb 	kernel: ad10: WARNING - SETFEATURES ENABLE WCACHE taskqueue timeout - completing request directly
Jan 14 00:08:41 	bfmb 	kernel: ad10: WARNING - SET_MULTI taskqueue timeout - completing request directly
Jan 14 00:08:41 	bfmb 	kernel: ad10: TIMEOUT - READ_DMA48 retrying (0 retries left) LBA=337459900
Jan 14 00:09:00 	bfmb 	kernel: ad10: WARNING - SETFEATURES SET TRANSFER MODE taskqueue timeout - completing request directly
Jan 14 00:09:04 	bfmb 	kernel: ad10: WARNING - SETFEATURES SET TRANSFER MODE taskqueue timeout - completing request directly
Jan 14 00:09:08 	bfmb 	kernel: ad10: WARNING - SETFEATURES ENABLE RCACHE taskqueue timeout - completing request directly
Jan 14 00:09:12 	bfmb 	kernel: ad10: WARNING - SETFEATURES ENABLE WCACHE taskqueue timeout - completing request directly
Jan 14 00:09:16 	bfmb 	kernel: ad10: WARNING - SET_MULTI taskqueue timeout - completing request directly
Jan 14 00:09:16 	bfmb 	kernel: ad10: TIMEOUT - READ_DMA48 retrying (0 retries left) LBA=337460029
Jan 14 00:09:35 	bfmb 	kernel: ad10: WARNING - SETFEATURES SET TRANSFER MODE taskqueue timeout - completing request directly

What should I do? Looks like two drives might be working, should I disconnect the ad10 drive. or is it hardware?

[Onichan]

The log sounds like ad10 is having communication problems like it is dieing. I am not familiar with the old version so I don't know where to find the RAID level in it, but I doubt this would be caused by the client suddenly being cut off in the middle of a transfer. That should only corrupt that one file and I am not sure how to find which file. You probably want to ask somebody who is more familiar with something like this to see if they can help you. You could try the IRC and see if anybody is on that could help.

[idmah]

I tried to zpool scrub ZFS_Pool and was getting the above errors for 4 days. Now I think the machine has crashed.
Is there a way to safely restart the system? or will I mess up the other drives too, if I do?

Next thing I was thinking of trying was use ad10 remove from pool, and the zpool replace ad8 drive. Hopefully giving me 3 working drives.
and then replace the ad10, with a working drive? does this seem a reasonable way to go?
or am I up poo creek without a paddle?

[raulfg3]

be patient, DO NOT BOOT from your actual boot disk, in fact use CD, boot from CD and first try to fsck your boot disk, and post result of

Code: Select all

dmesg -a

to see what happend.

Post all info posible about your system, is very , very importand to identify what kind of raid do you use, can be hardware raid, software raid ( if soft, can be raid5 or raidz1).

if is a raidz1, post result of

Code: Select all

zpool import 

or

Code: Select all

zpool -f import

, I need to know too

Code: Select all

zpool history 

to see what disk are members of your pool.

once we have all the info, perhaps can help you.

in the middle, you can read wiki about fsck and/ or raid crashed systems:
http://wiki.nas4free.org/doku.php?id=faq:0001
http://wiki.nas4free.org/doku.php?id=faq:0078
http://wiki.nas4free.org/doku.php?id=faq:0074


and for general info: http://wiki.nas4free.org/doku.php?id=faq:0144