zpool status reports unrecoverable error

[sean70]

I'm hoping someone can give me some pointer in what to do next. I have 4 WD 3TB Red drives in raidz2. I did some tests over the network and was able to get a pretty good rate of 118-122MB/s. However, after some writings to the pool, issuing the status command returns the "One of more devices has experienced an unrecoverable error...". Issuing the scrub command resulted in recovering some data. So there must be really something really wrong with the drives. I used smartctl to do a long test took nearly 4 days to go thru all 4 drives. All tests pass without any errors whatsoever. So my question is why am I getting unrecoverable error but the tests tell me the drives are fine? Are the drives really bad? What other tools can I use to test the drives?

Thanks in advance.

[shakky4711]

Hello,

So my question is why am I getting unrecoverable error but the tests tell me the drives are fine

SMART is operating with many different indicators and can show the health condition.

Compared to human these values can show if the drive is fit like a man with 20years, 40 years or 60years.
As long as it is alive and no relevant value has exceeded a critical value the SMART test will report OK.

I would suggest to replace the SATA cables first, always use better quality cables with the metal latch, really good cables have an additional EMC screen.
Try to keep power cables out of reach from the data cables.
Then check if you get these errors again and if it is always the same drive.

Shakky

[sean70]

Thank you for the reply. I'll try to use different cables and see if the problem still persist. At the moment, I don't know which drive is giving me the problem. I was hoping smartctl would tell me after the long test but that wasn't the case.

[sean70]

Took a while but I ordered some better SATA cables and I'm still getting the same error. Running scrub recovers 5.25 MB out of 365 GB of data. I know the recovered data is small compared to the overall data but having the error still concerns me.

Are there any options to give more information as to what disk have the problem when issuing the "zpool scrub" command? It would be nice to know if scrub tells you that X amount of data was recovered from ada0 or something like that.

Are they any other tools out there that I can use to test the drive without worrying about destroying the data?

TIA

[Earendil]

I had some similar issues with a four 2 TB HDD of WD greens. At one point I lost 3 TB of data. I checked everything including cables. I found if I didn't touch the computer there were less incidents. It took nearly a year to determine the problem was with a Molex connector of a Y power cable that powered one of the four HDDs. Result was it rarely happened with short dropouts. Enough to mess up my pools all the time so I scrubed often. I only found it by being near the computer, my ear nearly in the case (several fans going so it is never silent), moving the power cable a certain way and hearing the HDD spindown a little before spinning up again.

Shakky reminds you to look at the SMART readings. If you learn how to read them you can get an idea if a HDD is going to fail soon. If you have data errors then there should be something obvious in those SMART readings. If there is nothing the issue is somewhere else. In my case my SMART readings were fine but you'd expect that with a power problem.

WD diagnostic tools don't destroy the HDD data, do they? If there are any sector errors then SMART should show that. If there are HDD controller errors then SMART should catch most of those. Sometimes some data is not recoverable so I would delete those corrupted files. I remember doing that once but it was so many files that I paused. The pool crashed soon after and that's when I lost my data.