Zpool degraded! Need help with recovery. Very important data.

[bluewaffle]

Hey guys,

I've used Nas4free for years and I am a very basic user. I set up a raidz1 for my local temple so that they can have a server share for all data. It's been working for three years however it failed yesterday. I panicked and didn't do any reasearch and found the bad harddrive (looking at which serial number wasn't active), replaced it and now I can't even log onto the webgui. I did use the same harddrive.

I really need help. If there is a way I can recover the data and then redo a Raidz2, that would be ideal. I'll even start doing regular back ups outside the server. Please reach me here and I'll forward you teamviewer or whatever is necessary to help fix this.

I appreciate any help!

Thank you in advance.

[Parkcomm]

post the output of:

Code: Select all

zpool status
zdb -C
zdb -C your_pool_name

[bluewaffle]

Zpool Status

$ zpool status
pool: ZFSPool1
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://illumos.org/msg/ZFS-8000-8A
scan: resilvered 558M in 2h32m with 5868400 errors on Sat Sep 26 16:16:48 2015
config:

NAME STATE READ WRITE CKSUM
ZFSPool1 DEGRADED 0 0 6.09M
raidz1-0 DEGRADED 0 0 13.2M
ada0 ONLINE 0 0 0
ada2 ONLINE 0 0 0
ada3 ONLINE 0 0 0
ada4 ONLINE 0 0 0
3199947566593843760 UNAVAIL 0 0 0 was /dev/ada5

errors: 5868400 data errors, use '-v' for a list

[bluewaffle]

$ zdb -C
ZFSPool1:
version: 28
name: 'ZFSPool1'
state: 0
txg: 9820277
pool_guid: 325254308014916420
hostname: ''
vdev_children: 1
vdev_tree:
type: 'root'
id: 0
guid: 325254308014916420
children[0]:
type: 'raidz'
id: 0
guid: 5938103690785723377
nparity: 1
metaslab_array: 30
metaslab_shift: 37
ashift: 12
asize: 15002940211200
is_log: 0
create_txg: 4
children[0]:
type: 'disk'
id: 0
guid: 929947763412115185
path: '/dev/ada0'
phys_path: '/dev/ada0'
whole_disk: 1
DTL: 137
create_txg: 4
children[1]:
type: 'disk'
id: 1
guid: 7338220102995057091
path: '/dev/ada2'
phys_path: '/dev/ada2'
whole_disk: 1
DTL: 104
create_txg: 4
children[2]:
type: 'disk'
id: 2
guid: 12281237443500408748
path: '/dev/ada3'
phys_path: '/dev/ada3'
whole_disk: 1
DTL: 133
create_txg: 4
children[3]:
type: 'disk'
id: 3
guid: 11321312116056338844
path: '/dev/ada4'
phys_path: '/dev/ada4'
whole_disk: 1
DTL: 33
create_txg: 4
children[4]:
type: 'disk'
id: 4
guid: 3199947566593843760
path: '/dev/ada5'
phys_path: '/dev/ada5'
whole_disk: 1
not_present: 1
DTL: 139
create_txg: 4

[Parkcomm]

OK - your pool is fine, you just have something preventing you from logging on to your web GUI

Just follow this http://wiki.nas4free.org/doku.php?id=faq:0149 - you don't even need the web GUI to get the pool resilvering and get access to your data.

What are symptoms stoping you from using the GUI? This web page not available? Incorrect password? etc

Also do you know how to use the console - if so,

1:/ You can se the ip address of your NAS
2:/ You can reset the password

[b0ssman]

have you got a full or an embedded install?

[crowi]

OK - your pool is fine,

???
-->One or more devices has experienced an error resulting in data
-->errors: 5868400 data errors, use '-v' for a list
-->3199947566593843760 UNAVAIL 0 0 0 was /dev/ada5

I would not say this pool is fine, it is questionable if it can be recovered completely.
I would immediately exchange the faulted drive.
Please post also the SMART values of the disks.

In general, perform frequent scrubs, keep an eye on the SMART values and backup your data, to prevent such things.

[Parkcomm]

You are right, big assumption on my part. Here is my thinking:

Pulled a faulty disk - put same disk back in. Pool resilvers and generates 6m checksum errors - however four of the five drives have 0 errors. So I put all of the errors down to the newly inserted faulty disk, If I'm right the pool is still operational.

Anyway - Bluewaffle definitely get that replacement disk in and we'll see if those errors are cleared.

It is not hard tof follow the procedure in the wiki even if you don't have the gui:

Code: Select all

zpool offline ZFSPool1 /dev/ada5

<shutdown the server and replaced the failed drive>

Code: Select all

zpool online ZFSPool1 /dev/ada5

[b0ssman]

and please run a memtest on the machine.

http://www.memtest.org/

[bluewaffle]

OK - your pool is fine, you just have something preventing you from logging on to your web GUI

Just follow this http://wiki.nas4free.org/doku.php?id=faq:0149 - you don't even need the web GUI to get the pool resilvering and get access to your data.

What are symptoms stoping you from using the GUI? This web page not available? Incorrect password? etc

Also do you know how to use the console - if so,

1:/ You can se the ip address of your NAS
2:/ You can reset the password

So I followed what that said and let it resilver yesterday. Today I checked and this is what I got.

[bluewaffle]

So this is the story. Again I am completely new to this. I've used my server for years because it simply just worked.

A few days ago I checked the status of the server and it said "degraded." So this has box has five hard drives in it, and one failed. I had three replacement hdd ordered in case this happened, so I had a replacement ready.

I shut the server down, then went to check the harddrives (I knew which one was bad by checking the serial numbers on the WebGUI). I looked to find out that the sata cable came lose, so I plugged it back in and then started the server.

I thought it would be as simple as that. So I went to check my network drive in on my windows pc and all the files were gone and only one folder was there "guest," and it said that the drive was only 4.5mb big. I panicked because I had 8TB of data on there.

one the server page it still says that the 8TB of data is still there, I just want to access it, back up the data, and redo the server so its a RaidZ2.

here is the screenshot of what "system" tab:

[bluewaffle]

I can let someone team viewer in to see if they can help me out. I appreciate all the advice in here, thank you again.

[bluewaffle]

have you got a full or an embedded install?

full install on the dedicated nas box on a specific ip address

[bluewaffle]

WOOT!!!!!!!! THank you sooo much. I was able to access my files. I'll backup what I needed and then redo the server.

Any suggestions this time. on how to do regular maintenance on the server? Def going to setup a Z2 raid this time, to get even more fail safe. But I keep reading about scrubbing your drives. I have no idea what that means and how to do it.

[b0ssman]

please post the smart values of all your drives.

you have permanent file system errors.

most likely another disk is about to die.

[bluewaffle]

do i just go into, "S.M.A.R.T" section and post that screen?

[b0ssman]

yes

[crowi]

Any suggestions this time. on how to do regular maintenance on the server?

Depending on the useage, do scrub once a month: Disks|ZFS|Pools|Tools --> Scrub
Enable SMART monitoring --> Disks|Management|Disk|Edit
And check the syslog from time to time.