After HD replacement old HD still in list

[gecko1a]

Hi,

I replaced a HD (offline, replace) and after resilver the old one ist still in place and the pool is degraded.

Code: Select all

ool information and status
  pool: Pool
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
	corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
	entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: resilvered 1.20T in 5h51m with 1 errors on Thu Oct 10 12:04:57 2013
config:

	NAME                       STATE     READ WRITE CKSUM
	Pool                       DEGRADED     0     0     1
	  raidz1-0                 ONLINE       0     0     0
	    ada3.nop               ONLINE       0     0     0
	    ada2.nop               ONLINE       0     0     0
	    ada0.nop               ONLINE       0     0     0
	    ada1.nop               ONLINE       0     0     0
	  raidz1-1                 ONLINE       0     0     0
	    ada4                   ONLINE       0     0     0
	    ada5                   ONLINE       0     0     0
	    ada6                   ONLINE       0     0     0
	    ada7                   ONLINE       0     0     0
	  raidz1-2                 DEGRADED     0     0     2
	    replacing-0            DEGRADED     0     0  213K
	      4149419797668455597  OFFLINE      0     0     0  was /dev/ada8/old
	      ada8                 ONLINE       0     0     0
	    ada9                   ONLINE       0     0     0
	    ada10                  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

any idea?

Regards

Frank

[kenZ71]

Sometimes I see the old drive but after a resilver the pool is always "online" and healthy.

I would double check that the new drive is connected with power & data cables, then re-run the resilver.

Does SMART show anything funky for any of the drives ??

[gecko1a]

The second resilver passed and no change. The old ada8 is still in the list and the pool degraded.
I scrub is running. Maybe it would helps

The problem is, I cann't do anything with ada8. Answer: "cannot xxxxxxxx ada8: no valid replicas"

ANy things I can do?

[raulfg3]

try to use new button "clear config & import Disk" to delete non existing ada8, and next go to Disks|ZFS|Configuration| and post your current and detected to see it.

[gecko1a]

Thanks raulfg3

"clear & import disk": I did it repeated.

see the ZSF Config and Disk Management in the attachment

[gecko1a]

Config of the vdev with ada8:

[raulfg3]

sorry , not enought knowledge to be sure, but suspect that you need to inform ZFS that replace disk:
http://wiki.nas4free.org/doku.php?id=faq:0149
http://docs.oracle.com/cd/E19253-01/819 ... index.html
but as I say not totally sure, wait other ZFS guru for instructions prior to do something potentially dangerous.

something like:

Code: Select all

zpool replace Pool 4149419797668455597 replacing-0

must work ( not totally sure because

Code: Select all

zpool replace Pool 4149419797668455597 ada8

is the other option)

[gecko1a]

There is a Backup of all important data

First answer:

Code: Select all

geckonas: /dev # zpool replace Pool 4149419797668455597 replacing-0
cannot open 'replacing-0': no such GEOM provider
must be a full path or shorthand device name

Second:

Code: Select all

geckonas: /dev # zpool replace Pool 4149419797668455597 ada8
invalid vdev specification
use '-f' to override the following errors:
/dev/ada8 is part of active pool 'Pool'

Maybe the problem could be, that replacing-0 is part of the vdev, and ada8 is only part of the pool. Something went wrong by replacing ada8

[gecko1a]

Next try:

Code: Select all

geckonas: /dev # zpool detach Pool replacing-0
cannot detach replacing-0: no such device in pool
geckonas: /dev # zpool detach Pool 4149419797668455597
cannot detach 4149419797668455597: no valid replicas
geckonas: /dev # zpool detach Pool replacing-0
cannot detach replacing-0: no such device in pool
geckonas: /dev # zpool detach Pool replacing
cannot detach replacing: no such device in pool
geckonas: /dev # zpool detach Pool ada8
cannot detach ada8: no valid replicas

[gecko1a]

Next:

geckonas: /dev # zpool online Pool 4149419797668455597
warning: device '4149419797668455597' onlined, but remains in faulted state
use 'zpool replace' to replace devices that are no longer present
geckonas: /dev #

Result:

Code: Select all

        NAME                       STATE     READ WRITE CKSUM
        Pool                       DEGRADED     0     0     2
          raidz1-0                 ONLINE       0     0     0
            ada3.nop               ONLINE       0     0     0
            ada2.nop               ONLINE       0     0     0
            ada0.nop               ONLINE       0     0     0
            ada1.nop               ONLINE       0     0     0
          raidz1-1                 ONLINE       0     0     0
            ada4                   ONLINE       0     0     0
            ada5                   ONLINE       0     0     0
            ada6                   ONLINE       0     0     0
            ada7                   ONLINE       0     0     0
          raidz1-2                 DEGRADED     0     0     4
            replacing-0            DEGRADED     0     0     0
              4149419797668455597  UNAVAIL      0     0     0  was /dev/ada8/old
              ada8                 ONLINE       0     0     0  (resilvering)
            ada9                   ONLINE       0     0     0
            ada10                  ONLINE       0     0     0  (resilvering)

I think, I will document my way, if we can solve the issue someone else can use it.
I know, I do it with high risk, that a can lose my pool.

[raulfg3]

try zpool -f replace Pool 4149419797668455597 ada8

PD: not sure if -f is in good place, can be zpool replace -f Pool 4149419797668455597 ada8

[gecko1a]

Code: Select all

geckonas: /dev # zpool replace Pool 4149419797668455597 ada8
invalid vdev specification
use '-f' to override the following errors:
/dev/ada8 is part of active pool 'Pool'

I will try -f as last option before a set up a new pool and a new poolwill take about 2 week

If the resilvering finished I try to remove 4149....

[kenZ71]

Have you looked at the documentation for replacing a disk?
http://docs.oracle.com/cd/E19253-01/819 ... index.html

You replaced ada8 with ada10 ? Has ada8 been removed from the box ? If yes, this command should work:
zpool replace Pool ada8

Have you rebooted since the new drive was added ?

[gecko1a]

Have you looked at the documentation for replacing a disk?
http://docs.oracle.com/cd/E19253-01/819 ... index.html

Yes

You replaced ada8 with ada10 ? Has ada8 been removed from the box ? If yes, this command should work:
zpool replace Pool ada8

No, I replaced ada8. During resilver ada8 went removed. Reboot without changing anything ada8 was in place and the result was the status above
After every reboot it restarts resilver ada8 and ada10

Have you rebooted since the new drive was added ?

More when one times .-)

[gecko1a]

Nothing changed after resilver.

I start my backup plan:
Out the moment I copy all data to a second NAS (about 10TB) and than I will rebuild the NAS4Free.

gecko

[gecko1a]

I couldn't resolve ist, so I did a new set up of the NAS
Now it is fine.