hotspare in RAID-Z

[prior_philip]

Hi all,
Im fighting with my RAID-Z1 configuration. Either have a lack of understanding or something is going wrong.
My configration is: 4 x 2TB HDDs WD20EURS

Code: Select all

NAME           STATE     READ WRITE CKSUM
	RAIDZ          ONLINE       0     0     0
	  raidz1-0     ONLINE       0     0     0
	    gpt/Disk0  ONLINE       0     0     0
	    gpt/Disk1  ONLINE       0     0     0
	    gpt/Disk2  ONLINE       0     0     0
	spares
	  gpt/Disk3    AVAIL

errors: No known data errors

I would expect, pulling one productive Disk[0-2] forces Disk3 to jump in.
But nothing happens. BTW; Autoreplace is ON

Code: Select all

pool: RAIDZ
 state: DEGRADED
status: One or more devices has been removed by the administrator.
	Sufficient replicas exist for the pool to continue functioning in a
	degraded state.
action: Online the device using 'zpool online' or replace the device with
	'zpool replace'.
  scan: none requested
config:

	NAME                     STATE     READ WRITE CKSUM
	RAIDZ                    DEGRADED     0     0     0
	  raidz1-0               DEGRADED     0     0     0
	    gpt/Disk0            ONLINE       0     0     0
	    2609822771131604473  REMOVED      0     0     0  was /dev/gpt/Disk1
	    gpt/Disk2            ONLINE       0     0     0
	spares
	  gpt/Disk3              AVAIL

errors: No known data errors

Configuration summary

Code: Select all

newton:~# zpool get all RAIDZ
NAME   PROPERTY       VALUE       SOURCE
RAIDZ  size           5.44T       -
RAIDZ  capacity       0%          -
RAIDZ  altroot        -           default
RAIDZ  health         DEGRADED    -
RAIDZ  guid           11049731454688481474  default
RAIDZ  version        28          default
RAIDZ  bootfs         -           default
RAIDZ  delegation     on          default
RAIDZ  autoreplace    on          local
RAIDZ  cachefile      -           default
RAIDZ  failmode       wait        default
RAIDZ  listsnapshots  off         default
RAIDZ  autoexpand     off         default
RAIDZ  dedupditto     0           default
RAIDZ  dedupratio     1.83x       -
RAIDZ  free           5.42T       -
RAIDZ  allocated      22.9G       -
RAIDZ  readonly       off         -
RAIDZ  comment        -           default
RAIDZ  expandsize     0           -

Whats wrong with it? Whats wrong with me?

[sleid]

Hello,
Whats wrong with me? : One or more devices has been removed by the administrator.

Whats wrong with it?
-The motherboard or SATA controller must be enabled hotplug capabilities.

- autoreplace must be activated before degradation.

- The disc must be removed on (hotplug)

[prior_philip]

ok, sounds like its not hot plug enabled. ZFS takes care about such stuff?
What happens if the disk goes offline due to hw issues, isnt it the same and ZFS doesnt replace the disk?

[sleid]

Try disconnecting a SATA cable, nas operating

[prior_philip]

Yes Sir, asap

[Nastradamus]

I have see the same issue here.

Set the autoreplace : zpool set autoreplace=on tank
add a spare device on the pool, statu was AVAIL.

Exctract a device of the storage chassis. Nothing append, zpool still in DEGRADED, spare on AVAIL. No scrub, no resilvering.

I stop my inverstigation after read this thread : http://forums.freebsd.org/showthread.php?t=35343

look at #3, #8, #10

http://freebsd.1045724.n5.nabble.com/ZF ... 55449.html

look at #2, #4

and here : http://lists.freebsd.org/pipermail/free ... 13351.html

It seems to be current according to our test.

[prior_philip]

Hi Nastradamus,
thats exactly the point. FreeBSD does not support hotplug features, a bit confusing since autoreplace suggest it does.
Thanks for your tip.
Currently Im using a script that checks for pool health and sends out an alert in case of issues. Fine for me now but not a permanent solution.
Fritz wrote some cool scripts that does a good job.
here: viewtopic.php?f=70&t=2197

[dnar]

Why don't you use a Z2 pool? Then your extra drive is already silvered.

[Lee Sharp]

I agree that a Z2 pool is better than a Z1 and hot spare. But I also agree that alerting is handy in any case.

[Nastradamus]

@prior_philip : you're welcome.
I have the impression that an agent for manage the fault (like FMA in Solaris) or something like that was not implemented in FreeBSD.

It seems to me that autoreplace option haven't direct link with hotspare. I think spare is always hotspare, this disk must replace the faulted one even if autoreplace set to off. For me autorepace is used when you physically change a drive (faulted or not) and you don't need to use the zpool replace command, it's automatic, if at this time, spare disk resilvering, it's must to be canceled, return to spare state and the new drive, physically replace, must resilvered.

@dnar, @lee sharp : We have some reasons for need hotspare :

1. Like You point, in Raidz2 configuration, the device is use contrary to hotspare, is not active, then the device is "like new", if it break in little time when the pool is degraded you realy have no luck. It's not specially better than Raidz2 it's a different approach.

2. If you have a zpool with a vdev of 3 drive in Raidz1 and one hotspare and you must change for a zpool with a vdev of 4 device (even if it's not realy convenient, 2 striped mirror must be better in almost case, especially if hotspare work) you need 4 more device (one vdev) for imporve your pool capacity instead of 3 before.

3. If you target high IOPS you have probably a mirror striped zpool. Hotspare is very effective in this case, you don't wan't to add one more device on each mirror for redundancy even if you have 10 vdev ^^

4. You can share the hotspare device between sereval zpool.

5. You can add it on a old zpool, full of data, use on production, that you don't want totaly backup for recreate a new zpool from scratch.

And maybe some other reasons.