Problem while replacing a drive (mirror)

[bunk3m]

Hi!

In an earlier post you guys helped me to discover that one of the drives in my pool was a zombie. So I bought a replacement drive, shutdown the nas, replaced the drive and then restarted. I thought it would be as easy as

Code: Select all

zpool replace Pool da1

except that it didn't work like last time.

I'm running stock 9.1.0.1 - Sandstorm (revision 775)

I got

Code: Select all

zpool status Pool1
  pool: Pool1
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
	the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: resilvered 1.33T in 3h37m with 0 errors on Sun Mar 30 14:02:33 2014
config:

	NAME                     STATE     READ WRITE CKSUM
	Pool1                    DEGRADED     0     0     0
	  mirror-0               DEGRADED     0     0     0
	    7198073037835413877  UNAVAIL      0     0     0  was /dev/da1.nop
	    da1.nop              ONLINE       0     0     0

The da1.nop used to be da2.nop. ??Huh??

I thought I'd make sure autoreplace was on. It was. But nothing happened. I was worried that if replace 7198073037835413877 with da1 I would end up with 2 empty drives.

I think I may have screwed up everything now.

I thought I could detach the 7198073037835413877 and then reattach and get it to mirror. This looks like it was a mistake.

Now I have only 1 drive. While the GUI shows mirror, it isn't.

Code: Select all

zpool status
  pool: Pool1
 state: ONLINE
  scan: resilvered 1.33T in 3h37m with 0 errors on Sun Mar 30 14:02:33 2014
config:

	NAME        STATE     READ WRITE CKSUM
	Pool1       ONLINE       0     0     0
	  da1.nop   ONLINE       0     0     0

errors: No known data errors

The Disks|ZFS|Pools|Virtual device|Edit still shows the Virtual Device as mirrored but only the one drive /dev/da1

I've tried to attach, add but it wouldn't work.

Then I deleted the Virtual Device and tried to readd the new disk. Also no luck. But it did work as a stripe. Then I tried to format the da2 drive for zfs and this appeared to work.

I then deleted the Virtual Device stripe and add both drives to Mirror. But got errors and nothing added.

Now I'm stuck and confused. I can't decide what to do next but I'm sure it shouldn't be too difficult assuming I haven't screwed it all up too much.

At the moment I have
Disks|Management

Code: Select all

da1 	2861589MB 	ATA ST3000VN000  	ST3000VN000  	Ser#1  	Always on  	ZFS storage pool device  	ONLINE  	Edit disk  Delete disk
da2 	2861589MB 	ATA WDC WD30EFRX  	WDC WD30EFRX  	Ser#2  	Always on  	Unknown or unformatted  	ONLINE 

Pool1 still exists as a mirror with only 1 drive in Device1 (vdev).
Device1 (virtual device) shows mirror with only 1 drive - da1.

How do I get the new WDC drive into the zpool with mirroring while avoiding to screw up all the data in da1 (ST3000VN000)?

Thanks.

[b0ssman]

Then I deleted the Virtual Device and tried to readd the new disk. Also no luck. But it did work as a stripe. Then I tried to format the da2 drive for zfs and this appeared to work.

I then deleted the Virtual Device stripe and add both drives to Mirror. But got errors and nothing added.

BE VERY CAREFULL ABOUT JUST DELETING A VDEV.

what you prob did is attach the new drive to a sata port that is behind the first one.

so your old da2 became da1 and the new one became da2.

by detaching the drive you turning your mirror into a single drive.

to convert a single drive to a mirror see
http://docs.oracle.com/cd/E19253-01/819 ... index.html

the command is prob
zpool attach Pool1 da1.nop da2

[bunk3m]

Then I deleted the Virtual Device and tried to readd the new disk. Also no luck. But it did work as a stripe. Then I tried to format the da2 drive for zfs and this appeared to work.

I then deleted the Virtual Device stripe and add both drives to Mirror. But got errors and nothing added.

BE VERY CAREFULL ABOUT JUST DELETING A VDEV.

what you prob did is attach the new drive to a sata port that is behind the first one.

so your old da2 became da1 and the new one became da2.

I hear you. I was very careful to pull out the zombie drive and as I only pulled out that disk I didn't change any cables. So this is a strange change. Is there any other reason the disk id may have changed? I'd appreciate knowing for the next time.

by detaching the drive you turning your mirror into a single drive.

to convert a single drive to a mirror see
http://docs.oracle.com/cd/E19253-01/819 ... index.html

the command is prob
zpool attach Pool1 da1.nop da2

This command worked. Not sure why it didn't work yesterday but maybe it needed some time to settle down. Yesterday I got an error about not finding the disk.

More strange stuff. The GUI still shows the new disk as "Unknown or unformatted".

However the command

Code: Select all

zpool attach Pool1 da1.nop da2

is off an running.

zpool status now shows:

Code: Select all

zpool status
  pool: Pool1
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Tue Aug  5 20:25:40 2014
        57.7M scanned out of 1.60T at 4.12M/s, 112h45m to go
        57.3M resilvered, 0.00% done
config:

	NAME         STATE     READ WRITE CKSUM
	Pool1        ONLINE       0     0     0
	  mirror-0   ONLINE       0     0     0
	    da1.nop  ONLINE       0     0     0
	    da2      ONLINE       0     0     0  (resilvering)

errors: No known data errors

Thanks so much for your help. I was really worried I screwed up everything and would need to start again from the backup.

[b0ssman]

syncronize the zfs array in the gui. then the disk should show up corrrectly

[bunk3m]

Thank you for the tip. I didn't know it did that.

Now the GUI shows the info. zpool status shows

Code: Select all

pool status
  pool: Pool1
 state: ONLINE
  scan: resilvered 1.60T in 4h33m with 0 errors on Wed Aug  6 00:58:44 2014
config:

	NAME         STATE     READ WRITE CKSUM
	Pool1        ONLINE       0     0     0
	  mirror-0   ONLINE       0     0     0
	    da1.nop  ONLINE       0     0     0
	    da2      ONLINE       0     0     0

errors: No known data errors

I'm curious where this ".nop" came from on da1 and why it isn't on da2?

Googling I've read that this nop is some sort of read/write performance improvement but I don't remember ever enabling this.

Is this da1.nop a setting that I enabled somewhere? If this is good to have, how would I get da2 to also be da2.nop?

This experience has at times been terrifying but your help got me through! Thanks again for your help.

[crowi]

I'm curious where this ".nop" came from on da1 and why it isn't on da2?

the .nop appear when you format in advanced (4k) format. Then a kind of "virtual" stripesize is created on the nop device.

You can get rid of the .nop mark by:

Code: Select all

zpool export Pool1
gnop destroy da1.nop
zpool import Pool1

The 4k alignment will stay, just the .nop is gone.

More interesting is, did you do a 4k alignment on the other drive, too?

[bunk3m]

I'm curious where this ".nop" came from on da1 and why it isn't on da2?

the .nop appear when you format in advanced (4k) format. Then a kind of "virtual" stripesize is created on the nop device.

You can get rid of the .nop mark by:

Code: Select all

zpool export Pool1
gnop destroy da1.nop
zpool import Pool1

The 4k alignment will stay, just the .nop is gone.

More interesting is, did you do a 4k alignment on the other drive, too?

Thank you for your quick reply. I don't remember formatting the da2 drive. It was a replacement for a failed drive in a mirror array. I did have some problems getting it work. I added it to the mirror and it resilvered. If I have forgotten formatting the drive, I am pretty sure I would have enabled the 4kb sector format.

When I check Disks|ZFS|Pools|Virtual device|Edit I see that the check box for 4kb sectors is enabled for the vdev. I don't know if that is the same as formatting the drive with 4k alignment?

[crowi]

When I check Disks|ZFS|Pools|Virtual device|Edit I see that the check box for 4kb sectors is enabled for the vdev. I don't know if that is the same as formatting the drive with 4k alignment?

you can check it in two ways
a) use zdb

Code: Select all

zdb |grep ashift

there the output should be "12" for 4k alignment

b) use

Code: Select all

diskinfo -v da1

and

Code: Select all

diskinfo -v da2

Here the stripesize should show 4096 if you have 4k drives and enabled advanced format

stripes.png

[bunk3m]

Thanks again for your help.

Here is the output of those commands. Looks like everything is 4kb sector. So it I presume I should have a da2.nop instead of da2?? How do I fix it?

Code: Select all

zdb |grep ashift
            ashift: 12

Code: Select all

diskinfo -v da1
da1
	512         	# sectorsize
	3000592982016	# mediasize in bytes (2.7T)
	5860533168  	# mediasize in sectors
	4096        	# stripesize
	0           	# stripeoffset
	364801      	# Cylinders according to firmware.
	255         	# Heads according to firmware.
	63          	# Sectors according to firmware.
	            Z300Q5XG	# Disk ident.

Code: Select all

diskinfo -v da2
da2
	512         	# sectorsize
	3000592982016	# mediasize in bytes (2.7T)
	5860533168  	# mediasize in sectors
	4096        	# stripesize
	0           	# stripeoffset
	364801      	# Cylinders according to firmware.
	255         	# Heads according to firmware.
	63          	# Sectors according to firmware.
	     WD-WCC4N0868670	# Disk ident.

[crowi]

So it I presume I should have a da2.nop instead of da2?? How do I fix it?

Nothing to fix, everything is fine.