[SOLVED] ZFS resilvering problem

prosiakus · Post by **prosiakus** » 25 Aug 2017 18:05

I have problem with one drive, after replace it was frezzing NAS. Pool is is 8x3TB HDD.
So I have changed to other drive without waiting to resilver ealier drive that was frezzing NAS.
Than I have made zpool replace p1 ada6 and it start resilvering and end after 30GB resilver in 20 min..
I cant do enything naw. I tryed replace again, "zpool detach p1 18446744071900754353" (was unavailable but did offline and it works)
What should I do to replace this ada6 to make online state? I did all command from WEBGUI. 10.3.0.3
AND this drive was used in other zfs pool.

pool.png

Code: Select all

                               capacity     operations    bandwidth
pool                        alloc   free   read  write   read  write
--------------------------  -----  -----  -----  -----  -----  -----
P1                          17.1T  4.66T     60      8  6.54M  57.0K
  raidz2                    17.1T  4.66T     60      8  6.54M  57.0K
    ada0                        -      -     16      1  1.14M  14.7K
    ada1                        -      -     16      1  1.14M  14.7K
    ada7                        -      -     47      1  1.12M  14.7K
    ada4                        -      -     50      1  1.12M  14.7K
    ada5                        -      -     47      1  1.12M  14.8K
    ada2                        -      -     50      1  1.12M  14.8K
    replacing                   -      -      2     54  32.5K  1.11M
      ada6                      -      -      2     41  33.0K  1.10M
      18446744071900754353      -      -      0      0      0      0
    ada3                        -      -     50      1  1.12M  14.7K
--------------------------  -----  -----  -----  -----  -----  -----

sleid · Post by **sleid** » 26 Aug 2017 10:53

The initial order should have been

zpool replace P1 18446744071900754353 /dev/ada6

Can not perform this operation from the replacement menu.

prosiakus · Post by **prosiakus** » 26 Aug 2017 11:11

I have tried like this too :/ Naw I have done "scrub a pool"

scrub.png

sleid · Post by **sleid** » 26 Aug 2017 18:56

Replace rebuilds the pool automatically there is no need to do a scrub.
It was necessary to wait until the end of the resilvering before launching the scrub which repairs ada3, currently your raidZ2 has no more redundancy if another disk dies.
Do not touch anything and cross your fingers.

prosiakus · Post by **prosiakus** » 27 Aug 2017 13:38

Problem was that resilvering stuck after 30gb and do nothing. As u can see on first attached screen there is status that resilvered 31 GB. Problem is that I have to change drive again after first change when first resilvering didn't finish job. Nas was freezing. So I changed to another new drive. And after that resilvering stuck after 31gb. So I start scrub.

Post by **b0ssman** » 27 Aug 2017 13:56

ist could be that more drives are failing. please post the smart values of all drives.

prosiakus · Post by **prosiakus** » 27 Aug 2017 14:07

SMART is ok but nobody run scrub at this machine. Sometimes in log there was information about bad sectors. At this moment nas is freezing sometimes (sometimes 8h sometimes 15 min after restart). I dont have to much experience with nas. Is there a chance that when repairing will finish NAS will run normaly?

Post by **b0ssman** » 27 Aug 2017 14:25

please post the smart values anyway.

prosiakus · Post by **prosiakus** » 28 Aug 2017 06:44

There are some errors becase of that an old motherboard had some problem with SATA controler. After hardware replace smart error doesnt incrise.
For exaple ada4 is a new drive that was connected to old MB and couter rise to Command_Timeout 19324 but naw its not a problem. On all new hardware its not changing. All error couter stoped after new mother board.

smart.php

- saved as php because cant attach txt file and i think its not good to write all8 hdd smart as post.

Post by **b0ssman** » 28 Aug 2017 14:59

Device /dev/ada3
187 Reported_Uncorrect 0x0032 095 095 000 Old_age Always - 5
any value about 0 means the drive will fail soon see:
https://www.backblaze.com/blog/hard-drive-smart-stats/
Also the drive internally experiences errors mutlible times.
Error 5 occurred at disk power-on lifetime: 52441 hours (2185 days + 1 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 e0 ff ff ff 4f 00 00:02:38.704 READ FPDMA QUEUED
2f 00 01 10 00 00 00 00 00:02:38.609 READ LOG EXT
60 00 e0 ff ff ff 4f 00 00:02:35.964 READ FPDMA QUEUED
2f 00 01 10 00 00 00 00 00:02:35.861 READ LOG EXT
60 00 e0 ff ff ff 4f 00 00:02:33.216 READ FPDMA QUEUED

Error 4 occurred at disk power-on lifetime: 52441 hours (2185 days + 1 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 e0 ff ff ff 4f 00 00:02:35.964 READ FPDMA QUEUED
2f 00 01 10 00 00 00 00 00:02:35.861 READ LOG EXT
60 00 e0 ff ff ff 4f 00 00:02:33.216 READ FPDMA QUEUED
2f 00 01 10 00 00 00 00 00:02:33.080 READ LOG EXT
60 00 e0 ff ff ff 4f 00 00:02:30.435 READ FPDMA QUEUED

Error 3 occurred at disk power-on lifetime: 52441 hours (2185 days + 1 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 e0 ff ff ff 4f 00 00:02:33.216 READ FPDMA QUEUED
2f 00 01 10 00 00 00 00 00:02:33.080 READ LOG EXT
60 00 e0 ff ff ff 4f 00 00:02:30.435 READ FPDMA QUEUED
2f 00 01 10 00 00 00 00 00:02:30.318 READ LOG EXT
60 00 e0 ff ff ff 4f 00 00:02:27.620 READ FPDMA QUEUED

Error 2 occurred at disk power-on lifetime: 52441 hours (2185 days + 1 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 e0 ff ff ff 4f 00 00:02:30.435 READ FPDMA QUEUED
2f 00 01 10 00 00 00 00 00:02:30.318 READ LOG EXT
60 00 e0 ff ff ff 4f 00 00:02:27.620 READ FPDMA QUEUED
60 00 e0 20 02 00 40 00 00:02:27.618 READ FPDMA QUEUED
60 00 e0 20 00 00 40 00 00:02:27.588 READ FPDMA QUEUED

Error 1 occurred at disk power-on lifetime: 52441 hours (2185 days + 1 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 e0 ff ff ff 4f 00 00:02:27.620 READ FPDMA QUEUED
60 00 e0 20 02 00 40 00 00:02:27.618 READ FPDMA QUEUED
60 00 e0 20 00 00 40 00 00:02:27.588 READ FPDMA QUEUED
61 00 10 ff ff ff 4f 00 00:02:27.094 WRITE FPDMA QUEUED
61 00 10 ff ff ff 4f 00 00:02:26.873 WRITE FPDMA QUEUED

Device /dev/ada4
188 Command_Timeout 0x0032 100 001 000 Old_age Always - 19324

if any of those command timeouts occur during resilvering that could cause the problems you are seeing.

sleid · Post by **sleid** » 28 Aug 2017 19:50

ada3,ada4,ada5 : 199 UDMA_CRC_Error_Count

Multiple sata connection errors from either the cables, connectors, or controller (s) may cause timeout and other errors.
So this is a priority issue.

Post by **b0ssman** » 29 Aug 2017 07:23

he already said that he crc errors were old.

sleid · Post by **sleid** » 29 Aug 2017 07:51

Yes but it is always interesting to follow the evolution of this counter.

prosiakus · Post by **prosiakus** » 29 Aug 2017 14:32

OK all is fine scrub 100% and no freeze since yesterday (1 Day 5 Hours 48 Minutes 7 Seconds) that is mega record since last 3 weeks!!!

yupi!!!

But there is a problem with an old ada 6 in pool information:

afterscrub.jpg

Code: Select all

                               capacity     operations    bandwidth
pool                        alloc   free   read  write   read  write
--------------------------  -----  -----  -----  -----  -----  -----
P1                          17.2T  4.50T    892     28   101M   961K
  raidz2                    17.2T  4.50T    892     28   101M   961K
    ada0                        -      -    232      6  17.8M   189K
    ada1                        -      -    235      6  17.8M   189K
    ada7                        -      -    491     10  17.7M   187K
    ada4                        -      -    542     10  17.6M   187K
    ada5                        -      -    509     10  17.6M   187K
    ada2                        -      -    545     10  17.6M   187K
    replacing                   -      -    835     23  17.4M   322K
      ada6                      -      -    507     12  17.6M   320K
      18446744071900754353      -      -      0      0      0      0
    ada3                        -      -    531     10  17.6M   187K
--------------------------  -----  -----  -----  -----  -----  -----

Haw may i bring state: online?

This is replace window:

replace.jpg

And this is remove:

remove.jpg

I did "zpool remove p1 18446744071900754353" in webgui>tools>command and nothing had happend

I would like to set this pool to normal "online"

May I use in this situation "Overwrite already configured disks (only affects filesystem value)" in Disks > ZFS > Configuration > Synchronize? (i dont realy know what it will do)
I have tried replace and remove and nothing work. Ghost disk

Its there but cant remove or replace it.
Good for me that after scrub there is no freez enymore. Just need to know how to remove this crap "18446744071900754353"
mayby remove ada6 from manager, than "gpart destroy -F ada6" and add it in manager and than resilver will run?

Or other idea:
Mayby export P1. Remove zfs label from ada6, import P1 and replace ada6 will work after that, and old ada6 will dissapear?

sleid · Post by **sleid** » 30 Aug 2017 07:27

zpool clear P1

zpool clear P1 dev/ada6

zpool clear P1 18446744071900754353

If no success, the cleanest solution is the replace that has certainly done poorly previously.

zpool replace P1 18446744071900754353 /dev/ada6

prosiakus · Post by **prosiakus** » 30 Aug 2017 07:50

I have just done what u write. In console and by webgui and nothing happened ;(

sleid · Post by **sleid** » 30 Aug 2017 08:45

zpool detach P1 18446744071900754353

prosiakus · Post by **prosiakus** » 30 Aug 2017 08:55

Gr8. its DONE. THX all.
I have done it yesterday but your erlier post with this logic:

zpool clear P1

zpool clear P1 dev/ada6

zpool clear P1 18446744071900754353

zpool replace P1 18446744071900754353 /dev/ada6

give way to finish this:
zpool detach P1 18446744071900754353
with succes!!!

I realy done it yesterday but not in this way

Post by **raulfg3** » 30 Aug 2017 08:58

if problem is solved, please edit & mark first post as [SOLVED]

Thanks.

XigmaNAS

[SOLVED] ZFS resilvering problem

[SOLVED] ZFS resilvering problem

Re: ZFS resilvering problem

Re: ZFS resilvering problem

Re: ZFS resilvering problem

Re: ZFS resilvering problem

Re: ZFS resilvering problem

Re: ZFS resilvering problem

Re: ZFS resilvering problem

Re: ZFS resilvering problem

Re: ZFS resilvering problem

Re: ZFS resilvering problem

Re: ZFS resilvering problem

Re: ZFS resilvering problem

Re: ZFS resilvering problem

Re: ZFS resilvering problem

Re: ZFS resilvering problem

Re: ZFS resilvering problem

Re: ZFS resilvering problem

Re: ZFS resilvering problem