ZFS - Repairing extemly slow
Posted: 20 Aug 2016 16:53
Hello
I have Nas4free 9.1 (Sandstorm) running and replaced for a few days a disk,
Before i had 3TB,3TB,3TB,2TB and after replacing the old 2TB ,#
i have now 4x3TB Western Digital Red (NAS) WD30EFRX.
Autoexpand was set to on, Replacing was no Problem.
Before and after the Resilvering i also scubed the Pool.
Now after 10 days uptime, i noticed that occasionally it takes longer to Access some files
over SMB from the Windows Client.
I could see in the log file error Messages about the new, replaced 3TB harddisk.
I started scrubbing again to see if maybe a problem is in the pool.
Til 75% ths scrub runs normal , but then the Status switched to "repair".
Now, around 6 hours later i see 77,58%.
Everything is in slow Motion. Onyl a few 31 "Megabytes" are reported til yet,
I don't know if i should wait til scrub is complete.
The expected time till finish is growing in the last hours.
The Access over the Windows Client is also in slow Motion.
The NAS itself is idleing and has no CPU load.
I don't know if i should replace the recently new harddisk as soon as possible
or if i shall look elsewhere.
pool: RZ1
state: ONLINE
scan: scrub in progress since Sat Aug 20 01:21:42 2016
6.22T scanned out of 8.02T at 117M/s, 4h28m to go
31.0M repaired, 77.58% done
config:
NAME STATE READ WRITE CKSUM
RZ1 ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ada0 ONLINE 0 0 0
ada1 ONLINE 0 0 0
ada2 ONLINE 0 0 0
ada3 ONLINE 0 0 0 (repairing)
errors: No known data errors
Over the Log File i see every few seconds these error Messages.
Hundreds of them in the last hours. I only clipped These two sequences as an example.
Aug 20 16:47:55 nas4free kernel: (ada3:ahcich3:0:0:0): Retrying command
Aug 20 16:47:55 nas4free kernel: (ada3:ahcich3:0:0:0): RES: 41 40 b7 b5 1c 40 01 01 00 00 00
Aug 20 16:47:55 nas4free kernel: (ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Aug 20 16:47:55 nas4free kernel: (ada3:ahcich3:0:0:0): CAM status: ATA Status Error
Aug 20 16:47:55 nas4free kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 30 b5 1c 40 01 01 00 01 00 00
Aug 20 16:47:48 nas4free kernel: (ada3:ahcich3:0:0:0): Retrying command
Aug 20 16:47:48 nas4free kernel: (ada3:ahcich3:0:0:0): RES: 41 40 b0 b5 1c 40 01 01 00 00 00
Aug 20 16:47:48 nas4free kernel: (ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Aug 20 16:47:48 nas4free kernel: (ada3:ahcich3:0:0:0): CAM status: ATA Status Error
Aug 20 16:47:48 nas4free kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 30 b5 1c 40 01 01 00 01 00 00
Aug 20 16:47:41 nas4free kernel: (ada3:ahcich3:0:0:0): Retrying command
Aug 20 16:47:41 nas4free kernel: (ada3:ahcich3:0:0:0): RES: 41 40 b0 b5 1c 40 01 01 00 00 00
Aug 20 16:47:41 nas4free kernel: (ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Aug 20 16:47:41 nas4free kernel: (ada3:ahcich3:0:0:0): CAM status: ATA Status Error
Aug 20 16:47:41 nas4free kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 30 b5 1c 40 01 01 00 01 00 00
Aug 20 16:47:25 nas4free kernel: (ada3:ahcich3:0:0:0): Error 5, Retries exhausted
Aug 20 16:47:25 nas4free kernel: (ada3:ahcich3:0:0:0): RES: 41 40 a8 a5 1c 40 01 01 00 00 00
Aug 20 16:47:25 nas4free kernel: (ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Aug 20 16:47:25 nas4free kernel: (ada3:ahcich3:0:0:0): CAM status: ATA Status Error
Aug 20 16:47:25 nas4free kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 58 a5 1c 40 01 01 00 01 00 00
Aug 20 16:47:17 nas4free kernel: (ada3:ahcich3:0:0:0): Retrying command
Aug 20 16:47:17 nas4free kernel: (ada3:ahcich3:0:0:0): RES: 41 40 af a5 1c 40 01 01 00 00 00
Aug 20 16:47:17 nas4free kernel: (ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Aug 20 16:47:17 nas4free kernel: (ada3:ahcich3:0:0:0): CAM status: ATA Status Error
Aug 20 16:47:17 nas4free kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 58 a5 1c 40 01 01 00 01 00 00
I have Nas4free 9.1 (Sandstorm) running and replaced for a few days a disk,
Before i had 3TB,3TB,3TB,2TB and after replacing the old 2TB ,#
i have now 4x3TB Western Digital Red (NAS) WD30EFRX.
Autoexpand was set to on, Replacing was no Problem.
Before and after the Resilvering i also scubed the Pool.
Now after 10 days uptime, i noticed that occasionally it takes longer to Access some files
over SMB from the Windows Client.
I could see in the log file error Messages about the new, replaced 3TB harddisk.
I started scrubbing again to see if maybe a problem is in the pool.
Til 75% ths scrub runs normal , but then the Status switched to "repair".
Now, around 6 hours later i see 77,58%.
Everything is in slow Motion. Onyl a few 31 "Megabytes" are reported til yet,
I don't know if i should wait til scrub is complete.
The expected time till finish is growing in the last hours.
The Access over the Windows Client is also in slow Motion.
The NAS itself is idleing and has no CPU load.
I don't know if i should replace the recently new harddisk as soon as possible
or if i shall look elsewhere.
pool: RZ1
state: ONLINE
scan: scrub in progress since Sat Aug 20 01:21:42 2016
6.22T scanned out of 8.02T at 117M/s, 4h28m to go
31.0M repaired, 77.58% done
config:
NAME STATE READ WRITE CKSUM
RZ1 ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ada0 ONLINE 0 0 0
ada1 ONLINE 0 0 0
ada2 ONLINE 0 0 0
ada3 ONLINE 0 0 0 (repairing)
errors: No known data errors
Over the Log File i see every few seconds these error Messages.
Hundreds of them in the last hours. I only clipped These two sequences as an example.
Aug 20 16:47:55 nas4free kernel: (ada3:ahcich3:0:0:0): Retrying command
Aug 20 16:47:55 nas4free kernel: (ada3:ahcich3:0:0:0): RES: 41 40 b7 b5 1c 40 01 01 00 00 00
Aug 20 16:47:55 nas4free kernel: (ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Aug 20 16:47:55 nas4free kernel: (ada3:ahcich3:0:0:0): CAM status: ATA Status Error
Aug 20 16:47:55 nas4free kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 30 b5 1c 40 01 01 00 01 00 00
Aug 20 16:47:48 nas4free kernel: (ada3:ahcich3:0:0:0): Retrying command
Aug 20 16:47:48 nas4free kernel: (ada3:ahcich3:0:0:0): RES: 41 40 b0 b5 1c 40 01 01 00 00 00
Aug 20 16:47:48 nas4free kernel: (ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Aug 20 16:47:48 nas4free kernel: (ada3:ahcich3:0:0:0): CAM status: ATA Status Error
Aug 20 16:47:48 nas4free kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 30 b5 1c 40 01 01 00 01 00 00
Aug 20 16:47:41 nas4free kernel: (ada3:ahcich3:0:0:0): Retrying command
Aug 20 16:47:41 nas4free kernel: (ada3:ahcich3:0:0:0): RES: 41 40 b0 b5 1c 40 01 01 00 00 00
Aug 20 16:47:41 nas4free kernel: (ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Aug 20 16:47:41 nas4free kernel: (ada3:ahcich3:0:0:0): CAM status: ATA Status Error
Aug 20 16:47:41 nas4free kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 30 b5 1c 40 01 01 00 01 00 00
Aug 20 16:47:25 nas4free kernel: (ada3:ahcich3:0:0:0): Error 5, Retries exhausted
Aug 20 16:47:25 nas4free kernel: (ada3:ahcich3:0:0:0): RES: 41 40 a8 a5 1c 40 01 01 00 00 00
Aug 20 16:47:25 nas4free kernel: (ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Aug 20 16:47:25 nas4free kernel: (ada3:ahcich3:0:0:0): CAM status: ATA Status Error
Aug 20 16:47:25 nas4free kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 58 a5 1c 40 01 01 00 01 00 00
Aug 20 16:47:17 nas4free kernel: (ada3:ahcich3:0:0:0): Retrying command
Aug 20 16:47:17 nas4free kernel: (ada3:ahcich3:0:0:0): RES: 41 40 af a5 1c 40 01 01 00 00 00
Aug 20 16:47:17 nas4free kernel: (ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Aug 20 16:47:17 nas4free kernel: (ada3:ahcich3:0:0:0): CAM status: ATA Status Error
Aug 20 16:47:17 nas4free kernel: (ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 58 a5 1c 40 01 01 00 01 00 00