ZFS speed & memory issue

[Impact]

My ZFS pool has a dead drive and has begun to run very slowly. I am currently running the 'zpool replace' task, but it is moving very very slowly. I think my ZFS arc tuning may be part of the problem, as the machine never seems to use more than 1GB of the system memory (8GB installed).

Could somebody please offer some insight? I am pulling out my hair trying to get it back into an operating state. Here is all the relevant information I could find:

top
CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
Mem: 6524K Active, 9928K Inact, 529M Wired, 256K Cache, 14M Buf, 7335M Free

/boot/loader.conf
kernel="kernel"
bootfile="kernel"
kernel_options=""
kern.hz="100"
# ZFS kernel tune
vm.kmem_size="7000M"
vfs.zfs.arc_min="5000M"
vfs.zfs.arc_max="6000M"
vfs.zfs.prefetch_disable="1"
vfs.zfs.zil_disable="0"
vfs.zfs.txg.timeout="10"
vfs.zfs.vdev.max_pending="3"
vfs.zfs.vdev.min_pending="6"

zpool status

Code: Select all

  pool: Tank
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scrub: resilver in progress for 7h2m, 2.41% done, 285h59m to go
config:

	NAME           STATE     READ WRITE CKSUM
	Tank           ONLINE       0     0     0
	  raidz1       ONLINE       0     0     0
	    replacing  ONLINE       0     0     0
	      ad14     ONLINE      16     0     0
	      ad10     ONLINE       0     0     0  38.8G resilvered
	    ad15       ONLINE       0     0     0  62.9M resilvered
	    ad16       ONLINE       0     0     0  70.5M resilvered
	    ad17       ONLINE       0     0     0  62.8M resilvered
	    ad18       ONLINE       0     0     0  70.8M resilvered
	    ad20       ONLINE       0     0     0  62.9M resilvered

errors: No known data errors

uname -a
FreeBSD freenas 7.3-RELEASE-p7 FreeBSD 7.3-RELEASE-p7 #0: Sun Oct 9 05:11:39 JST 2011 aoyama@fbsd7.freenas.local:/usr/obj/freenas/usr/src/sys/FREENAS-amd64 amd64

[b0ssman]

vfs.zfs.prefetch_disable="1" should be "0"

[Impact]

Thank you for the quick reply, b0ssman. If I change the vfs.zfs.prefetch_disable setting, can I reboot while it is replacing the drive? Do you think that will increase the memory usage to speed things up?

Thanks!
Impact

[b0ssman]

i dont think it will impact the resilver speed, but i am not 100% sure.

2,4% in 7 hours is very slow.

Can you post the hardware and controller used?

[substr]

why is your max_pending less than your min_pending? Are the values reversed?

[Impact]

This same ZFS pool used to run at very usable speeds. I believe the machine used to sit at nearly 100% memory usage before I did an update and mucked with the ZFS settings. My best guess is that I messed something up, but I cannot figure out what. I know that ZFS tends to need a lot of memory to run quickly.

Here are the machine specs:
Motherboard: ASUS P5G43T-M Pro LGA 775 Intel G43 HDMI Micro ATX
Processor: Intel Celeron E3300 Wolfdale 2.5GHz LGA 775 65W Dual-Core
Drives: Western Digital WD AV-GP WD20EVDS 2TB 32MB Cache SATA 3.0Gb/s 3.5"
Controller: SYBA SD-SATA2-4IR PCI SATA II (3.0Gb/s) RAID Controller Card
Memory: G.SKILL Ripjaws Series 8GB (2 x 4GB) 240-Pin DDR3 SDRAM DDR3 1066 (PC3 8500)
Boot drive: Transcend 4GB Compact Flash (CF) Flash Card (connected by SYBA SD-CF-IDE-A IDE to Compact Flash Adapter)

Some of the drives are connected directly to the motherboard, others are connected through the PCI SATA II card.

Thanks,
Impact

[ku-gew]

Why are 38 GB resilvered in the new drive and some MB resilvered in the other drives too?

Since I never did a "replace" I ask: you seem to have a 5+1 RAIDZ1. Did you add the new drive, or did you remove the failed one?
If you didn't remove the failed one, I expect ZFS to try to use that one as well to resilver, making everything slower (if it is failing, reads take much longer).
Take the failed drive out and it may be faster.

And please keep us posted, this is interesting.

BTW: you have about 55% probability of having a read error in the GOOD drives during rebuild. If you keep the failing drive, that error will be corrected, if you take it out, the rebuild will be faster but then who knows what will be the outcome of an uncorretable read error during rebuild. If you are lucky, it will be skipped and you will lose a block, if you are unlucky... everything stops.

As info: formula to calculate the probability of an uncorrectable read error:
a= error probability (1E-14 for consumer drives, 1E-15 or 1E-16 for enterprise drives)
b= n-1 size of pool
c= size of each disk

probability= 1–(1-a)^(8*b*c)
a= 1E-14, b= 5, c= 2E12 (=2TB) and you get 55%.

[Impact]

ku-gew,

Thanks for the response. Yes, I have left the old drive in while replacing it with an identical drive that was already in the machine. I do suspect that this is slowing down the resilvering, however I still think that the memory usage is also involved.

substr,

The switched vfs.zfs.vdev.max_pending and vfs.zfs.vdev.min_pending values are definitely a problem. I wonder what affect they are having?

Today:

Code: Select all

  pool: Tank
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
	attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
	using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://www.sun.com/msg/ZFS-8000-9P
 scrub: resilver in progress for 20h32m, 10.75% done, 170h28m to go
config:

	NAME           STATE     READ WRITE CKSUM
	Tank           ONLINE       0     0     0
	  raidz1       ONLINE       0     0     0
	    replacing  ONLINE       0     0     0
	      ad14     ONLINE      16     0     0
	      ad10     ONLINE       0     0     0  173G resilvered
	    ad15       ONLINE       6     0     0  118M resilvered
	    ad16       ONLINE       0     0     0  130M resilvered
	    ad17       ONLINE       0     0     0  117M resilvered
	    ad18       ONLINE       0     0     0  131M resilvered
	    ad20       ONLINE       0     0     0  118M resilvered

errors: No known data errors

Thanks,
Impact

[ku-gew]

I never resilvered a RAIDZ1, but I can't see why memory should affect resilvering speed. Data are always taken from the drive during resilver and almost no memory is needed for the low-level process, it's just a bunch of XORs.
Try posting the first 3-5 outputs of "zpool iostat -v 5".

[substr]

If the switched values are having any effect, it would be to limit the number of I/Os that can be issued at once. There is a small chance it is the cause of slow performance.

Fixing the values and rebooting should be all it takes. The resilver will continue after the reboot.

You should be concerned why you have so many corrections over all the drives. Have you ever run a scrub before?

[Impact]

As requested, here are the first five results of the "zpool iostat -v 5" command. I have no idea if these are good numbers or not.

Code: Select all

                  capacity     operations    bandwidth
pool            used  avail   read  write   read  write
-------------  -----  -----  -----  -----  -----  -----
Tank           9.44T  1.47T     77      2  9.52M  12.8K
  raidz1       9.44T  1.47T     77      2  9.52M  12.8K
    replacing      -      -      0     80      0  1.94M
      ad14         -      -      0      0    292  1.65K
      ad10         -      -      0     50      0  1.94M
    ad15           -      -     38      1  1.94M  2.59K
    ad16           -      -     22      1  1.94M  2.84K
    ad17           -      -     22      1  1.94M  2.59K
    ad18           -      -     38      1  1.94M  2.85K
    ad20           -      -     38      1  1.94M  2.60K
-------------  -----  -----  -----  -----  -----  -----

                  capacity     operations    bandwidth
pool            used  avail   read  write   read  write
-------------  -----  -----  -----  -----  -----  -----
Tank           9.44T  1.47T      0      2      0  7.18K
  raidz1       9.44T  1.47T      0      2      0  7.18K
    replacing      -      -      0      2      0  1.50K
      ad14         -      -      0      0      0  1.50K
      ad10         -      -      0      3      0  5.99K
    ad15           -      -      0      1      0  1.30K
    ad16           -      -      0      3      0  5.79K
    ad17           -      -      0      4      0  5.09K
    ad18           -      -      0      4      0  5.59K
    ad20           -      -      0      4      0  5.19K
-------------  -----  -----  -----  -----  -----  -----

                  capacity     operations    bandwidth
pool            used  avail   read  write   read  write
-------------  -----  -----  -----  -----  -----  -----
Tank           9.44T  1.47T      0      0      0  1.40K
  raidz1       9.44T  1.47T      0      0      0  1.40K
    replacing      -      -      0      0      0    408
      ad14         -      -      0      0      0    408
      ad10         -      -      0      0      0      0
    ad15           -      -      0      0      0    408
    ad16           -      -      0      0      0      0
    ad17           -      -      0      0      0      0
    ad18           -      -      0      0      0      0
    ad20           -      -      0      0      0      0
-------------  -----  -----  -----  -----  -----  -----

                  capacity     operations    bandwidth
pool            used  avail   read  write   read  write
-------------  -----  -----  -----  -----  -----  -----
Tank           9.44T  1.47T      0      1      0  3.69K
  raidz1       9.44T  1.47T      0      1      0  3.69K
    replacing      -      -      0      1      0   1021
      ad14         -      -      0      0      0   1021
      ad10         -      -      0      0      0      0
    ad15           -      -      0      0      0    919
    ad16           -      -      0      0      0      0
    ad17           -      -      0      0      0      0
    ad18           -      -      0      0      0      0
    ad20           -      -      0      0      0      0
-------------  -----  -----  -----  -----  -----  -----

                  capacity     operations    bandwidth
pool            used  avail   read  write   read  write
-------------  -----  -----  -----  -----  -----  -----
Tank           9.44T  1.47T      0      1      0  5.59K
  raidz1       9.44T  1.47T      0      1      0  5.59K
    replacing      -      -      0      1      0  1.30K
      ad14         -      -      0      0      0  1.30K
      ad10         -      -      0      0      0      0
    ad15           -      -      0      0      0  1.20K
    ad16           -      -      0      0      0      0
    ad17           -      -      0      0      0      0
    ad18           -      -      0      0      0      0
    ad20           -      -      0      0      0      0
-------------  -----  -----  -----  -----  -----  -----

Also, here is the status today:

Code: Select all

  pool: Tank
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
	attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
	using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://www.sun.com/msg/ZFS-8000-9P
 scrub: resilver in progress for 45h29m, 19.24% done, 190h54m to go
config:

	NAME           STATE     READ WRITE CKSUM
	Tank           ONLINE       0     0     0
	  raidz1       ONLINE       0     0     0
	    replacing  ONLINE       0     0     0
	      ad14     ONLINE      16     0     0
	      ad10     ONLINE       0     0     0  310G resilvered
	    ad15       ONLINE       6     0     0  176M resilvered
	    ad16       ONLINE       0     0     0  194M resilvered
	    ad17       ONLINE       0     0     0  176M resilvered
	    ad18       ONLINE       0     0     0  195M resilvered
	    ad20       ONLINE       0     0     0  176M resilvered

errors: No known data errors

I am going to try correcting the loader.conf values (min_pending, max_pending, and prefetch_disable) and rebooting next. I will let you all know what happens.

Thanks for the help!
Impact

[ku-gew]

The first set of iostatus shows that on average since last boot your disks have some read/write operations per second, but in the following iostatuses you see there are practically no read/write operations per second: less then 5 per second, the disks can do at least 30 IOPs.
Result: it's not a matter of memory, as I thought, it's a matter of timeouts. Disconnect (o remove from pool, it's enough) the broken one and see if it gets faster. It should.
However, I have NO IDEA what will happen if there are failed checksums in the other drives (and apparently you have a lot of them).

So, basically get over it and wait for the resilver to complete at this speed, there is nothing you can do to speed up the process.
Next time do frequent (weekly) scrubs on the pool and then you will be able to disconnect the failing disk without too much fear. You should also set up daily "short selftest" and weekly "long selftest" on each drive, NAS4free makes it very easy. Just do not let the long selftest and the scrub overlap (they both take up to 1 day to complete).

[ku-gew]

Is ad10 or ad14 failing?

[Impact]

ku-gew,

ad14 is the drive that is failing. Thank you for the help, I will wait it out and see what happens when it finishes and I can safely remove ad14.

Thanks,
Impact