Page 1 of 1
L2ARCs are not growing anymore
Posted: 31 May 2014 11:29
by Mrxlazuardin
Hi,
My L2ARC size is stuck around 50GB allocated size even there is more room on SSDs. How can I achieve more L2ARC? Following is the output of "zpool iostat -v", "top" (Memory & ARC related lines), "sysctl -a" (l2arc related lines).
Best regards,
capacity operations bandwidth
pool alloc free read write read write
----------- ----- ----- ----- ----- ----- -----
pool0 1.09T 63.9T 2.96K 0 377M 0
mfid0.nop 1.09T 63.9T 2.96K 0 377M 0
cache - - - - - -
da0 11.7G 175G 2 0 384K 0
da1 12.2G 174G 4 0 639K 0
da2 12.1G 174G 0 0 0 0
da3 12.1G 174G 0 0 0 0
----------- ----- ----- ----- ----- ----- -----
Mem: 27M Active, 22M Inact, 47G Wired, 8816K Buf, 46G Free
ARC: 46G Total, 2268M MFU, 43G MRU, 16K Anon, 298M Header, 85M Other
vfs.zfs.l2arc_norw: 0
vfs.zfs.l2arc_feed_again: 1
vfs.zfs.l2arc_noprefetch: 0
vfs.zfs.l2arc_feed_min_ms: 200
vfs.zfs.l2arc_feed_secs: 1
vfs.zfs.l2arc_headroom: 2
vfs.zfs.l2arc_write_boost: 268435456
vfs.zfs.l2arc_write_max: 268435456
Re: L2ARCs are not growing anymore
Posted: 16 Jun 2014 19:04
by Mrxlazuardin
Any clue?
Re: L2ARCs are not growing anymore
Posted: 16 Jun 2014 21:57
by crowi
why are you using 4 SSDs as L2ARC and how are they configured?
Re: L2ARCs are not growing anymore
Posted: 17 Jun 2014 05:25
by Mrxlazuardin
Hi,
It is 800GB Intel 910 PCIe SSD. Internally, that card is devided as 4 200GB devices via LSI 2008 chip (mps).
da0 at mps0 bus 0 scbus0 target 0 lun 0
da0: <INTEL(R) SSD 910 200GB a424> Fixed Direct Access SCSI-6 device
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da0: 190782MB (390721968 512 byte sectors: 255H 63S/T 24321C)
da1 at mps0 bus 0 scbus0 target 1 lun 0
da1: <INTEL(R) SSD 910 200GB a424> Fixed Direct Access SCSI-6 device
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 190782MB (390721968 512 byte sectors: 255H 63S/T 24321C)
da2 at mps0 bus 0 scbus0 target 2 lun 0
da2: <INTEL(R) SSD 910 200GB a424> Fixed Direct Access SCSI-6 device
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 190782MB (390721968 512 byte sectors: 255H 63S/T 24321C)
da3 at mps0 bus 0 scbus0 target 3 lun 0
da3: <INTEL(R) SSD 910 200GB a424> Fixed Direct Access SCSI-6 device
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 190782MB (390721968 512 byte sectors: 255H 63S/T 24321C)
What do you think?
Best regards,
Re: L2ARCs are not growing anymore
Posted: 17 Jun 2014 08:10
by crowi
Hi Mrxlazuardin,
as far as I know, you can only add one vdev as cache to your pool and BSD sees your 800 GB drive as 4 drives.
So, you should create a ZFS stripe vdev of da0-da3 first, then add this stripe as L2ARC.
The problem I see here, you have just 48 GB RAM for a 64 TB Pool, you need 1GB RAM / 1TB Disk space plus 1-2 GB RAM overhead AND in your case you also need a lot of RAM to manage the BIG 800 GB L2ARC.
So in your setup you won't gain advantages from the L2ARC, as it consumes your RAM, which is already too small for your Pool.
Running this setup, I would recommend 128 GB RAM or more.
Re: L2ARCs are not growing anymore
Posted: 17 Jun 2014 08:30
by b0ssman
take a step back.
it looks like you have create a raid device on your controller mfid0 and used that device to create a zfs array.
this negates all the benefits that zfs gives you. you might as well use ufs.
Re: L2ARCs are not growing anymore
Posted: 17 Jun 2014 08:45
by Mrxlazuardin
Hi b0ssman,
You are right about the array, but I think itis not related to cache allocation problem.
Re: L2ARCs are not growing anymore
Posted: 17 Jun 2014 09:06
by b0ssman
can you also tell us in what environment this server is used and why you added a l2arc?
to be honest with your raid under zfs setup up i would recreate the pool and start from scratch.
Re: L2ARCs are not growing anymore
Posted: 17 Jun 2014 09:19
by crowi
I think itis not related to cache allocation problem.
This might be due to lack of RAM, because, as said before, L2ARC needs RAM to allocate space, if there is no RAM left, it can't be allocated.
Re: L2ARCs are not growing anymore
Posted: 17 Jun 2014 12:13
by Mrxlazuardin
HI Crowi,
As my first posting, there is 46GB free RAM. It seem that the allocated size of L2ARC is slightly more than ARC total size. Is it related? If yes, how can make ARC consume more from free RAM?
Best regards,
Re: L2ARCs are not growing anymore
Posted: 17 Jun 2014 12:21
by b0ssman
did you install zfskerntune?
tune the system for your memory and enabled prefetch?
Re: L2ARCs are not growing anymore
Posted: 17 Jun 2014 12:26
by crowi
You have 46G reserved for ARC so there is not much left for managing L2ARC
Re: L2ARCs are not growing anymore
Posted: 17 Jun 2014 13:10
by Mrxlazuardin
Hi Crowi, B0ssman,
I cannot install anything since it is embedded version. Any parameter I can adjust for solving this case?
Best regards,
Re: L2ARCs are not growing anymore
Posted: 17 Jun 2014 13:17
by b0ssman
you can install zfskerntune on the embedded version. the config from its is permanent.
Re: L2ARCs are not growing anymore
Posted: 17 Jun 2014 13:33
by crowi
yes, and you can also add your own additional parameters via webGUI under System | Advanced.
Try to enter the following settings to your loader.conf:
Code: Select all
vfs.zfs.arc_min="32768M"
vfs.zfs.arc_max="32768M"
vfs.zfs.prefetch_disable="0"
vfs.zfs.txg.timeout="5"
vfs.zfs.vdev.max_pending="10"
vfs.zfs.vdev.min_pending="4"
vfs.zfs.write_limit_override="0"
vfs.zfs.no_write_throttle="0"
vm.kmem_size="36864M"
vfs.zfs.l2arc_write_max="200MB"
vfs.zfs.l2arc_write_boost="380MB"
But most important for you is first to add more RAM and then even a bit more RAM.
This is a good read about:
http://constantin.glez.de/blog/2010/04/ ... erformance
Otherwise you don't have any benefit from your L2ARC.