L2ARCs are not growing anymore

[Mrxlazuardin]

Hi,

My L2ARC size is stuck around 50GB allocated size even there is more room on SSDs. How can I achieve more L2ARC? Following is the output of "zpool iostat -v", "top" (Memory & ARC related lines), "sysctl -a" (l2arc related lines).

Best regards,


capacity operations bandwidth
pool alloc free read write read write
----------- ----- ----- ----- ----- ----- -----
pool0 1.09T 63.9T 2.96K 0 377M 0
mfid0.nop 1.09T 63.9T 2.96K 0 377M 0
cache - - - - - -
da0 11.7G 175G 2 0 384K 0
da1 12.2G 174G 4 0 639K 0
da2 12.1G 174G 0 0 0 0
da3 12.1G 174G 0 0 0 0
----------- ----- ----- ----- ----- ----- -----


Mem: 27M Active, 22M Inact, 47G Wired, 8816K Buf, 46G Free
ARC: 46G Total, 2268M MFU, 43G MRU, 16K Anon, 298M Header, 85M Other


vfs.zfs.l2arc_norw: 0
vfs.zfs.l2arc_feed_again: 1
vfs.zfs.l2arc_noprefetch: 0
vfs.zfs.l2arc_feed_min_ms: 200
vfs.zfs.l2arc_feed_secs: 1
vfs.zfs.l2arc_headroom: 2
vfs.zfs.l2arc_write_boost: 268435456
vfs.zfs.l2arc_write_max: 268435456

[Mrxlazuardin]

Any clue?

[crowi]

why are you using 4 SSDs as L2ARC and how are they configured?

[Mrxlazuardin]

Hi,

It is 800GB Intel 910 PCIe SSD. Internally, that card is devided as 4 200GB devices via LSI 2008 chip (mps).

da0 at mps0 bus 0 scbus0 target 0 lun 0
da0: <INTEL(R) SSD 910 200GB a424> Fixed Direct Access SCSI-6 device
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da0: 190782MB (390721968 512 byte sectors: 255H 63S/T 24321C)
da1 at mps0 bus 0 scbus0 target 1 lun 0
da1: <INTEL(R) SSD 910 200GB a424> Fixed Direct Access SCSI-6 device
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 190782MB (390721968 512 byte sectors: 255H 63S/T 24321C)
da2 at mps0 bus 0 scbus0 target 2 lun 0
da2: <INTEL(R) SSD 910 200GB a424> Fixed Direct Access SCSI-6 device
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 190782MB (390721968 512 byte sectors: 255H 63S/T 24321C)
da3 at mps0 bus 0 scbus0 target 3 lun 0
da3: <INTEL(R) SSD 910 200GB a424> Fixed Direct Access SCSI-6 device
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 190782MB (390721968 512 byte sectors: 255H 63S/T 24321C)

What do you think?

Best regards,

[crowi]

Hi Mrxlazuardin,

as far as I know, you can only add one vdev as cache to your pool and BSD sees your 800 GB drive as 4 drives.
So, you should create a ZFS stripe vdev of da0-da3 first, then add this stripe as L2ARC.
The problem I see here, you have just 48 GB RAM for a 64 TB Pool, you need 1GB RAM / 1TB Disk space plus 1-2 GB RAM overhead AND in your case you also need a lot of RAM to manage the BIG 800 GB L2ARC.

So in your setup you won't gain advantages from the L2ARC, as it consumes your RAM, which is already too small for your Pool.

Running this setup, I would recommend 128 GB RAM or more.

[b0ssman]

take a step back.

it looks like you have create a raid device on your controller mfid0 and used that device to create a zfs array.
this negates all the benefits that zfs gives you. you might as well use ufs.

[Mrxlazuardin]

Hi b0ssman,

You are right about the array, but I think itis not related to cache allocation problem.

[b0ssman]

can you also tell us in what environment this server is used and why you added a l2arc?

to be honest with your raid under zfs setup up i would recreate the pool and start from scratch.

[crowi]

I think itis not related to cache allocation problem.

This might be due to lack of RAM, because, as said before, L2ARC needs RAM to allocate space, if there is no RAM left, it can't be allocated.

[Mrxlazuardin]

HI Crowi,

As my first posting, there is 46GB free RAM. It seem that the allocated size of L2ARC is slightly more than ARC total size. Is it related? If yes, how can make ARC consume more from free RAM?

Best regards,

[b0ssman]

did you install zfskerntune?
tune the system for your memory and enabled prefetch?

[crowi]

You have 46G reserved for ARC so there is not much left for managing L2ARC

[Mrxlazuardin]

Hi Crowi, B0ssman,

I cannot install anything since it is embedded version. Any parameter I can adjust for solving this case?

Best regards,

[b0ssman]

you can install zfskerntune on the embedded version. the config from its is permanent.

[crowi]

yes, and you can also add your own additional parameters via webGUI under System | Advanced.

Try to enter the following settings to your loader.conf:

Code: Select all

vfs.zfs.arc_min="32768M"
vfs.zfs.arc_max="32768M"
vfs.zfs.prefetch_disable="0"
vfs.zfs.txg.timeout="5"
vfs.zfs.vdev.max_pending="10"
vfs.zfs.vdev.min_pending="4"
vfs.zfs.write_limit_override="0"
vfs.zfs.no_write_throttle="0"
vm.kmem_size="36864M"
vfs.zfs.l2arc_write_max="200MB"
vfs.zfs.l2arc_write_boost="380MB"

But most important for you is first to add more RAM and then even a bit more RAM.
This is a good read about:
http://constantin.glez.de/blog/2010/04/ ... erformance
Otherwise you don't have any benefit from your L2ARC.