Another Slow Transfer Speeds Post

[rentageekla]

Hi,

We have a rack server running 9.1.0.1 with a 10 hard drive ZFS Pool. We are getting approximately 30-50 MB/s (read or write) per machine and can get a total of 180 MB/s (read) max with 4 or more machines downloading at once. Link aggregation is working well, in other words, the 4 nics seem to be splitting up work almost evenly. CPU usage is generally minimal, but the RAM is often close to full (90%).

Based on the hardware running other software, the speeds should saturate all four gigabit ethernet ports (aka 452 MB/s total output) in link aggregation. We would be happy to just see 1 machine download at 113 MB/s (full saturation of one gigabit line), which another server we have here does.

We've tried different switches, MTUs of 1500-9000 (client and server), Windows and Mac clients (CIFS and AFP), link aggregation on and off, and various samba configuration commands. Any ideas would be greatly appreciated. We have production content on the zfspool, so we can't try reinstalling the OS or rebuilding the RAID.

Specs:
AMD Opteron 6212
Supermicro H8SGL (BIOS 3.0)
32GB RAM
4x Gigabit Adapters w/ 9000 jumbo frames
10x 2TB 7200 RPM HDDs

Thanks so much for your help!

[b0ssman]

pls read siftus list

http://n4f.siftusystems.com/index.php/2 ... leshooting.

i am especially interested in your dd speed. also try turning off the link aggregation to test single link speed.

[rentageekla]

iperf

76MB/s


dd

write=36MB/s
read=2833MB/s (say wha?!)


Faulty drive

All drives passed SMART

http://pastebin.com/Kc840Xum


Monitor your CPU

8-Core beast barely broke a sweat.. most cores were at 0% and the highest was at 24%


Power Daemon

Already set to off


PCI interfaces

Is there something I can try to look into this? It doesn't seem likely to me that this is what is slowing us down.


Advanced Format

10x ST1000NM0011 - Not AF


Is it your client?

Similar results on every client tested


ZFSKerntune

Cool! I installed, set RAM to 32GB (real physical size), and tested iperf and a 1GB file transfer to 1 PC (below are results).

iperf=77MB/s
transfer (read)=33MB/s
transfer (write)=55MB/s


Full filesystem

We're pretty full, but the workload is extremely light. It's a small production house that grabs a file and then an hour later writes that file and 7-8 people do this throughout the day.


Deduplication

N/A


Mixing HDD’s

All the same


RAID level choice

We're using a ZFS Pool


AHCI

Can I verify this setting through Nas4Free? It's difficult to reboot here.


ZIL and L2ARC

N/A


Synchronous writes

Only using CIFS


Dataset attributes

I'm not sure what this is, but I'm pretty sure we aren't using any kind of compression.


RAM

32GB enough?


64 bit

We're on it

[siftu]

RAM should not be full 90% without ZFSkerntune, what was using it? Also how full was your pool? Is it an old pool you have imported from somewhere else?

ZFS does suffer fragmentation which gets worse when the pool is > 80% full.

[rentageekla]

This is a relatively new system and pool. The pool was created on this system from scratch.

Also the RAM seems to have corrected itself after a reboot. I will keep you guys posted as to whether it ramps up again.

In the meantime, is there anything else I can try to figure out why we have such poor performance? OR, does that data tell us anything interesting we should be looking into?

[b0ssman]

since the board only has 6 sata ports what controller are you using to connect the drives?

can you do an individual dd read (not write) test from each drive?

[rentageekla]

since the board only has 6 sata ports what controller are you using to connect the drives?

can you do an individual dd read (not write) test from each drive?

Sure, how do I do it?

[b0ssman]

dd if=/dev/<individual drive> of=/dev/null bs=2M count=10000

where <individual drive> is something like ada0 or da0 etc..