Slow transfer speeds for first 15-20 seconds ...

[psechser]

I observe some weird transfer speeds when copying to my NF server. For the first 15 - 20 seconds transfer speed goes up and down between almost 0 and some 100 mbit/s. After that it goes up to some 400 mbit/s and remains there pretty constantly for this transfer.

For additional file copy operations it may show the same speed pattern, but sometimes it just goes up to 400 mbit/s right away and stays there until file transfer finishes. But whether or not it shows initially slow speed is not predictable excet the very first transfer, during which it always shows slow speed ...

any idea?

thanks

[ChriZathens]

Do you perhaps have set your disks to spindown after x time and at the first transfer it happens because it waits to spin up the disks?

[psechser]

Actually power management was disabled for all disks. I now explicitely set them to "max performance", but still same behavior ... ???

[b0ssman]

spindown is not the same setting as max performance.

have you got wd greens?

[psechser]

I set them to "max performance" and to "always on". My disks are all either Seagate or Samsung, no WD ...

[raulfg3]

Please post your exact disk and how is used ( mixed), I suspect 7200 and 5400 rpm Disk mix.

In this case you allways see different read speeds.

Post too Adaptec SATA Controller and if this controller is PCI .

PD: If not installed, try ZFSkerntune to tune your NAS to 16Gb: viewtopic.php?f=71&t=1278&p=19924#p4894

[psechser]

ah ... should have said, that this behavior only applies to WRITES to NF. READS from NF are ok. Write cache is disabled and read cache is enabled on the SATA controller.

ZFSkerntune is installed and tune to 16gb.

[EDIT 1]

SATA controller: PCIe Adaptec ICP5085BR, Rel. 5.2-0[15617] supporting 2TB disks with firmware upgrade

Disks:

<SAMSUNG HD204UI 0001>
<ST2000DL 003-9VT166 CC32>
<ST2000DL 003-9VT166 CC32>
<SAMSUNG HD204UI 0001>
<ST2000DL 003-9VT166 CC32>
<ST2000DL 003-9VT166 CC32>
<ST2000DL 003-9VT166 CC32>
<ST2000DM 001-1CH164 CC24>

[raulfg3]

Please revise firmware version of Samsumg disk:
https://ata.wiki.kernel.org/index.php/K ... corruption
http://forums.seagate.com/t5/Samsung-In ... m-p/142695

[b0ssman]

whats the speed with the write cache enabled?

[raulfg3]

go to Disks|ZFS|Pools|I/O Statistics
http://wiki.nas4free.org/doku.php?id=do ... statistics

copy (write) and Read several Gb so writes & reads time where 5 minutes at least.

Post result of read and post result of write to detect if some disk are the bottleneck.

Post Disks|ZFS|Pools|Information to see if there are some read or write errors detected.
http://wiki.nas4free.org/doku.php?id=do ... nformation

[raulfg3]

one question more: do you have Installation : x64-full, but, do you define swap?.

If yes, save your config and do a new fresh full install, this time without swap file, and when promt about size for boot slice, change 384MB (default), by 2048 MB =2GB and do not define swap, once done, boot, restore your previosly saved config and repeat test and see if transfer are stable this time.

[psechser]

I do have 64 bit full insatllation, no swap ...

will test a bit with disks speed ... immedaite checks showed almost equal speed across all disks ...

I used same disks in first raidz1 vdev on earlier version of NF (replaced the second vdev to include all 2TB disks, though) and experienced speeds up to 600 mbit/s. FTP shows an immediate speed of 250 mbits , but not more ...

[alexey123]

Hmm, If you use zfs, I think you need swap. When system create snapshot it need more then 16G ram.
Also zfs kernel tune need for amd64
Test also without stand by disks? may be you have disk with autoparking after 5 min inactivity

[raulfg3]

I find a lot of info about ZFS and Swap but not sure if it's aplicable, all talk about ZFS root volumes ( not the nas4Free case):

http://docs.oracle.com/cd/E19253-01/819 ... index.html
http://docs.oracle.com/cd/E23824_01/htm ... hp?t=30298

when I say no swap is because his boot disk is old & slow

System HDD : ancient 4GB QUANTUM BIGFOOT CY4320A A03.0800

and have enought RAM, so in my understand do not need swap, in fact I originally think that user have swap and that this swap are the slow botleneck ( Actually is clear that is not the botleneck).


PD: Do you test with write cache ON like say b0ssman ?

[alexey123]

I not tried, but nas4free give use swap as file. May be possible place it into zfs?

[psechser]

I find a lot of info about ZFS and Swap but not sure if it's aplicable, all talk about ZFS root volumes ( not the nas4Free case):

http://docs.oracle.com/cd/E19253-01/819 ... index.html
http://docs.oracle.com/cd/E23824_01/htm ... hp?t=30298

when I say no swap is because his boot disk is old & slow

System HDD : ancient 4GB QUANTUM BIGFOOT CY4320A A03.0800

and have enought RAM, so in my understand do not need swap, in fact I originally think that user have swap and that this swap are the slow botleneck ( Actually is clear that is not the botleneck).


PD: Do you test with write cache ON like say b0ssman ?

Even I'm not a friend of write caches, I'll do some more testing. I'll also try with other system disk, just to verify this is not the bottleneck, even I wouldn't know, how it would be used for regular file transfers on data disks during a CIFS copy ..

I had also tried another SATA controller (ADAPTEC RAID 6805E, which NF wouldn't support by default), but got even worse results = started and remained slow.

Spinned down disks would slow down the first copy operation but not subsequent ones, so can't be reason (and I turned everything on or off, respectively, which sounded like power/spin down) and FTP has no such ramp up time even staying at relatively low level (~ 250 mbit/s)

Will get back one I tested a bit more ...

[00Roush]

Not sure if I missed it or not... Have you had a chance to test local NF disk speed and network speeds? http://n4f.siftusystems.com/index.php/2 ... leshooting steps 1 and 2 outline the tests. For the Iperf tests I recommend adding -w 64k or -w 128k to the command line to get a realistic max network bandwidth.

Next thing after the local NF server tests would be client tests.
Is the client disk speed good and what OS?
Are large enough files being used for testing? (larger than 1GB)

If all the above checks out I would go back to the NF server and adjust Samba settings to see if it helps. For me I have found the best performance with Samba send and receive buffers set to 0, AIO enabled, and SMB2. If you do end up changing Samba settings I would only make one change at a time.

Let us know how it goes.

00Roush

[psechser]

... surprising observation today:

I have two dfferent shares: one is called ZFS which points to /mnt/zfs and is the root of my zfs pool. And another share called MNT pointing to /mnt. As on my windows system MNT only shows a bit less than 1 GB in size windows wouldn't let me copy files greater than 1 GB to that share. That's why I usually use share ZFS for my transfers with the speed limitations discussed in this thread.

But today I accidentily copied to share MNT as the file I wanted to transfer to NF was less than 1 GB in size. And SURPRISE: I got almost 600 mbit/sec right form the beginning.

This means that issues can't be caused by any disk bottlenecks.

If I now can figure out how to get rid of the error message on my windows system when file is > 1 GB and transferring it to share MNT, I'd be very happy.

How come two different shares result in different transfer speeds?

[00Roush]

Interesting... I am pretty sure /mnt folder is part of system disk folder structure. /mnt/zfs is your zfs pool with all your drives. So it looks like you are able to transfer data faster to your old Quantum drive than to your drives in zfs pool. It could be that you only have 1 gb free on you system disk which is why windows is not allowing larger files.

Have you gotten a chance to do iperf and dd disk tests to verify network and disk speeds are ok for your zfs pool?

00Roush

[raulfg3]

use siftu recomendations for test: http://n4f.siftusystems.com/index.php/2 ... leshooting

[psechser]

Interesting... I am pretty sure /mnt folder is part of system disk folder structure. /mnt/zfs is your zfs pool with all your drives. So it looks like you are able to transfer data faster to your old Quantum drive than to your drives in zfs pool. It could be that you only have 1 gb free on you system disk which is why windows is not allowing larger files.

Have you gotten a chance to do iperf and dd disk tests to verify network and disk speeds are ok for your zfs pool?

00Roush

I am not copying to the system disk. I was just using the share which is mapped to /mnt to navigate to the directory I wanted to copy my file to. So in both cases the target dir is the same like /mnt/zfs/media/Video, however the transfer speed is dependant on the initial share I am using. This also means that the file is always going to the same physical disks ...

Will still do some testing with iperf and dd as soon as I have a bit more time ...

[psechser]

ok .. did some more testing:

iperf: 2 runs

[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 462 MBytes 387 Mbits/sec
[ 5] 0.0-10.0 sec 567 MBytes 475 Mbits/sec

dd shows similar results:

# dd if=/dev/zero of=test.dd bs=2M count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 350.878279 secs (59768647 bytes/sec)

# zdb | grep ashift shows 4k- sectors used
ashift: 12
ashift: 12

MTU is not set

So, neither networknor disks seem to be the bottleneck ... ???

[b0ssman]

i would call that a bottleneck.

387 mbit is terrible for gigabit ethernet. you should have values around 800

and 59mbyte/sec write is very bad as well. you will need numbers above 100.

so there is something seriously wrong with your system.

whats the power management setting for your cpu?

[00Roush]

Iperf speeds are lower than I would expect to see. 800+ Mbps should be possible on most modern gigabit hardware. Did you add -w 64k or 128k to the command to test with larger window sizes? I usually use iperf -c 192.168.0.2 -w 128k. The larger window size ensures you get to see the best case scenario for network throughput.

Based on your DD results it looks like your disk write speed appears to be about 57 MB/sec like b0ssman said. That seems to be a bit low to me for a 4 drive RAIDZ pool. I would expect with your CPU/chipset combo you should be able to get 100+ MB/sec local write speed. My opinion is this could be a bottleneck. For reference I have a 3 drive RAIDZ pool I have been testing here at home that averages about 200 MB/sec local write speed.

Any special settings you are using on your disk controllers or are disks being passed to OS as standard SATA disks? Just checking as ZFS tends to work best when it has direct access to disks. Any special cache settings or RAID options can cause problems.

If I get a chance I might see how one of my AMD setups perform. Just to see if I can possible replicate your issue.

00Roush

[00Roush]

I took some time and tested a bit with an AMD setup I have here at home.
Here is the setup I tested:
Athlon II X4 635 (2.9Ghz) CPU
Asus M3A78-T Motherboard
2 x 2GB RAM (4GB)
3 x 1TB Hitachi (7200 RPM) HD

After a bit of testing and a few setting changes I am seeing around 175 MB/sec local write speed and write speeds over Samba (Win 7 client) are consistently 100-110 MB/sec on large files. Setting changes from default were Samba Send/Receive Buffer Sizes to 0, Samba AIO enabled, and ZFS kernel tune extension set to 4 GB.

From what I can tell my AMD setup works just fine with the onboard SATA ports. Have you had a chance to test using the onboard SATA ports?

You mentioned above you disabled write cache on your SATA (Adaptec) controller card. I looked up the card you mentioned and it looks like there might be two different settings for write cache. The write cache on the disk itself and the write cache available on the controller itself. Are both disabled on your setup? My understanding is it is best to keep the cache on the disk enabled if possible to allow for better performance. (http://www.solarisinternals.com/wiki/in ... ices_Guide) Just wanted to pass along the info.

Hope it helps.

00Roush

[psechser]

I just disabled the write cache on the controller. Never touched the disk cache (woudln't actually know, how to do that). I prefer data protection over performance, thus turning off write caching.

In the meantime I replaced the system disk, reinstalled NF and configured the serfevr from scrtach without using any of the stored config files to avoid some weird settings are restored as well.

I am now getting better transfer rates of up to 600 mbits/sec, but the inital phase of very low rates still pertains. So, final speed is better, but problem still remains.

I haven't tried onboard SATA connections for several reasons the first being, that I don't have enough connectors for my 8 disks. (Otherwise I would have connected them internally in the first place)

And there aremany reasons, why transfer would not perform well. All above tips are addressing the generic low performance and I'd really thank you guys fro bringing them up.

But none of them actually address the different speeds for one single file. I do get speeds which I am happy with, but not during the first 20 seconds of the transfer.

So, a bit of a rephrase of my initial quesion: What elements in a transfer impact its speed during a single file tansfer?

[b0ssman]

since noone has a similar problem and you are using the adaptec controller that would be a thing to investigate

[00Roush]

Not sure of your exact setup but if you wanted you can use your onboard SATA connections along with the Adaptec card. So you could have some hard drives connected to onboard and some drives connected to Adaptec card. For testing I was just maybe suggesting testing out 4 disks on the onboard ports in their own RAIDZ1 array. Now that I think about it do you have data on the disks or is this a fresh setup of ZFS pools?

There are lots of things that affect transfer speeds during a single file transfer...
Server/client raw network speed (and also network latency)
Server/client raw disk speed
Server/client OSes
Client file transfer engine
Size of the file being transferred

The raw network and disk speeds basically set the maximum possible speeds you can attain. High latency (long distances between computers) networks can cause slow speeds as acknowledgments between server/client take longer. If disk speeds or network speeds are low it can be caused by hardware not fully being supported.

The OS on the server and client can make a huge difference in transfer speeds as they might not be setup to provide high performance file transfers out of the box. For example Windows to Windows typically works well as they are the same but Mac OSX to Windows might not work as well.

File transfer engine is the software being used on the client to do the file transfer. Lot of software can copy a file from one place to another on your computer but they are not all created equal. I know starting with Windows Vista SP1 the built in file copy engine in Windows was designed to be able to keep multiple IO's in flight at a time. This is necessary to be able to get high performance file copies over a network connection. Windows XP for example had a built in file copy engine that would only really do 1 IO at a time. Worked fine for local file copies as there is very minimal latency. File copies over a network however would typically max out at around 60 MB/sec in XP versus Vista/7 being capable of 100+ MB/sec and higher

For your case it is possible that the Adaptec card is not working well with NAS4Free which could be contributing to your slower initial transfer speeds. That is why I suggested trying out the onboard SATA. Another thing that could be causing a slow initial speeds is the client setup. OS, disk setup etc can affect how quickly speeds ramp up.

00Roush

[00Roush]

I was doing a bit more reading about RAID controllers and ZFS... Recommendations I have seen are to disable both write cache and read ahead for RAID cards. The logic behind this is because ZFS is designed to be the disk controller and if something is in between that is doing read ahead or write cache it can cause performance problems or data loss. Example would be if read ahead is enabled on RAID card it might start reading ahead of data that is being requested by ZFS. Then ZFS starts trying to write data but the RAID controller is busy reading other data.

You could maybe try disabling read ahead to see if it helps with your issue.

Just wanted to pass the info along.

00Roush

[psechser]

Unfortunately the controller doesn't offer an option to turn off read caching on a generic level, I defined this on disk level. However it is turned off for write caching. And on disk level I can't change read cache once the array is defined in the controller without having to reinit the disk, which I am a bit hesitant to do, as it may require to restore my 8 TBs from my backup ...

Finally I got a SUSE Linux SLED 11.2 machine up and CIFS-mounted the shares on my NF server. On this system I get reads maxed out at 300 mbits/sec but from first second onwards. Interesting enough writes go up to 600 mbits/sec right away and remain between 400 mbits/sec and 600 mbits/sec during that transfer.

So, it has something to do with the CIFS-client, even I don't know what, yet ...