[SOLVED] Transfer speed not fast - Teracopy problem

neptunus · Post by **neptunus** » 25 Aug 2013 21:21

I don't understand why my transfer speed with cifs is not faster then 35MB/s. My gigabit lan connection is fast enough. See my iperf test.

Skyfire Server> Sideswipe Client

Code: Select all

roy@skyfire:~$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 192.168.1.100 port 5001 connected with 192.168.1.30 port 60691
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   888 MBytes   743 Mbits/sec

Code: Select all

sideswipe: movies # iperf -c skyfire -w 130k -fM
------------------------------------------------------------
Client connecting to skyfire, TCP port 5001
TCP window size: 0.13 MByte (WARNING: requested 0.13 MByte)
------------------------------------------------------------
[  3] local 192.168.1.30 port 60691 connected with 192.168.1.100 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   888 MBytes  88.7 MBytes/sec

Sideswipe Server > Skyfire Client

Code: Select all

sideswipe: movies # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  256 KByte (default)
------------------------------------------------------------
[  4] local 192.168.1.30 port 5001 connected with 192.168.1.100 port 42358
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   956 MBytes   801 Mbits/sec

Code: Select all

roy@skyfire:~$  iperf -c sideswipe -w 130k -fM
------------------------------------------------------------
Client connecting to sideswipe, TCP port 5001
TCP window size: 0.25 MByte (WARNING: requested 0.13 MByte)
------------------------------------------------------------
[  3] local 192.168.1.100 port 42358 connected with 192.168.1.30 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   956 MBytes  95.5 MBytes/sec

Settings of CIFS:

Doubling of divide the Send/Receiver Buffer Size with 2, does not change the transfer speeds.

I'm sure it is not on my hard drive speed:

Code: Select all

sideswipe: movies # dd if=/dev/zero of=tmp.dat bs=2048k count=10k
10240+0 records in
10240+0 records out
21474836480 bytes transferred in 16.756023 secs (1281618930 bytes/sec)

=> 1 281 618 930 (bytes / second) = 1.19360064 gigabytes / s

Code: Select all

sideswipe: movies # dd if=tmp.dat of=/dev/null bs=2048k count=10k
10240+0 records in
10240+0 records out
21474836480 bytes transferred in 5.976868 secs (3592991645 bytes/sec)

=> 3 592 991 645 (bytes / second) = 3.34623423 gigabytes / s

I have 10x 2TB HDD with 2 ssd's for logs/cache.

Why is the transfer speed not faster than 70 MB/s?

Post by **raulfg3** » 26 Aug 2013 01:53

Try enable AIO.

And revise ZFS_prefetch_disable must be disable ( prefetch must be enable).

neptunus · Post by **neptunus** » 28 Aug 2013 17:57

Test information in a summarized way, because of server crash!

I have done some extra benchmarking. What do you think of the results and why is NFS and CIFS slow?

ZFS config speed looks oke to my. FTP speeds are around 105 MB/s.

PC Config (bare metal nas4free)
- X9SRL-F
- Intel Xeon E5-1620 - 3.6 GHz
- Seasonic Platinum Series 760 Watt
- Kingston KVR1333D3D4R9SK2/16G (16 GB totaal)

Used disks:
- 10x Hitachi US7K3000 (SATA 600)
- 2x intel 320 (SATA 300)

Configuration: 2x M1015 in IT with disks connected as following:
- 5x Hitachi US7K3000 and 1x intel 320 connected to 1x M1015

Without ZIL & L2ARC
WRITE

Code: Select all

sideswipe:/mnt# dd if=/dev/zero of=zero.file bs=1M count=40960
40960+0 records in
40960+0 records out
42949672960 bytes transferred in 55.288349 secs (776830447 bytes/sec)

READ

Code: Select all

sideswipe:/mnt# dd if=zero.file of=/dev/null bs=1m
40960+0 records in
40960+0 records out
42949672960 bytes transferred in 50.604693 secs (848729046 bytes/sec)

With ZIL & L2ARC
WRITE

Code: Select all

sideswipe:/mnt# dd if=/dev/zero of=zero.file bs=1M count=40960
40960+0 records in
40960+0 records out
42949672960 bytes transferred in 75.277081 secs (570554442 bytes/sec)

READ

Code: Select all

READ
sideswipe:/mnt# dd if=zero.file of=/dev/null bs=1m
40960+0 records in
40960+0 records out
42949672960 bytes transferred in 61.050132 secs (703514828 bytes/sec)

Code: Select all

sideswipe: /zstore # iozone -a -s 24g -r 4096
        Iozone: Performance Test of File I/O
                Version $Revision: 3.397 $
                Compiled for 64 bit mode.
                Build: freebsd

        Run began: Wed Jun 26 17:40:08 2013

        Auto Mode
        File size set to 25165824 KB
        Record Size 4096 KB
        Command line used: iozone -a -s 24g -r 4096
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                            random  random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
        25165824    4096  534718  504527   808964   845510  252624  541583  240000  9654992   248405   506617   525360  845465   858172

iozone test complete.

Code: Select all

 sideswipe: /zstore # iozone -t 2
        Iozone: Performance Test of File I/O
                Version $Revision: 3.397 $
                Compiled for 64 bit mode.
                Build: freebsd

        Run began: Wed Jun 26 17:36:45 2013

        Command line used: iozone -t 2
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
        Throughput test with 2 processes
        Each process writes a 512 Kbyte file in 4 Kbyte records

        Children see throughput for  2 initial writers  =  331156.22 KB/sec
        Parent sees throughput for  2 initial writers   =   19601.20 KB/sec
        Min throughput per process                      =       0.00 KB/sec
        Max throughput per process                      =  331156.22 KB/sec
        Avg throughput per process                      =  165578.11 KB/sec
        Min xfer                                        =       0.00 KB

        Children see throughput for  2 rewriters        = 1219392.81 KB/sec
        Parent sees throughput for  2 rewriters         =  103444.93 KB/sec
        Min throughput per process                      =  598877.31 KB/sec
        Max throughput per process                      =  620515.50 KB/sec
        Avg throughput per process                      =  609696.41 KB/sec
        Min xfer                                        =     500.00 KB

        Children see throughput for  2 readers          = 2749245.25 KB/sec
        Parent sees throughput for  2 readers           =   49035.01 KB/sec
        Min throughput per process                      = 1365384.00 KB/sec
        Max throughput per process                      = 1383861.25 KB/sec
        Avg throughput per process                      = 1374622.62 KB/sec
        Min xfer                                        =     512.00 KB

        Children see throughput for 2 re-readers        = 2717170.38 KB/sec
        Parent sees throughput for 2 re-readers         =   49175.27 KB/sec
        Min throughput per process                      = 1336867.12 KB/sec
        Max throughput per process                      = 1380303.25 KB/sec
        Avg throughput per process                      = 1358585.19 KB/sec
        Min xfer                                        =     508.00 KB

        Children see throughput for 2 reverse readers   = 2547417.12 KB/sec
        Parent sees throughput for 2 reverse readers    =   48100.36 KB/sec
        Min throughput per process                      = 1224101.75 KB/sec
        Max throughput per process                      = 1323315.38 KB/sec
        Avg throughput per process                      = 1273708.56 KB/sec
        Min xfer                                        =     492.00 KB

        Children see throughput for 2 stride readers    = 2516891.00 KB/sec
        Parent sees throughput for 2 stride readers     = 1101501.58 KB/sec
        Min throughput per process                      = 1211275.62 KB/sec
        Max throughput per process                      = 1305615.38 KB/sec
        Avg throughput per process                      = 1258445.50 KB/sec
        Min xfer                                        =     488.00 KB

        Children see throughput for 2 random readers    = 2528375.62 KB/sec
        Parent sees throughput for 2 random readers     = 1115632.23 KB/sec
        Min throughput per process                      = 1254513.75 KB/sec
        Max throughput per process                      = 1273861.88 KB/sec
        Avg throughput per process                      = 1264187.81 KB/sec
        Min xfer                                        =     512.00 KB

        Children see throughput for 2 mixed workload    = 1463057.25 KB/sec
        Parent sees throughput for 2 mixed workload     =   20276.53 KB/sec
        Min throughput per process                      =       0.00 KB/sec
        Max throughput per process                      = 1463057.25 KB/sec
        Avg throughput per process                      =  731528.62 KB/sec
        Min xfer                                        =       0.00 KB

        Children see throughput for 2 random writers    =  589190.19 KB/sec
        Parent sees throughput for 2 random writers     =   20000.63 KB/sec
        Min throughput per process                      =  589190.19 KB/sec
        Max throughput per process                      =  589190.19 KB/sec
        Avg throughput per process                      =  294595.09 KB/sec
        Min xfer                                        =     512.00 KB

        Children see throughput for 2 pwrite writers    =  994503.81 KB/sec
        Parent sees throughput for 2 pwrite writers     =   34344.83 KB/sec
        Min throughput per process                      =  487639.97 KB/sec
        Max throughput per process                      =  506863.84 KB/sec
        Avg throughput per process                      =  497251.91 KB/sec
        Min xfer                                        =     492.00 KB

        Children see throughput for 2 pread readers     = 2819027.25 KB/sec
        Parent sees throughput for 2 pread readers      =   48994.26 KB/sec
        Min throughput per process                      = 1389092.75 KB/sec
        Max throughput per process                      = 1429934.50 KB/sec
        Avg throughput per process                      = 1409513.62 KB/sec
        Min xfer                                        =     504.00 KB

iozone test complete.

Post by **b0ssman** » 29 Aug 2013 08:11

cifs speed of 35 and ftp speed of 105. that seems very odd.

what are the cifs speeds with Async IO?

what is the speed with NT1?

neptunus · Post by **neptunus** » 29 Aug 2013 09:48

b0ssman wrote:cifs speed of 35 and ftp speed of 105. that seems very odd.

what are the cifs speeds with Async IO?

what is the speed with NT1?

CIFS speeds with Async IO are around the 37 MB/s. Speeds with SMB2 or NT1 are almost the same.

I also have change TX/RX buffer sizes with /2 or *2 from the default settings. Also almost no changes.

neptunus · Post by **neptunus** » 07 Sep 2013 17:32

neptunus wrote:
b0ssman wrote:cifs speed of 35 and ftp speed of 105. that seems very odd.

what are the cifs speeds with Async IO?

what is the speed with NT1?
CIFS speeds with Async IO are around the 37 MB/s. Speeds with SMB2 or NT1 are almost the same.

I also have change TX/RX buffer sizes with /2 or *2 from the default settings. Also almost no changes.

Some any ideas?

Post by **raulfg3** » 08 Sep 2013 16:17

no,in the past change TX/RX buffers works fine for me, but do not works for you, so my only suggest is scientific aproach, change only one parameter at time, reboot NAS to be sure that change is active, and repeat test, until find apropiate tune parameters.

I need to do something simmilar when instaled FreeNAS 0.6 several years ago, and finally I have a lot of experience ( at least with my NAS/Motherboard, sorry that this experience do not work for you).

do you read tune-up post?: viewforum.php?f=10

demetri90 · Post by **demetri90** » 12 Sep 2013 23:23

I too am having this issue, disabled LACP, didn't change.

demetri90 · Post by **demetri90** » 13 Sep 2013 17:35

Yeah, bad performance, sometimes my large transfers even stop for a few seconds then resume to around 20-30mbps. I attached a picture showing the inconsistency. Sometimes it will hang and windows will spit out an error.

00Roush · Post by **00Roush** » 15 Sep 2013 10:50

Which NAS4Free version are you running?

I have found that getting rid of the Samba TX/RX buffers seems to work the best for me. Try testing with TX/RX buffers set to 0.

Also you might benefit from using the ZFS kernel tune extension with the amount of memory you have. viewtopic.php?f=71&t=1278

For my testing inside of a VM (4GB RAM) I found the above changes along with SMB2 and AIO have allowed me to see sustained file transfer speeds of about 110 MB/sec (maxed out gigabit) on large files from 1GB to 43GB in size. This is with a 3 disk RAIDZ array.

Let us know if any of that helps. If not I might have a couple of other ideas.

00Roush

demetri90 · Post by **demetri90** » 16 Sep 2013 20:33

00Roush wrote:Which NAS4Free version are you running?

I have found that getting rid of the Samba TX/RX buffers seems to work the best for me. Try testing with TX/RX buffers set to 0.

Also you might benefit from using the ZFS kernel tune extension with the amount of memory you have. viewtopic.php?f=71&t=1278

For my testing inside of a VM (4GB RAM) I found the above changes along with SMB2 and AIO have allowed me to see sustained file transfer speeds of about 110 MB/sec (maxed out gigabit) on large files from 1GB to 43GB in size. This is with a 3 disk RAIDZ array.

Let us know if any of that helps. If not I might have a couple of other ideas.

00Roush

I'm running 9.1.0.1. I set the buffers to 0, started a large transfer, and it spiked around 80-90mbps, but after about 5-10 seconds, it stared getting slower and slower, and eventually stopped, with windows spitting out an error. Would the ZFS tune extension even do anything with a UFS filesystem?

Post by **raulfg3** » 16 Sep 2013 22:06

demetri90 wrote:Would the ZFS tune extension even do anything with a UFS filesystem?

NO, but if possible for you try ZFS is better than UFS, you agree when test it.

demetri90 · Post by **demetri90** » 16 Sep 2013 23:28

raulfg3 wrote:
demetri90 wrote:Would the ZFS tune extension even do anything with a UFS filesystem?
NO, but if possible for you try ZFS is better than UFS, you agree when test it.

I changed the file system of the array to "ZFS storage pool device" under disk management, but my mount point is still a UFS file system for CIFS/SMB. No difference on large transfers.

00Roush · Post by **00Roush** » 17 Sep 2013 08:29

Which 9.1.0.1 version? From what I remember anything earlier than 636 had Samba problems with AIO and SMB2. I recall having something like you described happen to me... Pretty sure when I finally started testing 9.1.0.1.636 problems went away. Any chance you are running a version older than 636?

00Roush

neptunus · Post by **neptunus** » 17 Sep 2013 08:36

00Roush wrote:Which NAS4Free version are you running?

I have found that getting rid of the Samba TX/RX buffers seems to work the best for me. Try testing with TX/RX buffers set to 0.

Also you might benefit from using the ZFS kernel tune extension with the amount of memory you have. viewtopic.php?f=71&t=1278

For my testing inside of a VM (4GB RAM) I found the above changes along with SMB2 and AIO have allowed me to see sustained file transfer speeds of about 110 MB/sec (maxed out gigabit) on large files from 1GB to 43GB in size. This is with a 3 disk RAIDZ array.

Let us know if any of that helps. If not I might have a couple of other ideas.

00Roush

My nas4free version is 9.1.0.1.847. I'm running it with 16GB RAM on. This is with a 10 disk RAIDZ2 array (10x 2TB HDD with 2 ssd's for logs/cache). I also have the ZFS kernel tune extension installed an optimized it for use of 16GB RAM.

When TX/RX buffers set to 0. My speed is about 55 MB/s stable (win7 copy with teracopy r used as tool).

When TX/RX buffers set to 64240. My speed is about 39 MB/s stable (win7 copy with teracopy r used as tool).

You wrote that you had more ideas. I'm very curious, because I think it really should possible to pick up more speed with my hardware configuration.

Buhu · Post by **Buhu** » 17 Sep 2013 09:08

All problems with transfer speed in the last weeks sounds very crazy but i can understand them. I also was impacted with my AMD E350 Board. My plan was to switch to another plattform and the plan exists before there was upcoming problems with transferspeed now. I have did some tests on my old AMD e350 and my new one ASRock A75M-HVS and A$-3300 CPU.

The new hardware plattform is able to push and read data with nearly 110 MB/s without changing the rest off the hardware! Looking for the big differnce my asumption is the the given OS support for "lower and older hardware like AMD E350" is no so good as it was in the past. From my point of view, the platform OS is the the "problem". Now we have "FreeBSD 9.1-RELEASE-p5 (kern.osreldate: 901000) " and my assumption is that "P5" is the "core problem" . In older versions, the release was lower without any problems.....

only my assumptions............

Christian

00Roush · Post by **00Roush** » 18 Sep 2013 03:02

neptunus wrote:
00Roush wrote:Which NAS4Free version are you running?

I have found that getting rid of the Samba TX/RX buffers seems to work the best for me. Try testing with TX/RX buffers set to 0.

Also you might benefit from using the ZFS kernel tune extension with the amount of memory you have. viewtopic.php?f=71&t=1278

For my testing inside of a VM (4GB RAM) I found the above changes along with SMB2 and AIO have allowed me to see sustained file transfer speeds of about 110 MB/sec (maxed out gigabit) on large files from 1GB to 43GB in size. This is with a 3 disk RAIDZ array.

Let us know if any of that helps. If not I might have a couple of other ideas.

00Roush
My nas4free version is 9.1.0.1.847. I'm running it with 16GB RAM on. This is with a 10 disk RAIDZ2 array (10x 2TB HDD with 2 ssd's for logs/cache). I also have the ZFS kernel tune extension installed an optimized it for use of 16GB RAM.

When TX/RX buffers set to 0. My speed is about 55 MB/s stable (win7 copy with teracopy r used as tool).

When TX/RX buffers set to 64240. My speed is about 39 MB/s stable (win7 copy with teracopy r used as tool).

You wrote that you had more ideas. I'm very curious, because I think it really should possible to pick up more speed with my hardware configuration.

With your specs you definitely should be able to max out a gigabit connection provided the client computer has a fast enough drive.

Just to check I downloaded Teracopy and tested it out. Got about 50 MB/sec for large file transfer to/from several different network drives. Using the normal Win 7 copying I generally see 110 MB/sec sustained on larger files. Based on my testing I recommend not using Teracopy as it appears to be slower than the built in windows file copy engine. The built in Win 7 file copy engine is multi-threaded and capable of keeping multiple I/O's in flight at one time which is necessary to be able to get high performance when copying over a network. Is it possible for you to test using just the normal windows copying?

Let me know.

00Roush

demetri90 · Post by **demetri90** » 18 Sep 2013 17:35

My biggest concern is stability at this point, my speeds are jumping around on large transfers.

neptunus · Post by **neptunus** » 18 Sep 2013 20:28

00Roush wrote: With your specs you definitely should be able to max out a gigabit connection provided the client computer has a fast enough drive.

Just to check I downloaded Teracopy and tested it out. Got about 50 MB/sec for large file transfer to/from several different network drives. Using the normal Win 7 copying I generally see 110 MB/sec sustained on larger files. Based on my testing I recommend not using Teracopy as it appears to be slower than the built in windows file copy engine. The built in Win 7 file copy engine is multi-threaded and capable of keeping multiple I/O's in flight at one time which is necessary to be able to get high performance when copying over a network. Is it possible for you to test using just the normal windows copying?

Let me know.

00Roush

After removing Teracopy my connection is stable at 110MB/s to/from host! YES THANKS MEN.

I have copied file's of 1/2/20GB to/from al stable and fast.

Tomorrow I will try with the default settings of nas4free.

00Roush · Post by **00Roush** » 19 Sep 2013 04:09

neptunus wrote:
00Roush wrote: With your specs you definitely should be able to max out a gigabit connection provided the client computer has a fast enough drive.

Just to check I downloaded Teracopy and tested it out. Got about 50 MB/sec for large file transfer to/from several different network drives. Using the normal Win 7 copying I generally see 110 MB/sec sustained on larger files. Based on my testing I recommend not using Teracopy as it appears to be slower than the built in windows file copy engine. The built in Win 7 file copy engine is multi-threaded and capable of keeping multiple I/O's in flight at one time which is necessary to be able to get high performance when copying over a network. Is it possible for you to test using just the normal windows copying?

Let me know.

00Roush
After removing Teracopy my connection is stable at 110MB/s to/from host! YES THANKS MEN.

I have copied file's of 1/2/20GB to/from al stable and fast.

Tomorrow I will try with the default settings of nas4free.

Now that is what I'm talking about! Glad I could help.

Let us know how it turns out with default settings. If you get a chance open a task manager session to view the network throughput. Without zfskerntune I always would see a see-saw for network speed.

00Roush

00Roush · Post by **00Roush** » 19 Sep 2013 04:31

demetri90 wrote:My biggest concern is stability at this point, my speeds are jumping around on large transfers.

Which version of NAS4free are you using?

It has been a while since I tested UFS but I don't remember having too much trouble with sustained speeds. What are the specs on your nas4free setup. CPU, ram, NIC, and drive setup?

00Roush

demetri90 · Post by **demetri90** » 20 Sep 2013 22:30

Since this topic has been solved, I'll continue discussion over on the one I created.

viewtopic.php?f=74&t=5028

XigmaNAS

[SOLVED] Transfer speed not fast - Teracopy problem

[SOLVED] Transfer speed not fast - Teracopy problem

Re: Transfer speed not fast then 35 MB/s

Re: Transfer speed not fast then 35 MB/s

Re: Transfer speed not fast then 35 MB/s

Re: Transfer speed not fast then 35 MB/s

Re: Transfer speed not fast then 35 MB/s

Re: Transfer speed not fast then 35 MB/s

Re: Transfer speed not fast then 35 MB/s

Re: Transfer speed not fast then 35 MB/s

Re: Transfer speed not fast then 35 MB/s

Re: Transfer speed not fast then 35 MB/s

Re: Transfer speed not fast then 35 MB/s

Re: Transfer speed not fast then 35 MB/s

Re: Transfer speed not fast then 35 MB/s

Re: Transfer speed not fast then 35 MB/s

Re: Transfer speed not fast then 35 MB/s

Re: Transfer speed not fast then 35 MB/s

Re: Transfer speed not fast then 35 MB/s

Re: Transfer speed not fast then 35 MB/s

Re: Transfer speed not fast then 35 MB/s

Re: Transfer speed not fast then 35 MB/s

Re: [SOLVED] Transfer speed not fast - Teracopy problem