Speed problem with new SSDs

[spinoza]

Hello,

Doing some tests with different SSDs in order to use them later as ZIL or L2ARC.

Usually I am creating a simple volume/dataset using a single SSD, and set the sync of the dataset to always.
then I launch some

dd if=/dev/zero of=/mnt/vol1/vol1_ds/output.tmp bs=128k count=6000 ; rm -f /mnt/vol1/vol1_ds/output.tmp

and watch the disk use with a "gstat" and see the final writing speed of the dd command.

Using a Vertex3, I can get up to 195MB/s (creating the 4k alignment by hand or simply using the WEB GUI doesn't seem to change a lot), when it is announced for up to 550MB/s

All the others I tried had awful transfer speed (from 6Mb/s to 67Mb/s):
- OCZ Vertex4
- Samsung 840 Pro Series 256GB SSD
- SanDisk SDSSDP-128G-G25 128GB
- Samsung 840 Pro Series 128GB.

Are they too recent to be used with nas4free or am I missing something in the process?
(Don't know if it's relevant but only the Vertex3 shows the temperature in the "Disk information" panel)

Note: They are plugged to some LSI2008 controllers. But if I recall correctly, it was the same when plugged on the motherboard sata ports.

[b0ssman]

something like this?

http://forums.servethehome.com/raid-con ... tible.html

[spinoza]

Today I stopped my tests with the SAS2008 cards and I tried to plug the SSDs directly on a motherboard sata II ports (set to AHCI).
Same process: I create a vdev with a single SSD. I create a volume. I create a dataset with Sync=always.

sandisk 128Gb

Code: Select all

filer3: ~ # gstat
dT: 1.001s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0| md0
    0      0      0      0    0.0      0      0    0.0    0.0| da0
    0      0      0      0    0.0      0      0    0.0    0.0| md1
    0      0      0      0    0.0      0      0    0.0    0.0| da0a
    0      0      0      0    0.0      0      0    0.0    0.0| ada0
    0      0      0      0    0.0      0      0    0.0    0.0| ada1
    1    704      0      0    0.0    470  30989    0.4   95.6| ada2
    0      0      0      0    0.0      0      0    0.0    0.0| ada3
    0      0      0      0    0.0      0      0    0.0    0.0| ufsid/5210289c950
a1dba

filer3: ~ # dd if=/dev/zero of=/mnt/vol1/vol1_nfs/output.tmp bs=128k count=6000 ; rm -f /mnt/vol1/vol1_nfs/output.tmp
6000+0 records in
6000+0 records out
786432000 bytes transferred in 24.082951 secs (32655134 bytes/sec)

vertex 4 128Gb

Code: Select all

filer3: ~ # gstat
dT: 1.001s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0| md0
    0      0      0      0    0.0      0      0    0.0    0.0| da0
    0      0      0      0    0.0      0      0    0.0    0.0| md1
    0      0      0      0    0.0      0      0    0.0    0.0| da0a
    2   1419      0      0    0.0    945  62374    0.3   93.0| ada0
    0      0      0      0    0.0      0      0    0.0    0.0| ada1
    0      0      0      0    0.0      0      0    0.0    0.0| ada2
    0      0      0      0    0.0      0      0    0.0    0.0| ada3
    0      0      0      0    0.0      0      0    0.0    0.0| ufsid/5210289c950
a1dba

filer3: ~ # dd if=/dev/zero of=/mnt/vol1/vol1_nfs/output.tmp bs=128k count=6000 ; rm -f /mnt/vol1/vol1_nfs/output.tmp
6000+0 records in
6000+0 records out
786432000 bytes transferred in 14.474889 secs (54330779 bytes/sec)

Vertex 3 240b

Code: Select all

filer3: ~ # gstat
dT: 1.001s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0| md0
    0      0      0      0    0.0      0      0    0.0    0.0| da0
    0      0      0      0    0.0      0      0    0.0    0.0| md1
    0      0      0      0    0.0      0      0    0.0    0.0| da0a
    0      0      0      0    0.0      0      0    0.0    0.0| ada0
    1   2870      0      0    0.0   1914 126328    0.9   87.2| ada1
    0      0      0      0    0.0      0      0    0.0    0.0| ada2
    0      0      0      0    0.0      0      0    0.0    0.0| ada3
    0      0      0      0    0.0      0      0    0.0    0.0| ufsid/5210289c950
a1dba

filer3: ~ # dd if=/dev/zero of=/mnt/vol1/vol1_nfs/output.tmp bs=128k count=6000 ; rm -f /mnt/vol1/vol1_nfs/output.tmp
6000+0 records in
6000+0 records out
786432000 bytes transferred in 6.152584 secs (127821420 bytes/sec)

I can understand for the vertex 3.
If I align the partition on 4k, maybe I will reach the Sata II theorical max throughput (will do the test later today).

But as with the HBA, the results on the motherboard for the Vertex4 and the Sandisk are strangely low.
I cannot find a logical explanation.
What am I missing?

[Lee Sharp]

If you look at the Intel documentation for their fast SSD drives, they say 550MB/s for SATA3 and around 220MB/s for SATA2. On spinning disks, SATA2 is fine, but on the faster SSDs you need SATA3 for the bandwidth. (Also, the file system will have some overhead as well.)

[spinoza]

Yes I agree with that. I was not expecting to go higher than 220/250MB/s on my SATA2 test.
And even if SATA2 was the bottleneck, I would still be very happy if I had 220MB/s with all my SSDs.

The problem here is that, whatever the controller (tested with SAS2008 and onboard intel AHCI) and the bandwitdth available (SATA2, SATA3), I get very low results with some SSDs.
For example, as you can see on my previous post, with the Sandisk 128Gb, it goes as low as 32MB/s for a brand new disk.

The only SSD that gets me decent (but not optimal) speed for the moment is the Vertex3 but it's EOL so I cannot validate my filers relying on this SSD.

I probably misconfigured something (tried everything I could, even changing all cables) or it will be solved by some update. I'm confident it will get solved but for the moment, I cannot make any sense out of it.
Searching.

[spinoza]

Today I tested on a completely different machine (Fujitsu, core i5, sata3, AHCI).
Same SSDs --> Same result

Vertex4:

Code: Select all

filer3: ~ # gstat
dT: 1.001s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0| md0
    0      0      0      0    0.0      0      0    0.0    0.0| cd0
    0      0      0      0    0.0      0      0    0.0    0.0| da0
    0      0      0      0    0.0      0      0    0.0    0.0| md1
    0      0      0      0    0.0      0      0    0.0    0.0| da1
    0      0      0      0    0.0      0      0    0.0    0.0| da2
    0      0      0      0    0.0      0      0    0.0    0.0| da3
    0      0      0      0    0.0      0      0    0.0    0.0| da4
    0      0      0      0    0.0      0      0    0.0    0.0| ada0
    1   1466      0      0    0.0    977  64483    0.3   93.4| ada1
    0      0      0      0    0.0      0      0    0.0    0.0| da4a
    0      0      0      0    0.0      0      0    0.0    0.0| ada0s1
    0      0      0      0    0.0      0      0    0.0    0.0| ada0s2
    0      0      0      0    0.0      0      0    0.0    0.0| ufsid/5210289c950
a1dba
    0      0      0      0    0.0      0      0    0.0    0.0| ntfs/System Reser
ved
    0      0      0      0    0.0      0      0    0.0    0.0| ntfs/System

filer3: ~ # dd if=/dev/zero of=/mnt/vol1/vol1_nfs/output.tmp bs=128k count=6000 ; rm -f /mnt/vol1/vol1_nfs/output.tmp
6000+0 records in
6000+0 records out
786432000 bytes transferred in 12.850552 secs (61198305 bytes/sec)

Sandisk 128G (created with advanced format)

Code: Select all

filer3: ~ # gstat
dT: 1.001s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0| md0
    0      0      0      0    0.0      0      0    0.0    0.0| cd0
    0      0      0      0    0.0      0      0    0.0    0.0| da0
    0      0      0      0    0.0      0      0    0.0    0.0| md1
    0      0      0      0    0.0      0      0    0.0    0.0| da1
    0      0      0      0    0.0      0      0    0.0    0.0| da2
    0      0      0      0    0.0      0      0    0.0    0.0| da3
    0      0      0      0    0.0      0      0    0.0    0.0| da4
    0      0      0      0    0.0      0      0    0.0    0.0| ada0
    1    521      0      0    0.0    348  22945    0.4   97.2| ada1
    1    521      0      0    0.0    348  22945    0.4   97.4| ada1.nop
    0      0      0      0    0.0      0      0    0.0    0.0| da4a
    0      0      0      0    0.0      0      0    0.0    0.0| ada0s1
    0      0      0      0    0.0      0      0    0.0    0.0| ada0s2
    0      0      0      0    0.0      0      0    0.0    0.0| ufsid/5210289c950
a1dba
    0      0      0      0    0.0      0      0    0.0    0.0| ntfs/System Reser
ved
    0      0      0      0    0.0      0      0    0.0    0.0| ntfs/System

filer3: ~ # dd if=/dev/zero of=/mnt/vol1/vol1_nfs/output.tmp bs=128k count=6000 ; rm -f /mnt/vol1/vol1_nfs/output.tmp
6000+0 records in
6000+0 records out
786432000 bytes transferred in 24.803846 secs (31706051 bytes/sec)

Sandisk 128G (without advanced format)

Code: Select all

filer3: ~ # gstat
dT: 1.001s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0| md0
    0      0      0      0    0.0      0      0    0.0    0.0| cd0
    0      0      0      0    0.0      0      0    0.0    0.0| da0
    0      0      0      0    0.0      0      0    0.0    0.0| md1
    0      0      0      0    0.0      0      0    0.0    0.0| da1
    0      0      0      0    0.0      0      0    0.0    0.0| da2
    0      0      0      0    0.0      0      0    0.0    0.0| da3
    0      0      0      0    0.0      0      0    0.0    0.0| da4
    0      0      0      0    0.0      0      0    0.0    0.0| ada0
    1    747      0      0    0.0    498  32839    0.4   96.6| ada1
    0      0      0      0    0.0      0      0    0.0    0.0| da4a
    0      0      0      0    0.0      0      0    0.0    0.0| ada0s1
    0      0      0      0    0.0      0      0    0.0    0.0| ada0s2
    0      0      0      0    0.0      0      0    0.0    0.0| ufsid/5210289c950
a1dba
    0      0      0      0    0.0      0      0    0.0    0.0| ntfs/System Reser
ved
    0      0      0      0    0.0      0      0    0.0    0.0| ntfs/System

filer3: ~ # dd if=/dev/zero of=/mnt/vol1/vol1_nfs/output.tmp bs=128k count=6000 ; rm -f /mnt/vol1/vol1_nfs/output.tmp
6000+0 records in
6000+0 records out
786432000 bytes transferred in 25.891251 secs (30374430 bytes/sec)

The SSDs were tested under Windows and seem to all display correct bandwidth (around 250MB/s for a non-aligned partition, just plugged, formatted, benchmarked)

So now the question is:
Do I have the wrong way of testing for speed? (I'm OK to hear it )
If not, then I guess I need to wait for an update of some kind.