Difficult decision, need advice..

[sunshine]

I have a performance problem, and I need some advice.

My SAS disk expansion is a Dell MD1000. I set my NAS4Free system up with a SAS HBA, and all the 15 1TB SATA disks as JBOD - not RAID, just JBOD.

Performance is sub-par. I see a max of about 320-350 MB/sec off 15 spindles.. I've isolated the issue to a compatibility problem between the MD1000 and anything that isn't a PERC 6/e RAID card. It REALLY wants a PERC 6/e, but I thought hey - it's just a SAS expander, go SAS and JBOD. BAD IDEA.

I did some benchmarks, and using a PERC 6/e with the MD1000, I see more than double the performance (benchmarked using raw dd reads of all drives concurrently. For the SAS HBA, the drives are presented to the OS directly. For the PERC 6/e, I had to config each drive as a RAID0 device). The numbers are VASTLY different - more than double the performance from the PERC 6/e.. Damn MD1000... what is it doing?!

The difficult decision is what to do about this. I really like the NAS4free software, it makes NAS easy etc.. so I want to continue using it.

I think it's obvious that using the MD1000 and getting only 320MB/sec is not good, and worth the pain of moving to the PERC6/e. I don't have the money to buy an new SAS expander/new tray of disk.

- Should I use the PERC6/e with 15 x RAID0 disks, with ZFS on top of that?
- Should I use the PERC6/e with 15 disks in hardware RAID6, and just run ext2/UFS on that?
- Or maybe ZFS on top of a RAID array, so I at least get the self-healing, compression, snapshots, etc?
- How about stripe-width on PERC? 128KB to align with ZFS?

Using the PERC6/e sucks, but I don't think I have any other (free) choice.. =(

I'd love some ideas or recommendations.

[b0ssman]

which sas hba did you try?
did you try updating the firmware of the md1000?

combining hardware raid and zfs is a bad idea. you will loose all the self healing capabilities.

a cheap alternative to the dell
http://unixsurplus.com/product/sgi-3u-m ... -jbod-good

[Lee Sharp]

I don't have the money to buy an new SAS expander/new tray of disk.

Really? You can get one for $700... http://www.amazon.com/DS-1500-External- ... B001OP1KBI Add the resale of your MD1000, and it gets quite cheap. (And I have used a lot of Norco stuff and I am very impressed with them.)

However, your problem may just be that port replication. The bandwidth of all 15 drives are squeezed down to 3 SATA2 ports, so you are looking at 9gig tops. Which is not a big deal as your spindle speed is maxed out at just over 2 gig...

[sunshine]

which sas hba did you try?
did you try updating the firmware of the md1000?

I used a Dell SAS 6gbps HBA (it's essentially a rebranded LSI 9200-8e). All firmware is latest/greated on cards, hard drives, and MD1000.

combining hardware raid and zfs is a bad idea. you will loose all the self healing capabilities.

Are you referring to ZFS on top of a RAID6 array of 15 drives? If so, I agree, but the PERC6/e does do its own patrol, scrub, etc.. maybe not as nice as ZFS, but it does do it to heal the data.

a cheap alternative to the dell
http://unixsurplus.com/product/sgi-3u-m ... -jbod-good

Yea, I'm looking at those as alternatives.. With shipping to Canada though, it's a lot of $$ for me.

[sunshine]

I don't have the money to buy an new SAS expander/new tray of disk.

Really? You can get one for $700... http://www.amazon.com/DS-1500-External- ... B001OP1KBI Add the resale of your MD1000, and it gets quite cheap. (And I have used a lot of Norco stuff and I am very impressed with them.)

However, your problem may just be that port replication. The bandwidth of all 15 drives are squeezed down to 3 SATA2 ports, so you are looking at 9gig tops. Which is not a big deal as your spindle speed is maxed out at just over 2 gig...

If anything, I'd find something that would take my bare drives like an old SGI Rackable. Still too expensive, and I'll be lucky to get $150 for the MD1000. Maybe another $150 on the Dell SAS HBA though, so there's that..

With the PERC6/e, I can easily hit 6gbps, which is nice. I think my theoretical limit is about 4x 3gbps, but I'm not sure how many lanes are being used, or how to tell.

[sunshine]

combining hardware raid and zfs is a bad idea. you will loose all the self healing capabilities.

I have an update.

I decided to do some experimenting, and have discovered that disks from the SAS card (JBOD disks, with RAIDZ2) work fine via the PERC card (as a bunch of RAID0 devices). I have a pool full of data on disks built on the SAS card, then removed it, installed the PERC, plugged in the same drives, and they work fine. I was surprised. Scrubs confirm the data is intact, and the mounted data behaves as normal as ever.

I understand that disk issues will be "masked" by the PERC card, but I also know issues show up in dmesg and should be able to be caught and addressed.

I'm really curious how/why using RAID0 via the PERC would be detrimental to self-healing. Can you elaborate?

[b0ssman]

interesting but you wont be able to monitor smart values that whay

[sunshine]

interesting but you wont be able to monitor smart values that whay

Yup - that seems to be the biggest disadvantage.. That said, for more than twice the performance, and with good backups, I'm thinking I'll go for it!

I'll update this thread if/when I have relevant new info.. like "shit all my data is gone" or "3 years in, no issues"

[sunshine]

interesting but you wont be able to monitor smart values that whay

Found a way!

Smartctl is smarter than I knew. It can talk to drives behind a raid controller. In my case, all I needed to do was load the mfip module with

Code: Select all

kldload mfip

, and then run

Code: Select all

smartctl -d sat -a /dev/passX

for each of my devices.

Unfortunately, the SMART monitoring web UI does not allow me to specify these "passX" devices. I'm going to add the module to loader.conf, bounce the box, and report back. Maybe they'll show up then, who knows. Will edit in 10 mins.

Edit: Bad news, even though the devices are there in /dev/ now, they are not visible in the "Disk Management" UI, and not able to be added for SMART polling. I may need to make a feature request.