RAIDZ2 parity load

[sodalimon]

Hi,

I noticed I have this idea about parity calculations / operations for a RAIDZ array:
More drives mean more parity calculations.
More calculations require more CPU power.
Or else, overall performance decreases.

Do you find this to be truthful?

I require a new stripe for my not-yet-operational 6-way RAIDZ2 pool (I'm not satisfied with the 7TiB it provides).
I'm intending to add another 6-way RAIDZ2 vdev to this pool (and another 6 later). I believe this will increase my IOPS too.
But I worry about the CPU performance, concerning the parity issues on my mind.

The actual reason to my doubts/ question is:
Why in the world can I get only 90-100 MiBps performance (read / write testing with 20GiB zero and random files to / from a RAM disk on the Windows 7 client side) over SMB/FTP...
a) While the NAS is able to do way more than that (I mean internal dd testing)...
b) While I can transfer 125MiB/s using when Windows 7 on both machines...

I believe the reason to be CPU power. The machine has 1600 MHz 16GiB RAM. But works on a Pentium G2020 (but seems like it will lose its chair to an i5).

On the other hand:
When I use the 6 drives as plain ZFS stripe -like a raid0 array- all I get is 5-10MiB/s more on FTP/SMB while there is a huge gain on dd tests (It should be less CPU intensive).
By the way, on RAIDZ2 usage, the CPU average is almost always below 45%.

Does ZFS limit the number of threads with number of actual cores?

And also, it seems clear that RAIDZ2 is not suitable with better IOPS. So in practical daily usage, it's natural to not observe the scores mentioned above (a lot worse in fact).
And I tend to believe, adding more stripes (vdevs) to the zpool will increase my IOPS as well as my throughput with hopes to increase my transfer rates.
So if someone doesn't stop me, I'll be carrying 18 drives on my back just for the sake of superstition but I know I'll definitely need a better CPU.

Confused, heh

This is why I have second thoughts on adding another vdev to the pool...

Conclusion:
I know I have a lot of unanswered questions in my mind about this whole ZFS universe.
After a couple of weeks' reading I got to acquire more questions than facts it seems.

Hope this is the right place to be asking my questions and I hope someone lends me a hand carrying those drives
Thanks in advance.

Regards,
Emre

[Onichan]

Are you saying you get ~100MB/s transfer speeds over CIFS to/from your NAS? If so then that is normal. CIFS is single threaded and not the most efficient process which is why currently it gets around 100MB/s. You can try tweaking it a little and maybe get it up to 110-115.

So adding more strips to your pool will not help in that aspect. Even then as you said using dd directly from the NAS gets you much higher speeds so you would hit the 1Gbps max throughput with your current setup if you exclude CIFS limits so adding a strip will net you no help in that regards. Now as you said it should give you more IO, but I am not sure why you would need that as that's not really the bottleneck for a home NAS.

[sodalimon]

Hi Onichan,
I'd like to thank you for your kind response...

I'm getting ~100MB/s over FTP and max ~90 MB/s over CIFS/SMB. Of course, using a sequential access from large single file (I know I've seen 20MB/s values with nasty tiny file collections)...
After reading your post; I'm now ready to accept CIFS/SMB as it is (I'll probably try tweaking later anyway). And I thank you for that.

Another question is:
Will I be able to maintain this level of performance after adding another 6wayRAIDZ2 (and another later) array to the pool, using this CPU?
Or is it going to be too big a burden for my it (or even for an ordinary Core i5) to handle?

If so, I'll choose to move to the 2-way mirror stripes approach (which is also way more flexible in my opinion). But then I'll have to provide a weekly backup / sync server for reliable redundancy. Cause I don't think a single mirror for stripes will satisfy my redundancy needs. Or maybe I'll attach secondary mirrors in time (Gosh! Way too expensive, both as number of disks and number of ports. Don't even want to think about Watts )...

If I may, another question...

Using a 6-way RAIDZ2, I worry about potential resilvering stress / resilvering times which would cause more fails.
And I think, attaching an additional disk to a problematic mirror for resilver is less stressing for disks. Do you agree with it?
Still not immune to both-disks-failing-at-the-same-time issue, but it sounds more flexible in that aspect as well.

Many pretty options. Hard to choose from

Thanks for you time...

Best Regards...
Emre

[Onichan]

I get around 110MB/s sustained on CIFS for large read/writes by upping the CIFS buffers a little. I have mine set to 131072 which did increase it from ~100 to the 110 for me. I tried twice that, but it didn't help much, but that's a quick and easy thing you can try to help a little.

As far as your CPU it seems like it is a fairly recent duel core processor so I think it should be more than enough for two 6 disk RAIDZ2 vdevs, I don't think it will hurt your performance. I currently run two 6 disk RAIDZ2 vdevs in a pool with a i5-3450 and writing large files I only see around 25-30% CPU usage. I am not sure how much of that is parity and how much CIFS, but I think your processor should still be fine.

For resilvering, a major point of RAIDZ2 is to protect again disk failure during rebuilds. It doesn't matter which redundancy method you use they all cause the same amount of stress on the disks as they all work by essentially just reading all data on the pool in order and either verifying the parity or making sure the mirror is good. The only difference is that RAIDZ 1 and 2 will just be more CPU intensive because of the parity instead of mirrors not using parity. So no I do not agree that one is better as far as disk stress goes, it only makes a difference if you have a crappy CPU. Now if you did have a single core CPU then a resilver might be seen as worse with RAIDZ2 because it would take a lot longer so you might be more likely to loose a disk during it as if it takes 3 days to verify.

[sodalimon]

Thanks a lot Onichan...

I'd tried increasing the CIFS buffers to 128k before. And nothing I could notice happened actually. Tried again now, and still maximum 90MB/s, generally evenly distributed around 85... In time, I'm sure I'll learn how to make my way through CIFS.

Depending on your explanations on CPU usage, I'm deprioritizing the purchase of the i5 3470 for now (until at least I need the AES-NI). I'll go for the new vdev(s) first.

Now I get what you mean about CPU dependency on resilvering. You've led me to see the resilvering process in a whole different aspect. I thank you for that as well... Will be trying to see differences through various trials soon...

Regards,
Emre

[Onichan]

As far as worrying about disk failures here is a good article that talks about it http://www.zdnet.com/blog/storage/why-r ... n-2009/162

Then here is a counter argument that says it isn't as bad as the first article stated http://www.high-rely.com/hr_66/blog/why ... -2009-not/

One thing to remember is if you are just doing a normal resilver then you assume none of your disks are currently failed. So for you to loose your pool with RAIDZ2 you would need to loose 3 disks in a row before the pool has finished scrubbing. So I think 6 disk RAIDZ2 is safe with 3 or 4TB drives still. The question is how large of a HDD does it take for it to start requiring RAIDZ3.

[sodalimon]

Thanks for the links Onichan... I'll need time to read thoroughly, it looks like there's important figures over there.
By the way, I've always thought I was exaggerating when considering the "probability of disk failure" for my calculations as 10 percent. It seems I was way too optimistic

Wish the production quality hadn't been suffering lately.
And I also hope you're right depending on 6-way RAIDZ2, cause I do depend on it too. Since I don't have the resources to build 3-way mirror stripes...

After your post which reminded me another point I came to find strategically important... How much percentage one lets one's pool (in this instance vdev) to fill up... Keeping too much unused space might be considered waste for some. But I think it kind of reduces the workload of the pool (vdev) when it's in harsh conditions

Even using regular NTFS on my clients and workstations, I've always found that adjusting partition sizes in order to keep at least half of them free was a good practice.

You've got me thinking in different perspectives, I'm so glad for your kind contribution...

Regards,
Emre

<I love this universe where drives live and die, where their music is flowing around us and it warms my heart to feel the heat of tens of drives spinning spinning spinning...>

[Onichan]

About the pool/vdev filling up, that depends on how you use the pool. I have read that it is not recommended to let the pool get over 80% full for performance reasons, which I think is because ZFS is a copy on write system. Though this would be more for a work environment where there is constant/frequent writes. For home usage this isn't really an issue because there's really no more than a handful of users with infrequent writes. I would still avoid filling it to 100%, but ZFS does reserve a little of the disk to prevent an issue when at 100%.

The problem with disk reliability is that it's measure in bytes/sectors read/written and is staying the same, but disk size keeps increasing. Meaning that as disks get larger you have a greater chance of failure if reading the entire disk. Though I think you are worrying too much about that as that is the whole point of dual parity. Disk will fail it is only a matter of time. I do recommend testing new disks first though. I do a 1 pass write read verify on all new disks before I start using them. So I at least know it won't fail right away as I did catch 1 bad disk that way.

Though the most important detail is RAID is NOT a backup. You should still have a 2nd copy of all important data on a totally different system. Personally I keep all my important stuff backed up to a 2nd standalone hard drive and backed up using online backup. While it would be quite bad if I lost my NAS it would only be stuff I could replace, though would take years.

[sodalimon]

About the pool/vdev filling up, that depends on how you use the pool. I have read that it is not recommended to let the pool get over 80% full for performance reasons, which I think is because ZFS is a copy on write system. Though this would be more for a work environment where there is constant/frequent writes. For home usage this isn't really an issue because there's really no more than a handful of users with infrequent writes. I would still avoid filling it to 100%, but ZFS does reserve a little of the disk to prevent an issue when at 100%.

Well, what I meant was different, though I experienced about the issue you mentioned first hand.
What I meant was, resilvering with a vdev that is not highly utilised is easier (faster), so less stressing. Especially considering today's large Terabyte drives. So my tendency to use several stripes is also an advantage, decreasing the data load on vdevs...

The problem with disk reliability is that it's measure in bytes/sectors read/written and is staying the same, but disk size keeps increasing. Meaning that as disks get larger you have a greater chance of failure if reading the entire disk. Though I think you are worrying too much about that as that is the whole point of dual parity. Disk will fail it is only a matter of time. I do recommend testing new disks first though. I do a 1 pass write read verify on all new disks before I start using them. So I at least know it won't fail right away as I did catch 1 bad disk that way.

I agree with this as well. Actually I do a month of test runs on new disks I buy before depending on them.
But I can say, the odds are high for me. For the latest 6 SV35 drives I bought, I had to have 3 replacements within the first week (the week I stress them most). All due to performance inconsistencies I observed during syntetic tests...
Yet, one of the replacements already seems to have started showing some weird SMART data.
I try to avoid single reseller and I also spread my purchase over the weeks, yet the result is clear.

Though the most important detail is RAID is NOT a backup. You should still have a 2nd copy of all important data on a totally different system. Personally I keep all my important stuff backed up to a 2nd standalone hard drive and backed up using online backup. While it would be quite bad if I lost my NAS it would only be stuff I could replace, though would take years.

I do. I keep important data (e.g my entire life) on distinct offline disks and on cloud as encrypted containers. But still not everthing fits there.
And replacable data is just, never-mind-I-do-not-have-enough-time-for-that-anymore data.
As you have said, no one would like to lose their entire NAS.

[Onichan]

Yeah I have never been a fan of Seagate. I have bought a couple of them many years ago and they always had problems. Ever since I have stuck with WD.

[sodalimon]

That was the same reason when I left using WD years ago
When I was selling computers years ago, I remember a time I preferred WD. At some point I noticed, I'd never had one for some time, that didn't end up with multiple bad sectors.
I believe, in years trends tend to fluctuate...

Also, the question is: what else would one expect from a consumer-grade hard disk drive...

I guess the reason for me using Seagate now is that it tends to fail within the guarentee period so I can have it a little longer (considering a replacement or full refund)

Maybe it's time to try those slow WD Reds...

By the way, I've been using three of these SV35's (early versions) for like a year. Two within a RAID-0 array on my desktop, one as an external backup drive at work. Without any problems so far. And they're fast too (and not crazily expensive). Knock on wood

About the replacements I had with the latest batch I bought...
They somehow displayed lower performance in tests (around 15 percent slower in average). Moreover they always encountered high ripples through benchmarks. (I have some screenshots if you're interested)
I didn't want to take any chances, so I got rid of them (since replacing is instant within the first one or two weeks of purchase)...
Not that, any one of them failed read / write tests or surface scans.

By the way, have you been using WD Blacks lately? I'd like to hear about their performance from you. And which models do you prefer in your NAS setups, REDs?

[Onichan]

I have never bought WD blacks as for a NAS of 5+ disks your LAN is going to be your bottleneck. I actually use 6 greens and 6 reds in mine and my old NAS was 8 greens, but I didn't think about comparing speeds on the reds vs greens before I added them to my pool. Doesn't really matter though as I said the LAN is the bottleneck. For my desktop I actually have been using blues for around 8 years, but a RAID0 of them. Have had 4 in RAID0 for around 3 years and 2 prior to that. Then got a SSD for system and apps almost a year ago.