ZFS needs processing power

[waldo22]

I was having serious speed issues with my NAS4Free installation.

It was really unacceptably slow streaming over NFS. It was fast enough to allow me to stream an HD movie, but really nothing else.

This was with Gigabit Ethernet, too.

From my reading, it always sounded like RAM was the real bottleneck for ZFS, and my system has 8GiB, so I figured I was fine.

However, to save energy, I was using a 30 Watt TDP 1.8Ghz Celeron 430. Plenty, right?

Wrong.

When I replaced my first 3TB drive, it took 96 hours (!!!) to resilver.

When I replaced my 2nd, it also said it would take 96 hours, and it was on track to take that long, so I stopped it.

I replaced the 1.8Ghz Celeron 430 with an E7300 Core2 Duo at 2.66Ghz, and it took 26 hours!

I should have realized this was a problem when I first tried to copy my data over to my ZFS pool. It literally took 5 days to copy all the data over.

So, just a helpful tip for all of those new to ZFS, like I am.

Not only do you need plenty of RAM (at LEAST 4GiB), you also need enough processing horsepower to do the calculations or it just won't work.

-Wes

[Onichan]

That's not really surprising at all. Parity takes processing power and my phone has a better CPU then the one you initially had. As a note for others if your CPU sucks and you don't want to upgrade then just go with RAID10 as that doesn't use parity so you won't be CPU bottle-necked.

[b0ssman]

a celeron 430 is barly above atom in terms of performance.
i am not surprised.

and the 30 watt tdp for energy saving says nothing.
the idle power usage will be more important and i bet any modern sandy/ivy bridge will use less power while idle.

[ku-gew]

Do you have compression or dedup on?

[crowi]

no big news, ZFS is known to be hungry for resources, you should use at least a dual core cpu and plenty of RAM, what also increased the performance on my system, was adding a fast SSD for cache (L2ARC).
You need to decide if you want to have a system with high performance or low power consumption, both is not possible.
I am quite happy with the E-350 (18W TDP) solution, reading rates of >80MB/s and writing rates between 50-70 MB/s, that's a compromise I can live with.

[waldo22]

Do you have compression or dedup on?

No.

You need to decide if you want to have a system with high performance or low power consumption, both is not possible.

Well, that's relative, isn't it...

I decided on low power consumption over "high performance". I don't need "high performance", but being able to copy data at greater than 5MB/sec doesn't count as high performance in my book. That counts as "barely usable".

If I were trying to build a ZFS NAS system to serve an office with 20 users, I'd use an Ivy Bridge, or even Xeon or Opteron with 32GB of RAM or even more.

For a home server, because there were no guidelines that said otherwise, I figured 8GB of RAM and a semi-modern processor would be enough to serve 2 or 3 computers at once.

It can't. At least not with ZFS. And since it doesn't really say that anywhere, I'm saying it here.

It looks like an AMD E-class cpu or Core2 Duo/Sandy bridge may be a minimum requirement for decent ZFS performance.

Does anyone have any better guidelines than this?

For what it's worth, my E7300 Wolfdale Core2 Duo only uses marginally more power than the Celeron did.

[b0ssman]

You need to decide if you want to have a system with high performance or low power consumption, both is not possible.

well the power saving options on the latest generation cpus are really good so you dont have to choose anymore.

my rig has an intel i5 2500k for encryption (AES-NI) 8gb with 8 Sata Disks,
5 Fans and M1015 Contoller and sperate intel network card.

on boot the system needs 140W, Idle with disc running 77W and with discs not spinning 50W.

[ku-gew]

Well, Celeron 430 is not very modern:
http://www.cpubenchmark.net/low_end_cpus.html
it's 4 years old.
Still, I'm surprised. Have you checked if the cpu is used at max during resilver and copy?
It's a different machine, but my Xeon E3 3.1 GHz @1.6 GHz (due to power save) uses about definitely less than 5% CPU while transferring at 100 MB/s. If you have 1 cpu instead of 4 and 1.8 GHz instead of 1.6 GHz, you could use at most 20% cpu for a copy at 100 MB/s.
I have OpenIndiana and not FreeBSD here, ok, but FreeBSD is not 100 times slower! Something else is wrong in your setup.

[waldo22]

Well, Celeron 430 is not very modern:
http://www.cpubenchmark.net/low_end_cpus.html
it's 4 years old.
Still, I'm surprised. Have you checked if the cpu is used at max during resilver and copy?
It's a different machine, but my Xeon E3 3.1 GHz @1.6 GHz (due to power save) uses about definitely less than 5% CPU while transferring at 100 MB/s. If you have 1 cpu instead of 4 and 1.8 GHz instead of 1.6 GHz, you could use at most 20% cpu for a copy at 100 MB/s.
I have OpenIndiana and not FreeBSD here, ok, but FreeBSD is not 100 times slower! Something else is wrong in your setup.

Well, it certainly seems like something is wrong, but changing processors seems to have fixed it.

It's definitely older, that's why I called it "semi-modern"

Trying to run top was nearly impossible. In fact, ssh was so slow as to be nearly unusable. Typing a command would take up to 10 seconds to appear on the command line. (while re-silvering, I mean; normally it was fine)

How could it be anything other than the CPU?

I was really trying to use hardware that I had laying around, and I was able to do that with this Core 2 Duo (and the Celeron, too; I didn't go out and buy it for this...)

Do you think it's worth it, power savings wise, to purchase an E-350 or something similar?

-Wes

[ku-gew]

I really have no experience about processors... I used an Athlon 64 X2 for a NAS4free box because I had it at home and I used a Proliant ML110 G7 (Xeon E3-1220) for my personal NAS (illumos based: OpenIndiana but soon OmniOS) because it has ECC ram and it was cheap (380 Euro with shipment).

Have you tried to disable power saving on that celeron? maybe it was keeping a low frequency by mistake, and when max is 1.8 GHz, a energy save state could have as little as 1 GHz!

[Earendil]

Interesting determination, just swapping the CPUs helped with the resilver. Since resilvering is not using as much of the CPU as scrubbing does but is doing lots of data transfer instead, this is a bit surprising. Still, as Crowi said it should be no big surprise to those who know ZFS. But then as I've always said, NTFS is to DOS as ZFS is to NTFS and few people know much about any of these file systems.

You will generally see that NAS4Free will run on any hardware, I've heard of it running on an old 486! What's less said but should be is that ZFS shouldn't be used for anything below a GHz if you want Blu-Ray (yeck, Sony, bad taste!) transfer rates (<= 54 Mbps). For decent file transfers (>50 MBps, note this is BYTES, not bits) you need dual cores or more over 2 GHz. From your example Wes (thanks for sharing) and from my own (see signature below), you can now see the limits of running ZFS and it's number crunching requirements in terms of CPU usage.

Here's another reason ZFS is cool. Scrubbing a pool is similar to NTFS and DOS chkdsk. The one HUGE difference is you can access the data during a scrub! You can't during a chkdsk. Slow access is better than no access at all sometimes.

[waldo22]

Just a follow-up on this.

I replaced my setup with a new Ivy Bridge Pentium G2120 last week, and I recently had a drive failure and had to resilver again.

Here are the performance numbers for re-silvering pretty much the same amount of data when replacing a 3TiB drive:
1.8Ghz Celeron 430D: 96 hours
2.66Ghz Core2 Duo "Wolfdale" E7300: 26 hours
3.1Ghz Pentium G2120 "Ivy Bridge": 7 hours

...and it uses much less power. Whole system is at less than 100W now vs. ~140W before.

-Wes

[substr]

My first guess is that this is a CPU cache issue.

The celeron has 512KB cache and 800mhz FSB and one core.
The Core 2 Duo has a 3MB cache and 1066 FSB and two cores.
The G2120 has 3MB 'smart cache' and 1333/1600 memory and two cores.

ZFS was designed to run on Sun SPARC systems which generally have very large caches and multiple cores. It probably does very poorly when moving data around without a big cache; remember, it has all those trees, metadata, checksums, and it has to calculate checksums as data is loaded. With small caches, most of that is probably happening over the memory bus instead of out of the CPU cache.

ZFS is a large module bolted on to FreeBSD's kernel, so you've probably got another copy of the data in/out of the network stack, plus swapping the code itself in to the CPU caches.

In the end, this is probably about the size of your CPUs caches. Plus, a typical users will have compression on, since that is the default. That's another run of the data over the memory bus when your cache is too small.

[chrisf4lc0n]

I must agree with substr theory, as I have had an opportunity to test AMD E350 (1MB L2) Intel Celeron G1610 (512KB L2+ 2MB L3) with the same number of disks and configuration. Celeron, even though it has got less L2 cache, has got more cache all in total as the AMD E350 has not got L3 cache at all.

When I tested the NAS with Crystal Mark on the AMD CPU, it got to 80% load with 4 in a stripped mirror config and was only able to transfer about 70-80Mb/s.
The same disks with Celeron G1610 when also tested with Crystal Mark were maxing out my 1Gbit network showing transfers in the range 100-110Mb/s and the CPU was only under 20% load!

That means, it was no CPU speed which really mattered as for some reason full power of the E350 was not utilized anyway, there was something more to it which stopped the AMD CPU from performing at 100%!
I only wonder if sticking in some super fast RAM in my previous configuration would change anything, or the 1333MHz memory controller on the AMD CPU would still be bottlenecking everything?

[b0ssman]

Looking at the cpu mark that is not much of a surprise that the celeron is that much faster.

E350 -- 770
G1610 -- 2596

http://www.cpubenchmark.net/cpu.php?cpu ... Hz&id=1819

http://www.cpubenchmark.net/cpu.php?cpu ... 350&id=249

[Onichan]

I'd have to agree with b0ssman and I find it hard to believe the L cache makes that much a different. It seems most likely it is just a matter of CPU performance as shown in the benchmarks.