*New 11.4 series Release:
2020-07-03: XigmaNAS 11.4.0.4.7633 - released!

*New 12.1 series Release:
2020-04-17: XigmaNAS 12.1.0.4.7542 - released


We really need "Your" help on XigmaNAS https://translations.launchpad.net/xigmanas translations. Please help today!

Producing and hosting XigmaNAS costs money. Please consider donating for our project so that we can continue to offer you the best.
We need your support! eg: PAYPAL

OrioniS' N4F Mark III - Asus M5A78L-M/USB3 uATX - update

Show off that XigmaNAS system!
Forum rules
Set-Up GuideFAQsForum Rules
Post Reply
OrioniS
Starter
Starter
Posts: 15
Joined: 08 May 2013 17:13
Location: London, UK
Contact:
Status: Offline

OrioniS' N4F Mark III - Asus M5A78L-M/USB3 uATX - update

#1

Post by OrioniS »

Little bit about history...

I had my first NAS4Free build about 1.5 years ago as an experiment and it was based on VIA C3 1GHz and 256MB memory (from another project - CarPC)... All specs are here:

CPU: VIA C3 @ 1GHz (passmark 100 Points)
MB: VIA EPIA M10000 ITX
RAM: 256MB DDR 266
HDD: Maxtor 80GB ATA (UFS)
CASE: No case
SYSTEM: UFS on system drive
POWER USAGE: 75W @ idle, 90W @ full usage
NIC: VIA VT6103 10/100

It worked without any problems, I had transfer at about max 8-9MB/sec (don't forget it was 10/100 LAN), so I plugged 10/100/1000 and it topped up at 25MB/sec... After this I was hooked on N4F ;-) Next version involved completely different approach as it was need to be production unit ;-) Experiment successful, but I need real thing now. Played with N4F, I knew what I wanted (at the time ;-). There was born Mark II:

CPU: G1610 @ 2.6GHz (Passmark 2553 points)
MB: Asus P8H61-I ITX
RAM: 6GB DDR3 1333 Non ECC
HDD: 2x Seagate 500GB 7200.11 (I know.. bad drives series... but I had no problem with), 2x Hitachi 500GB 7200.C
SYSTEM: Raid10 (4x500GB) ZFS
BOOT: Seagate Momentus 120GB 2.5" HDD
CASE: Master cooler 120 Elite
POWER USAGE: 45W @ with HDDs powered off, 55W @ idle, 75W @ full usage
NIC: Realtek 8111F

Image

Hmm... It was good for 3 HDDs, but with my 4.5, it was in cramped and top one was quite hot reaching 45 degrees sometimes. I start using ZFS and after 1 year I realised three major drawbacks with my setup:
1. Raid10 can ONLY handle 1 failure EACH side of mirror
2. Not sure, if mirroring is done straight at the controller OR with ZFS checking it. I didn't like the idea of doing it by controller only.
3. If I want to expand, case is the problem.
4. Non ECC memory. I realised that it was bulletproofing my setup, and my experience was saying one thing but my wallet the other ;-) After a year, I realised that ECC is NOT more expensive than non-ECC and I proved this buying 4 sticks for 25% of non ECC alternative later...
The SMB transfer was around 50 MB/s both read/write, using dataset with standard compression. CPU was at max 30% of usage most of the time.

So, there was need to upgrade/downgrade (pick your poison) my system and look for ECC compatible motherboard & CPU. Notable, after this purchase, I realised another truth: Intel is getting less expensive, if you going to buy better MB and CPU comparing to AMDs of similar specs.

CPU: AMD 210e @ 2.6GHz (Passmark 1419 Points)
MB: Asus M5A78L-M/USB3 uATX
RAM: 6GB DDR3 1333 ECC
HDD: 4x Seagate 500GB 7200.11 (I know.. bad drives series... but I had no problem with), 2x Hitachi 500GB 7200.C
SYSTEM: RaidZ2 (6x500GB) ZFS
BOOT: Seagate Momentus 120GB 2.5" HDD
CASE: Rackmount Logic Case - SC-33380B
POWER USAGE: 57-65W @ with HDDs powered off, 78-85W @ idle, 102-116W @ full usage
NIC: Realtek 8111E
PCI: Sil3112A 2xSATA for System bootup

Image

SMB transfer is at 70-75MB/s writing and 75-80 MB/s reading and I'm happy about. It's using Dataset with LZ4 compression and CPU usage is all over the place, from 0 to 90% in spikes, so I thought I got it nearly at minimum necessary specs. It could be better, but it's OK for now. I could activate one more core and it change to Phenom II x4 945, but I'm losing CPU temperature facility on screen. One more core and is is unstable completely. Also, 3rd core may be broken somehow... We will see...

As I mentioned before, for higher specs the price difference is not that different between AMD and Intel, especially if you want something like AES compatible CPUs and ECC altogether. For AMD is FX series on AM3 socket (DDR31333 support) OR Opteron AM3 series and for Intel, it's i5 cheapest but you need Cxxx series northbridge and this is costly plus there are not that many of them in UK. AMD FX-4300 is cheapest but is quite hot and power hungry, Opteron on AM3 is much better but expensive...

If you don't need ECC or AES than get yourself cheap 6x SATA port MB and slap G1610 on it. Cheap and cheerful and it works amazingly good. When You need ECC than it's getting complicated for both sides with AMD being more accessible & cheaper in short run as you can use almost any AMD AM3 (if you want DDR3) or AM2 (if you don't) CPU, but only AMD Asus has confirmed its motherboards to work with ECC/Non ECC memories. With the other, some yes, some no. Intel? needs Cxx northbridge to use ECC, even is CPU allows it. I think this is straight from Intel as they don't want anyone to use cheaper desktop motherboards on servers/workstations. Side note: my Z87 is reported as a C226 in some programs, so there is a clue... All MB seems to be at £165 with c224/c226 and up vs £49 for AMD here in UK, so your mileage may vary...
If you need AES than your choice is : AMD FX series (hot and hungry) or AMD Opteron AM3 (nice temps and kinda power efficient), LGA 1155, 1150 Intel i5, i7, Xeon... Cooler and better efficient than AMD and not more expensive than FX 4300 / Opteron 3320, with half of power envelope and you can use any MB for intel.
But... If you need ECC & AES... This is getting into bigger money because you need more expensive AMDs CPU vs Intel requirement for Cxx northbridge... where AMD will give you more control over motherboard (still Asus confirmed choice) and Intel more control over power efficiency and faster CPUs as per clock.

ITX vs uATX... Once you immerse yourself in N4F, there is no way you will stop at one point. Once you start to play with 4+ HDDs you want to get better cooling -> bigger case, so there is no point to get ITX and close yourself with that format. uATX is next in line. Bigger case mean more space, so HBA card on PCIe x8 will be next choice... By end of the year, you will have a uATX MB, 16GB ECC RAM, FX-4300, 6x 500GB RaidZ2, case able to take 6-9 HDDs, veeeery fast HBA card with 8x SATA ( forget about onboard, they are useless once you have good HBA)... and another £300 spend ;-)

...and there are always Christmas presents ;-)

SPECIFICATION UPDATE

6GB ECC @ 1333 memory replaced with Hynix 4x8GB (32GB) ECC 1600MHz but running @800MHz, see viewtopic.php?f=60&t=7561&p=44493#p44493 for more info.
Also, in search for 2 new hard drives... 2 out of 4 Barracudas dying...

Done some power consumption test via Kill-a-watt:

56W - disks down
3W - system off
173W max - when system start
105W - booting
88W - system ready
110W max - SMB file reading
115W max - SMB file writing

Wondering about changing 400W PSU for PICO 150-200W... It may be just fine...
Last edited by OrioniS on 27 Oct 2014 18:02, edited 2 times in total.
OS: NAS4Free 10.2.0.2.2268
MB: SUPERMICRO X9SCL-F
CPU: Celeron G1610 @2.6GHz
RAM: 32GB DDR3 @ 1333 ECC
HBA: 9211-8i/IT (LSI flashed M1015)
STORAGE: RAIDZ2 (4x Toshiba DT01ACA200 2TB, 2x WD RED 2TB)
LAN: 1x Mellanox-2 (10Gbit SFP+ desktop only), 2x Intel (1Gbit Ethernet, local lan)
BOOT: 32GB USB3.0 Stick
CASE: Fractal Design Node 804 Micro-ATX
POWER: Seasonic 550W

http://www.orionis-pm.co.uk/blog/

User avatar
b0ssman
Forum Moderator
Forum Moderator
Posts: 2454
Joined: 14 Feb 2013 08:34
Location: Munich, Germany
Status: Offline

Re: OrioniS' N4F Mark III

#2

Post by b0ssman »

some notes on the intel part

the haswell generation supports ecc on every model. even the cheapest celeron G1610.
you only need a motherboard with ecc support (c2xx range)
AES is supported from i3 (haswell) onwards.
the c2xx officially does not support the i5.

another important point. most c2xx motherboards come with some kind of ipmi support, which is very usefull for a headless nas setup.
Nas4Free 11.1.0.4.4517. Supermicro X10SLL-F, 16gb ECC, i3 4130, IBM M1015 with IT firmware. 4x 3tb WD Red, 4x 2TB Samsung F4, both GEOM AES 256 encrypted.

OrioniS
Starter
Starter
Posts: 15
Joined: 08 May 2013 17:13
Location: London, UK
Contact:
Status: Offline

Re: OrioniS' N4F Mark III - Intel NIC update

#3

Post by OrioniS »

I didn't paid full attention to my G1610 (and other Intels for ECC) soon as I saw Cxx need for motherboard. I know that Intel has more and more features comparing to AMD, but try to get ANY motherboard with Cxx chipset below £100 ($150) and you are good ;-) As for Haswell i3, it's really good to know, as you can get them quite "cheap" and in my opinion they are better suited than FX-4300 (as for ECC & AES) from AMD for NAS. I didn't knew about i5 and Cxx... Plainly stupid, Intel where is your brain?

Never used IPMI... Cannot comment on that.

But... My total price for moving to ECC was £26 for AMD 210e @ 2.6GHz and £49 for Asus motherboard with 6x SATA = £76 (+£21 for 6GB ECC I need it anyway). I know Intel is ah and oh, but any G1820 is about £30 and cheapest ASUS P9D or Asrock Rack E3C224-V+ LGA1150 are way above £150, so is over 2x more... even if someone is talking about efficiency or they will eat electricity, this £80 will take another 2-5 years depending on usage to pay off...

If intel's Cxx was half the price then AMD doesn't have a standing chance as Intel WILL blow them away (speed, efficiency and other bits), but at the moment, I choose AMD as my main engine due to the fact that is half cheaper and for my needs as (not that) simple NAS any CPU above 1500 points in Passmark is more than adequate to most of the NAS tasks. I know that: my G1610 never saw more than 30% in Raid10 & Compressed Dataset, my AMD 210e is half fast (or slow) of Intel's but it does all of it like the other AND is doing RaidZ2 parity. I'm doing 70-80 MB/s through SMB/CIFS, it's still better than I had Intel in there (and I wonder why). I'm waiting for Intel NIC to see if I can get more. If not, than I have Raid x8 card to play with...

So, if you on budget like me? AMD ALL THE WAY ;-) until Intel drop the prices or I will get more money. A side note: my G1610 & MB ended up in Media Player where is happy to play video, radio and other things together with NVidia 610 @ 45-65W depending on power supply (tested few)...

Few tests I've done now with transfer of 20GB TrueCrypt virtual drive file...

Reading from raidz2 with LZ4 compression
Realtek 8111E Image Intel 82574L Image

Reading from raidz2 without compression
Realtek 8111E Image Intel 82574L Image

Writing to raidz2 with LZ4 compression
Realtek 8111E Image Intel 82574L Image

Writing to raidz2 without compression
Realtek 8111E Image Intel 82574L Image

Still don't know why writing is higher than reading...


Whole mess ;-)
Image

And 2 new fans as it is hitting 42 degrees on left hand side hard drives (no fans)
Image
OS: NAS4Free 10.2.0.2.2268
MB: SUPERMICRO X9SCL-F
CPU: Celeron G1610 @2.6GHz
RAM: 32GB DDR3 @ 1333 ECC
HBA: 9211-8i/IT (LSI flashed M1015)
STORAGE: RAIDZ2 (4x Toshiba DT01ACA200 2TB, 2x WD RED 2TB)
LAN: 1x Mellanox-2 (10Gbit SFP+ desktop only), 2x Intel (1Gbit Ethernet, local lan)
BOOT: 32GB USB3.0 Stick
CASE: Fractal Design Node 804 Micro-ATX
POWER: Seasonic 550W

http://www.orionis-pm.co.uk/blog/

Post Reply

Return to “Gallery”