*New 12.1 series Release:
2020-04-17: XigmaNAS 12.1.0.4.7542 - released

*New 11.3 series Release:
2020-04-16: XigmaNAS 11.3.0.4.7538 - released!


We really need "Your" help on XigmaNAS https://translations.launchpad.net/xigmanas translations. Please help today!

Producing and hosting XigmaNAS costs money. Please consider donating for our project so that we can continue to offer you the best.
We need your support! eg: PAYPAL

Centralized editing storage, Looking for feedback...

Show off that XigmaNAS system!
Forum rules
Set-Up GuideFAQsForum Rules
Post Reply
kjbuente
Starter
Starter
Posts: 30
Joined: 06 Jan 2016 04:21
Status: Offline

Centralized editing storage, Looking for feedback...

#1

Post by kjbuente »

This is currently a work in progress, I'll be posting images as I build this system...

I'm building a system for ten editing machines to have centralized storage. The plan is to have the editors to work directly from this storage instead of local disks. These editing machines will have 10Gb network cards installed and will be divided between a dual port 10Gb card due to the network topography. My plan is to have the network be my bottleneck as that is easier to expand in the future.

Planned System Specs:
24x 6TB WD Red Pro (RaidZ2, 4 VDevs)
SuperMicro MBD-X10SRL-F Motherboard
Intel Xeon E5-1650v2 Processor (3.5Ghz Six-Core with HT)
3x IBM M1015 flashed to IT mode
64GB DDR4 RAM (Maybe 128GB)
Intel 750 400GB SSD for L2ARC
2x Intel X540-T2 10Gb NICs
NAS4Free of course! (I have looked at commercial software\systems, but I honestly like NAS4Free better)
The chassis currently in use, so I can't check the model. I do know it is a rebranded SuperMicro 24bay.

I'll be posting pictures of the build progress here. I'm also hoping for some feedback on what everyone thinks. You might know of a problem I have not thought of with NAS4Free or on the hardware.

So what do you think?
SuperMicro 846 Chassis + 1200W Redundant PSUs + "A" model Backplane , SuperMicro X10SRL-F Motherboard, 512GB ECC RAM, Xeon E5 1650v3, 24*8TB WD Red Pro, 4 RaidZ2 Vdevs, 2 IBM M1015 Cross Flash to IT Mode, 2 IBM 46M0997 SAS Expanders, Dual port 10Gb Intel X560. RootOnZFS.

SuperMicro 846 Chassis + 1200W Redundant PSUs + SAS3-846-EL2 Backplane , SuperMicro X10SRL-F Motherboard, 512GB ECC RAM, Xeon E5 1650v3, 24*SanDisk Lightening 800GB 12Gbps SSDs, 12 Mirror vDevs, 4 Intel Optane 32GB NVMe drives (SLOG),Dual port 40Gb Intel XL710. RootOnZFS.

kjbuente
Starter
Starter
Posts: 30
Joined: 06 Jan 2016 04:21
Status: Offline

Re: Centralized editing storage, Looking for feedback...

#2

Post by kjbuente »

Current Project status:

10x WD RED Pro 6TB. 5 More on the way in.
4x IBM m1505 in IT mode, The fourth is a spare.
2x Intel x540-T2 10Gb NIC, one not in picture...

Arriving Monday: Motherboard and CPU
You do not have the required permissions to view the files attached to this post.
SuperMicro 846 Chassis + 1200W Redundant PSUs + "A" model Backplane , SuperMicro X10SRL-F Motherboard, 512GB ECC RAM, Xeon E5 1650v3, 24*8TB WD Red Pro, 4 RaidZ2 Vdevs, 2 IBM M1015 Cross Flash to IT Mode, 2 IBM 46M0997 SAS Expanders, Dual port 10Gb Intel X560. RootOnZFS.

SuperMicro 846 Chassis + 1200W Redundant PSUs + SAS3-846-EL2 Backplane , SuperMicro X10SRL-F Motherboard, 512GB ECC RAM, Xeon E5 1650v3, 24*SanDisk Lightening 800GB 12Gbps SSDs, 12 Mirror vDevs, 4 Intel Optane 32GB NVMe drives (SLOG),Dual port 40Gb Intel XL710. RootOnZFS.

User avatar
Parkcomm
Advanced User
Advanced User
Posts: 388
Joined: 21 Sep 2012 12:58
Location: Australia
Status: Offline

Re: Centralized editing storage, Looking for feedback...

#3

Post by Parkcomm »

I think you are bit of a pioneer - I'm not expecting anyone on the forum to have a more demanding application or better specified system to do it. I've never seen a (NAS) system with enough speed for video editing across the network, so I'm interested in seeing how you go.

I can't see anything wrong with the system, but I think you might be light on RAM.

With 24 disks bandwidth with be great, but iops will be limited by what you have in the L2ARC/ARC. 400G is a big L2ARC for file storage but not huge when you have ten video editors. However be aware that the SSD consumes RAM (each block in l2arc, needs 380 Bytes in arc) so you can end up with not much arc left if you use a big L2ARC.

Unfortunately the amount of RAM is data dependent, so you probably won't be able work out if this is an issue until you go live.
NAS4Free Embedded 10.2.0.2 - Prester (revision 2003), HP N40L Microserver (AMD Turion) with modified BIOS, ZFS Mirror 4 x WD Red + L2ARC 128M Apple SSD, 10G ECC Ram, Intel 1G CT NIC + inbuilt broadcom

User avatar
ChriZathens
Forum Moderator
Forum Moderator
Posts: 797
Joined: 23 Jun 2012 09:14
Location: Athens, Greece
Contact:
Status: Offline

Re: Centralized editing storage, Looking for feedback...

#4

Post by ChriZathens »

I hope that Intel® Xeon® Processor E5-1650 v2 is a typo and you mean v3, because v2 is a socket 2011 cpu and does not match your motherboard (2011-3).
My Nas
  1. Case: Fractal Design Define R2
  2. M/B: Supermicro x9scl-f
  3. CPU: Intel Celeron G1620
  4. RAM: 16GB DDR3 ECC (2 x Kingston KVR1333D3E9S/8G)
  5. PSU: Chieftec 850w 80+ modular
  6. Storage: 8x2TB HDDs in a RaidZ2 array ~ 10.1 TB usable disk space
  7. O/S: XigmaNAS 11.2.0.4.6625 -amd64 embedded
  8. Extra H/W: Dell Perc H310 SAS controller, crosflashed to LSI 9211-8i IT mode, 8GB Innodisk D150SV SATADOM for O/S

Backup Nas: U-NAS NSC-400, Gigabyte MB10-DS4 (4x4TB Seagate Exos disks in RaidZ configuration - 32GB RAM)

kjbuente
Starter
Starter
Posts: 30
Joined: 06 Jan 2016 04:21
Status: Offline

Re: Centralized editing storage, Looking for feedback...

#5

Post by kjbuente »

Parkcomm wrote:I think you are bit of a pioneer - I'm not expecting anyone on the forum to have a more demanding application or better specified system to do it. I've never seen a (NAS) system with enough speed for video editing across the network, so I'm interested in seeing how you go.

I can't see anything wrong with the system, but I think you might be light on RAM.

With 24 disks bandwidth with be great, but iops will be limited by what you have in the L2ARC/ARC. 400G is a big L2ARC for file storage but not huge when you have ten video editors. However be aware that the SSD consumes RAM (each block in l2arc, needs 380 Bytes in arc) so you can end up with not much arc left if you use a big L2ARC.

Unfortunately the amount of RAM is data dependent, so you probably won't be able work out if this is an issue until you go live.
Pioneer? That makes me a little nervous... I honestly feel NAS4Free can handle it. I've done tests with two editors utilizing iSCSI with a four port gig card to do MPIO. My variable on this machine will be samba and 10G. On RAM, I do want to start with a minimum of 128GB of RAM. Eventually max the motherboard out to 512GB. The 400GB L2Arc is large for now, I'm going to partition it so I'm using an appropriate size with the RAM installed. The Intel 750 series drives have the best IOPS I've seen available in that price range. (Anyone know of any better? I am open to suggestions as the L2Arc probably be the last thing purchased). I'm curious on what in I'm going to get with sequential read\writes.
Last edited by kjbuente on 04 Jun 2016 03:52, edited 3 times in total.
SuperMicro 846 Chassis + 1200W Redundant PSUs + "A" model Backplane , SuperMicro X10SRL-F Motherboard, 512GB ECC RAM, Xeon E5 1650v3, 24*8TB WD Red Pro, 4 RaidZ2 Vdevs, 2 IBM M1015 Cross Flash to IT Mode, 2 IBM 46M0997 SAS Expanders, Dual port 10Gb Intel X560. RootOnZFS.

SuperMicro 846 Chassis + 1200W Redundant PSUs + SAS3-846-EL2 Backplane , SuperMicro X10SRL-F Motherboard, 512GB ECC RAM, Xeon E5 1650v3, 24*SanDisk Lightening 800GB 12Gbps SSDs, 12 Mirror vDevs, 4 Intel Optane 32GB NVMe drives (SLOG),Dual port 40Gb Intel XL710. RootOnZFS.

kjbuente
Starter
Starter
Posts: 30
Joined: 06 Jan 2016 04:21
Status: Offline

Re: Centralized editing storage, Looking for feedback...

#6

Post by kjbuente »

ChriZathens wrote:I hope that Intel® Xeon® Processor E5-1650 v2 is a typo and you mean v3, because v2 is a socket 2011 cpu and does not match your motherboard (2011-3).

Umm, yea a typo. Let's go with that...

It's actually ment for a ESXi host,.I was hoping to "borrow" it, they are not cheap!
SuperMicro 846 Chassis + 1200W Redundant PSUs + "A" model Backplane , SuperMicro X10SRL-F Motherboard, 512GB ECC RAM, Xeon E5 1650v3, 24*8TB WD Red Pro, 4 RaidZ2 Vdevs, 2 IBM M1015 Cross Flash to IT Mode, 2 IBM 46M0997 SAS Expanders, Dual port 10Gb Intel X560. RootOnZFS.

SuperMicro 846 Chassis + 1200W Redundant PSUs + SAS3-846-EL2 Backplane , SuperMicro X10SRL-F Motherboard, 512GB ECC RAM, Xeon E5 1650v3, 24*SanDisk Lightening 800GB 12Gbps SSDs, 12 Mirror vDevs, 4 Intel Optane 32GB NVMe drives (SLOG),Dual port 40Gb Intel XL710. RootOnZFS.

User avatar
Parkcomm
Advanced User
Advanced User
Posts: 388
Joined: 21 Sep 2012 12:58
Location: Australia
Status: Offline

Centralized editing storage, Looking for feedback...

#7

Post by Parkcomm »

I wouldn't be nervous about being a pioneer - it's just that you application is quite niche and quite demanding.

With that many disks Sequential reads will probably max out your 10g link

How are you configuring the vdevs?



Sent from my foam - stupid auto correct.
NAS4Free Embedded 10.2.0.2 - Prester (revision 2003), HP N40L Microserver (AMD Turion) with modified BIOS, ZFS Mirror 4 x WD Red + L2ARC 128M Apple SSD, 10G ECC Ram, Intel 1G CT NIC + inbuilt broadcom

kjbuente
Starter
Starter
Posts: 30
Joined: 06 Jan 2016 04:21
Status: Offline

Re: Centralized editing storage, Looking for feedback...

#8

Post by kjbuente »

I'm thinking four vdevs in raidz2, but I'm not 100% sure on that yet.
SuperMicro 846 Chassis + 1200W Redundant PSUs + "A" model Backplane , SuperMicro X10SRL-F Motherboard, 512GB ECC RAM, Xeon E5 1650v3, 24*8TB WD Red Pro, 4 RaidZ2 Vdevs, 2 IBM M1015 Cross Flash to IT Mode, 2 IBM 46M0997 SAS Expanders, Dual port 10Gb Intel X560. RootOnZFS.

SuperMicro 846 Chassis + 1200W Redundant PSUs + SAS3-846-EL2 Backplane , SuperMicro X10SRL-F Motherboard, 512GB ECC RAM, Xeon E5 1650v3, 24*SanDisk Lightening 800GB 12Gbps SSDs, 12 Mirror vDevs, 4 Intel Optane 32GB NVMe drives (SLOG),Dual port 40Gb Intel XL710. RootOnZFS.

User avatar
Parkcomm
Advanced User
Advanced User
Posts: 388
Joined: 21 Sep 2012 12:58
Location: Australia
Status: Offline

Re: Centralized editing storage, Looking for feedback...

#9

Post by Parkcomm »

This might be of use http://blog.delphix.com/matt/2014/06/06 ... ipe-width/

You have an interesting challenge here - with ten users you are going to want decent IOPs as well as sequential, when multiple users access the system at the same time. If you only had a single access at a time you could go for (say) four groups of 6 raid-z2, but the wider the stripe the slower the iops.

The best iops will be mirrors, with the same capacity overhead as the 4 disk raid-z2. You have to do the maths on the likelihood of two disks in a pair failing (I'm guessing not as bad as you think.)

Then there is the effectiveness of the ARC/L2ARC, which would solve the iops issue for assets already cached, but if the video files are big enough, they'll be continuously overwriting the cache.

I'm looking forward to seeing how this goes
NAS4Free Embedded 10.2.0.2 - Prester (revision 2003), HP N40L Microserver (AMD Turion) with modified BIOS, ZFS Mirror 4 x WD Red + L2ARC 128M Apple SSD, 10G ECC Ram, Intel 1G CT NIC + inbuilt broadcom

kjbuente
Starter
Starter
Posts: 30
Joined: 06 Jan 2016 04:21
Status: Offline

Re: Centralized editing storage, Looking for feedback...

#10

Post by kjbuente »

My plan is to actually start with my four vdevs with 6 disks per and see how that does. It is a challenge of finding the balance between Space, Sequential, and IOPs. As one is almost as important as the other. If I have to move the data off while I redo the pool, no problem. have a 54TB system I can do that with while I reconfigure, though it is a bit slower.

The reason I'm using this configuration to start with is... Not all users will be using it all at once. Most of the time it's 5-8 users logged in at any given time. About half that doing any editing. Plus each half of the 10 machines will be sharing a 10Gb link, so that is a bottleneck. (For now). Also the video we are editing is a mix of H.264 and MPEG2, mostly H.264 though. The daily total of video edited is around 512GB. Between 128GB RAM, L2Arc, and local scratch disks. I feel comfortable staring here. I honestly feel that maxing the the motherboard in RAM is the first thing I need to do before redoing my pool configuration. This is also with 720P and 1080P videos right now. Once we start 4k, time to reevaluate.
SuperMicro 846 Chassis + 1200W Redundant PSUs + "A" model Backplane , SuperMicro X10SRL-F Motherboard, 512GB ECC RAM, Xeon E5 1650v3, 24*8TB WD Red Pro, 4 RaidZ2 Vdevs, 2 IBM M1015 Cross Flash to IT Mode, 2 IBM 46M0997 SAS Expanders, Dual port 10Gb Intel X560. RootOnZFS.

SuperMicro 846 Chassis + 1200W Redundant PSUs + SAS3-846-EL2 Backplane , SuperMicro X10SRL-F Motherboard, 512GB ECC RAM, Xeon E5 1650v3, 24*SanDisk Lightening 800GB 12Gbps SSDs, 12 Mirror vDevs, 4 Intel Optane 32GB NVMe drives (SLOG),Dual port 40Gb Intel XL710. RootOnZFS.

User avatar
Parkcomm
Advanced User
Advanced User
Posts: 388
Joined: 21 Sep 2012 12:58
Location: Australia
Status: Offline

Re: Centralized editing storage, Looking for feedback...

#11

Post by Parkcomm »

Sounds well thought out.
NAS4Free Embedded 10.2.0.2 - Prester (revision 2003), HP N40L Microserver (AMD Turion) with modified BIOS, ZFS Mirror 4 x WD Red + L2ARC 128M Apple SSD, 10G ECC Ram, Intel 1G CT NIC + inbuilt broadcom

kjbuente
Starter
Starter
Posts: 30
Joined: 06 Jan 2016 04:21
Status: Offline

Re: Centralized editing storage, Looking for feedback...

#12

Post by kjbuente »

Parkcomm wrote:Sounds well thought out.
I thought out a lot of things before, does not mean that all of them were good ideas....
SuperMicro 846 Chassis + 1200W Redundant PSUs + "A" model Backplane , SuperMicro X10SRL-F Motherboard, 512GB ECC RAM, Xeon E5 1650v3, 24*8TB WD Red Pro, 4 RaidZ2 Vdevs, 2 IBM M1015 Cross Flash to IT Mode, 2 IBM 46M0997 SAS Expanders, Dual port 10Gb Intel X560. RootOnZFS.

SuperMicro 846 Chassis + 1200W Redundant PSUs + SAS3-846-EL2 Backplane , SuperMicro X10SRL-F Motherboard, 512GB ECC RAM, Xeon E5 1650v3, 24*SanDisk Lightening 800GB 12Gbps SSDs, 12 Mirror vDevs, 4 Intel Optane 32GB NVMe drives (SLOG),Dual port 40Gb Intel XL710. RootOnZFS.

User avatar
b0ssman
Forum Moderator
Forum Moderator
Posts: 2454
Joined: 14 Feb 2013 08:34
Location: Munich, Germany
Status: Offline

Re: Centralized editing storage, Looking for feedback...

#13

Post by b0ssman »

make sure the backplane on the case is not an expander.
Nas4Free 11.1.0.4.4517. Supermicro X10SLL-F, 16gb ECC, i3 4130, IBM M1015 with IT firmware. 4x 3tb WD Red, 4x 2TB Samsung F4, both GEOM AES 256 encrypted.

kjbuente
Starter
Starter
Posts: 30
Joined: 06 Jan 2016 04:21
Status: Offline

Re: Centralized editing storage, Looking for feedback...

#14

Post by kjbuente »

Well, there has been a few small changes to the design. I found out that the E5-1600v3 series CPU apperently does not support LRDIMM, so I'm maxed out at 256GB of RAM for now. The E5-2600v3 series is just too expensive for the 3GHz+ clock speed, I do have a budget after all... I also decided to go to only one dual port 10Gb NIC instead of two. I thought it would be good to have for expansion, but space is getting tight on this board. The two ports remaining will be in LACP to load balance between them instead of a 50/50 split of clients via cabling.

The motherboard and a few more drives arrived. Picture hopefully tomorrow...


b0ssman, in the current setup I can see that there is a expander installed in a expansion slot on the motherboard. So I'm hopeful that the backplane does not have one installed, but there is no way to know until I go to try and use it.
SuperMicro 846 Chassis + 1200W Redundant PSUs + "A" model Backplane , SuperMicro X10SRL-F Motherboard, 512GB ECC RAM, Xeon E5 1650v3, 24*8TB WD Red Pro, 4 RaidZ2 Vdevs, 2 IBM M1015 Cross Flash to IT Mode, 2 IBM 46M0997 SAS Expanders, Dual port 10Gb Intel X560. RootOnZFS.

SuperMicro 846 Chassis + 1200W Redundant PSUs + SAS3-846-EL2 Backplane , SuperMicro X10SRL-F Motherboard, 512GB ECC RAM, Xeon E5 1650v3, 24*SanDisk Lightening 800GB 12Gbps SSDs, 12 Mirror vDevs, 4 Intel Optane 32GB NVMe drives (SLOG),Dual port 40Gb Intel XL710. RootOnZFS.

kjbuente
Starter
Starter
Posts: 30
Joined: 06 Jan 2016 04:21
Status: Offline

Re: Centralized editing storage, Looking for feedback...

#15

Post by kjbuente »

Well, many things got in the way of this build and caused delays. BUT it is running and mostly complete!

The backplane does have a expander, though it's replacement is on the way. For those who are wondering, the SAS-846EL1 from SuperMicro will work with 6TB SATA drives with the M1015 flashed in IT mode. (You never know who might want to know) I should have my SAS-846TQ backplane by the end of the week.

Once I get to my desktop I can post pictures later today!
SuperMicro 846 Chassis + 1200W Redundant PSUs + "A" model Backplane , SuperMicro X10SRL-F Motherboard, 512GB ECC RAM, Xeon E5 1650v3, 24*8TB WD Red Pro, 4 RaidZ2 Vdevs, 2 IBM M1015 Cross Flash to IT Mode, 2 IBM 46M0997 SAS Expanders, Dual port 10Gb Intel X560. RootOnZFS.

SuperMicro 846 Chassis + 1200W Redundant PSUs + SAS3-846-EL2 Backplane , SuperMicro X10SRL-F Motherboard, 512GB ECC RAM, Xeon E5 1650v3, 24*SanDisk Lightening 800GB 12Gbps SSDs, 12 Mirror vDevs, 4 Intel Optane 32GB NVMe drives (SLOG),Dual port 40Gb Intel XL710. RootOnZFS.

User avatar
ChriZathens
Forum Moderator
Forum Moderator
Posts: 797
Joined: 23 Jun 2012 09:14
Location: Athens, Greece
Contact:
Status: Offline

Re: Centralized editing storage, Looking for feedback...

#16

Post by ChriZathens »

Beware that the TQ backplane will cause a cable mess...
Not the easiest thing to cable manage 24 sata cables...!
Even if some are from a breakout cable.
My Nas
  1. Case: Fractal Design Define R2
  2. M/B: Supermicro x9scl-f
  3. CPU: Intel Celeron G1620
  4. RAM: 16GB DDR3 ECC (2 x Kingston KVR1333D3E9S/8G)
  5. PSU: Chieftec 850w 80+ modular
  6. Storage: 8x2TB HDDs in a RaidZ2 array ~ 10.1 TB usable disk space
  7. O/S: XigmaNAS 11.2.0.4.6625 -amd64 embedded
  8. Extra H/W: Dell Perc H310 SAS controller, crosflashed to LSI 9211-8i IT mode, 8GB Innodisk D150SV SATADOM for O/S

Backup Nas: U-NAS NSC-400, Gigabyte MB10-DS4 (4x4TB Seagate Exos disks in RaidZ configuration - 32GB RAM)

kjbuente
Starter
Starter
Posts: 30
Joined: 06 Jan 2016 04:21
Status: Offline

Re: Centralized editing storage, Looking for feedback...

#17

Post by kjbuente »

Don't laugh at the way the heatsink is mounted. Well you can actually, it's only temporarily...

And yes, the TQ backplane was not my first choice, I wanted the A model. Though I could not find one for the price of the TQ that I got for $100. My TQ backplane showed up this morning, so I'm waiting on breakout cables now.
I am impressed with the EL1 backplane, I'm getting over 900MB/s with it doing a scrub and around 500MB/s send\recv from my other system. The other system seems to be my limiting my speed, It's CPU is maxed out doing 4 ZFS sends at a time over SSH. I tried mbuffer and netcat, but they showed no benefit.

I am trying to figure out why I have a high temp alarm and why the fans are stuck at 100%...
You do not have the required permissions to view the files attached to this post.
SuperMicro 846 Chassis + 1200W Redundant PSUs + "A" model Backplane , SuperMicro X10SRL-F Motherboard, 512GB ECC RAM, Xeon E5 1650v3, 24*8TB WD Red Pro, 4 RaidZ2 Vdevs, 2 IBM M1015 Cross Flash to IT Mode, 2 IBM 46M0997 SAS Expanders, Dual port 10Gb Intel X560. RootOnZFS.

SuperMicro 846 Chassis + 1200W Redundant PSUs + SAS3-846-EL2 Backplane , SuperMicro X10SRL-F Motherboard, 512GB ECC RAM, Xeon E5 1650v3, 24*SanDisk Lightening 800GB 12Gbps SSDs, 12 Mirror vDevs, 4 Intel Optane 32GB NVMe drives (SLOG),Dual port 40Gb Intel XL710. RootOnZFS.

User avatar
mooblie
experienced User
experienced User
Posts: 88
Joined: 23 Jun 2012 13:31
Location: The Highlands, Scotland
Contact:
Status: Offline

Re: Centralized editing storage, Looking for feedback...

#18

Post by mooblie »

kjbuente wrote:........I'm getting over 900MB/s with it doing a scrub and around 500MB/s send\recv from my other system....
Good work. Is that really megabytes per second? Phew!
Martin

kjbuente
Starter
Starter
Posts: 30
Joined: 06 Jan 2016 04:21
Status: Offline

Re: Centralized editing storage, Looking for feedback...

#19

Post by kjbuente »

mooblie wrote:
kjbuente wrote:........I'm getting over 900MB/s with it doing a scrub and around 500MB/s send\recv from my other system....
Good work. Is that really megabytes per second? Phew!
Yep, MegaBYTES per second. I suspect that I'm being limited but not by much on the backplane.
SuperMicro 846 Chassis + 1200W Redundant PSUs + "A" model Backplane , SuperMicro X10SRL-F Motherboard, 512GB ECC RAM, Xeon E5 1650v3, 24*8TB WD Red Pro, 4 RaidZ2 Vdevs, 2 IBM M1015 Cross Flash to IT Mode, 2 IBM 46M0997 SAS Expanders, Dual port 10Gb Intel X560. RootOnZFS.

SuperMicro 846 Chassis + 1200W Redundant PSUs + SAS3-846-EL2 Backplane , SuperMicro X10SRL-F Motherboard, 512GB ECC RAM, Xeon E5 1650v3, 24*SanDisk Lightening 800GB 12Gbps SSDs, 12 Mirror vDevs, 4 Intel Optane 32GB NVMe drives (SLOG),Dual port 40Gb Intel XL710. RootOnZFS.

User avatar
Parkcomm
Advanced User
Advanced User
Posts: 388
Joined: 21 Sep 2012 12:58
Location: Australia
Status: Offline

Re: Centralized editing storage, Looking for feedback...

#20

Post by Parkcomm »

I was wondering how you were going - good work
NAS4Free Embedded 10.2.0.2 - Prester (revision 2003), HP N40L Microserver (AMD Turion) with modified BIOS, ZFS Mirror 4 x WD Red + L2ARC 128M Apple SSD, 10G ECC Ram, Intel 1G CT NIC + inbuilt broadcom

kjbuente
Starter
Starter
Posts: 30
Joined: 06 Jan 2016 04:21
Status: Offline

Re: Centralized editing storage, Looking for feedback...

#21

Post by kjbuente »

New backplane is installed, the old one was holding me back... New speeds with a scrub maxes out at 2.2GB/s and I'm still limited on the other side for send zfs streams by cpu.

I have moved a few editor users to the new system. They each max out at 300MB/s, this limitation is on the local machines. Not sure what is holding it back, don't know where to start looking. I do know that it is faster than their internal drives, so that makes them and me happy.

Some pics, others available on request if desired.
You do not have the required permissions to view the files attached to this post.
SuperMicro 846 Chassis + 1200W Redundant PSUs + "A" model Backplane , SuperMicro X10SRL-F Motherboard, 512GB ECC RAM, Xeon E5 1650v3, 24*8TB WD Red Pro, 4 RaidZ2 Vdevs, 2 IBM M1015 Cross Flash to IT Mode, 2 IBM 46M0997 SAS Expanders, Dual port 10Gb Intel X560. RootOnZFS.

SuperMicro 846 Chassis + 1200W Redundant PSUs + SAS3-846-EL2 Backplane , SuperMicro X10SRL-F Motherboard, 512GB ECC RAM, Xeon E5 1650v3, 24*SanDisk Lightening 800GB 12Gbps SSDs, 12 Mirror vDevs, 4 Intel Optane 32GB NVMe drives (SLOG),Dual port 40Gb Intel XL710. RootOnZFS.

User avatar
JoseMR
Hardware & Software Guru
Hardware & Software Guru
Posts: 1119
Joined: 16 Apr 2014 04:15
Location: PR
Contact:
Status: Offline

Re: Centralized editing storage, Looking for feedback...

#22

Post by JoseMR »

Gorgeous to my eyes, also get some stuff like shown below and get an OEM cable termination look like, a nice touch in any homebrew stuff.
You do not have the required permissions to view the files attached to this post.
System: FreeBSD 12 RootOnZFS Mirror, MB: Supermicro X8SI6-F, Xeon X3450, 16GB DDR3 ECC RDIMMs.
XigmaNAS RootOnZFS
Addons at GitHub
BastilleBSD
Boot Environments Intro
Resources Home Page

User avatar
zoon01
Developer
Developer
Posts: 794
Joined: 20 Jun 2012 21:06
Location: Netherlands
Contact:
Status: Offline

Re: Centralized editing storage, Looking for feedback...

#23

Post by zoon01 »

Congrads with this nice build!
I suspect you will enjoy it a lot :)
System specs: XigmaNAS 11.2.0.4 -embedded on Samsung 860 EVO 256GB and Supermicro X10SL7-F w / Bios v3.2, IPMI v.03.86 / CPU E3-1241 v3 @ 3.50GHz - 32GB Crucial DDR3L 1600mhz ECC 1.35v , LSI 2308 on PH20.00.07.00 IT mode, Storage: 5x Western Digital Red (WD30EFRX) raidz

Development system is same system in virtualbox.

kjbuente
Starter
Starter
Posts: 30
Joined: 06 Jan 2016 04:21
Status: Offline

Re: Centralized editing storage, Looking for feedback...

#24

Post by kjbuente »

@JoseMR: I actually have a large container full of those and many more. I usually save those for paying customers and not in-company setups.

@zoon01: I enjoy the build very much! I'd enjoy it much more once I can get Jumbo Frames enabled to get my full 10G speeds. Currently I set MTU in the web GUI and I get a boot loop due to a kernel panic. That'll be a post in the networking section.
SuperMicro 846 Chassis + 1200W Redundant PSUs + "A" model Backplane , SuperMicro X10SRL-F Motherboard, 512GB ECC RAM, Xeon E5 1650v3, 24*8TB WD Red Pro, 4 RaidZ2 Vdevs, 2 IBM M1015 Cross Flash to IT Mode, 2 IBM 46M0997 SAS Expanders, Dual port 10Gb Intel X560. RootOnZFS.

SuperMicro 846 Chassis + 1200W Redundant PSUs + SAS3-846-EL2 Backplane , SuperMicro X10SRL-F Motherboard, 512GB ECC RAM, Xeon E5 1650v3, 24*SanDisk Lightening 800GB 12Gbps SSDs, 12 Mirror vDevs, 4 Intel Optane 32GB NVMe drives (SLOG),Dual port 40Gb Intel XL710. RootOnZFS.

User avatar
Parkcomm
Advanced User
Advanced User
Posts: 388
Joined: 21 Sep 2012 12:58
Location: Australia
Status: Offline

Re: Centralized editing storage, Looking for feedback...

#25

Post by Parkcomm »

Great work.
NAS4Free Embedded 10.2.0.2 - Prester (revision 2003), HP N40L Microserver (AMD Turion) with modified BIOS, ZFS Mirror 4 x WD Red + L2ARC 128M Apple SSD, 10G ECC Ram, Intel 1G CT NIC + inbuilt broadcom

Post Reply

Return to “Gallery”