XigmaNAS

This is going to seem a like an odd request, but bear with me and you will see why I am asking:

I am in the process of moving from a RaidZ1 that was set up as a temporary home for files recovered from a failure (Ironically a failed RaidZ1, my fault for not noticing disks were dying, but it had been running a long time).

I have currently 6 x 3TB WD Red HDD's (but loking at getting 2/3 more in the near future having looked at available storage capacities, so probably wont setup anything until then).

When I set the original RaidZ1 up I used basically a setup guide and followed the considered wisdom that was prevalent around that time, nowadays I see advice like "creating a partition slightly under size for the zfs portion of the disk" this is to basically make swopping drives easier.

In this thread: viewtopic.php?f=66&t=9520#p58999 b0ssman talks about creating the disks with labels and how this helps avoid the problem that the OP is having.

I saw a thread from JoseMR stickied at the top of this forum, viewtopic.php?f=66&t=9522#p58982 this kind of thing is ideal. That is a much better base setup from which to run things imo.

So with bits like this in mind, I cannot seem to find a concise and comprehensive post anywhere on the best way of setting up ZFS.

I don't necessarily need advice on the differences of Z1,Z2,Z3 and so on, but best practice to setup a general OpenZFS system, for files (documents/media/backup vm storage/...maybe even vm hosting) but not really for database storage and that kind of thing (although I have read that performance in this regard can be improved). I don't need to much advice on hardware either, I have the gear, i know what i'm using, i know realtek nics are not recommended for performance purposes (mine are intel).

I need to know things such as:
Such as ideal disk setup (partition wise) for Z2/Z3.
What is the best way (for most people) to setup the boot disk
Multiple pools? Single pool?
When to enable compression if available and so on.

There are various guides floating about on the internet, but I am interested from a FreeBSD/nas4free standpoint and the opinions of its users in this regard. I'm willing to jump off the deep end and set it up in FreeBSD if needs be for messing about with, but once the data is on there it needs to be solid as a rock.

If there is an updated wiki post I have missed or anything just let me know.

Otherwise i'm going to be here another month still trying to compile a massive list from everywhere xD

Thanks.

Partition drives? Why?

Drive type I don't think really matters. Keep in mind your pool is only as strong / fast as the slowest. I have 3 to 5 year old WD greens that have been spinning 24 / 7 with no issues. Still, backup is critical to protect against hardware failures & physical perils such as fire, theft, flood.

Boot disk - use a cheap usb stick

^nighthawk^ wrote:Such as ideal disk setup (partition wise) for Z2/Z3.

Use whole disks - with glabels

^nighthawk^ wrote:What is the best way (for most people) to setup the boot disk

Use a separate disk for boot (from your ZFS array) - NAS4Free embedded works fantastic booting from a thumb drive.

^nighthawk^ wrote:Multiple pools? Single pool?

How long is a piece of string? -

The most secure solution would be two pools, using different hardware, in locations > 60km apart. The fastest would be lots of vdevs in parallel (striped). The lowest power should be a minimum count of large slow drives.

^nighthawk^ wrote:When to enable compression if available and so on.

Use the default Lz4 compression at all times.

^nighthawk^ wrote:jump off the deep end and set it up in FreeBSD

You don't have to lock your choice in - you can easily export your pool, boot from an alternative OS and then import - I run both NAS4Free and Freebsd, and sometimes Debian this way.

Thanks both. I get that i'm not being very precise and people need different setups for different needs. But small things like using glabels and so on and having that documented and recommended somewhere can really help.

Again many thanks, if anyone else wishes to contribute some more info to this thread that is greatly appreciated, i'll be looking over a lot of threads anyway for anything else that might be useful.

basically you set the labels before you create the zfs pool

then create the zfs pool using the labels. (dont forget about the 4k thing)

then sync the zfs config in the gui.

however far more important is to set up smart monitoring with email in my opinion.

b0ssman wrote:however far more important is to set up smart monitoring with email in my opinion.

Don't i know it!!

Out of the blue tips:

• If your pool is multi terrabyte do not use Raidz1 - use Raidz2 or Mirror
  There are good reasons to use mirror http://jrs-s.net/2015/02/06/zfs-you-sho ... not-raidz/
  Plan for growth (for instance will you add a new Raidz vdev, or swap and resilver every disk in the pool)
  Plan for snapshots and backups
  Use disks from different batches, within a vdev, to minimise the chance of simultaneous failure
  There are rules about number of devices in a pool - RAIDZ1 at at 3, 5, or 9, disks. RAIDZ2 at 4, 6, or 10 disks. RAIDZ3 at 5, 7, or 11 disks. You can probably ignore them. (if you use compression, you don't see any performance improvement, you might see some minimal improvement in overhead)
  Use LZ4 all the time on everything - even non compressible data
  Do not dedupe unless you REALLY know what you are doing
  You don't need a ZIL, you don't need L2ARC, do use as much (ECC) RAM as you can, do use a UPS

The last thing to say is ZFS is kind of black-boxish, and you might need to test some of these hard and fast rules for yourself.

As an example, I deduped one of my datasets (against all advice) and added an SSD for L2ARC (against all advice) and achieved:
dedup = 1.65, compress = 1.13, copies = 1.00, dedup * compress / copies = 1.86
without a perceivable performance hit

for compression:

if you are going to save only music / videos / other uncompressible files then you can leave the compression off.

I used to do that on a per dataset basis - but with Lz4, I don't notice any over-head, ever. And the benefit from more aggressive compression is not worth the performance hit.

So (fairly recently actually) I came to the conclusion that I can just leave Lz4 on all the time. One less thing to think about.

(but like I said above, its worth trying everything out for yourself)