A Poor man's DROBO?

[madgabz]

Please, this is a VERY interesting and relevant addition to the file system options: use NAS4Free as a way to reuse old hdds
maintaining the option of upgrading the drives, and/or adding drives.
In essence, this is a way to utilize random amount of harddisks of different sizes to get one big storage pool using ZFS, with a 1-disk fault tolerance (raid5-like). What needs to be done is a webgui frontend for managing this special setup of zfs pooling, and then code the (perhaps not so trivial) calculations of partition sizes etc. Especially handling adding and/or replacing harddisks will be essential, as this is far from trivial
It is already described here, a fantastic feast imho:
A Poor Man's Drobo

thanks in advance,
What do you think?

[SKL111]

It would be a nice feature, i did this myself so i could transition to bigger drives using n4f and ssh.

[madgabz]

Have you tried replacing/adding new disks yet? What is your experience?

[d0hboy]

agreed that this would be useful.. basically like synology's SHR or netgear's ReadyNAS X-RAID2. it was one of my critical decisions between this and a nas appliance.

[fsbruva]

Do you have any knowledge about why the referenced example created so many partitions, and only utilized raidz1? Another 100 GB could have been obtained if mirroring had been employed in the last two drives, and another 64 MB would have been obtained without the three 64 MB partitions.

I would think the hardest part is the determination of partition count/sizes. Darn it... now I see it as a puzzle.....

[fsbruva]

Figured out the partition sizing! This is a quick prototype in MATLAB:

Code: Select all

function [ parts ] = zparts( drives  )

%==========================================================================
%
%   Function:   zparts
%
%   Inputs:     drives      1 by N array of drive sizes (actual size)
%
%   Outputs:    parts       M by N array of partitions for the drives
%                           provided
%
%   Description:    
%   This function, given an arbitrary number of drive sizes, determines the
%   partition number and sizes to maximize usable disk space. This is
%   accomplished by creating as large a partition as the smallest disk
%   across all the drives. Then, the smallest disk is removed from
%   circulation. If there is any remaining space on the next smallest disk,
%   then a partition of that size is created across all remaining disks,
%   and the next smallest disk is removed. This process is repeated until
%   only one disk has any remaining space on it. If the two largest disks
%   do not match any other disk size, then a mirrored set of partitions
%   will be created.
%
%   Author:     fsbruva
%   Date:       14 Dec 2012
%
%==========================================================================
% Determine the total number of drives
dcount = length( drives );
% Initialize the count of available drives to be all of them
availcount = dcount;
% Initialize the array of partitions to be 1 row, and as many columns as
% there are disks
parts = zeros( 1 , dcount );
% Sort the drives so that the drive at the lowest index is the smallest
drives = sort( drives );
% Initialize the array to track the remaining space on each drive
remaining_space = drives;
% Initialize the count of partitions to 0 (we haven't any yest)
pcount = 0;

% This loop repeats as long as we have two drives with available space
% (because we tolerate a mirror across the last two, rather than raid5)
while availcount > 1
    % There are enough disks, so we can add a partition
    pcount = pcount + 1;
    % The current partition size is equal to the size of the smallest
    % remaining space on any drive. In MATLAB, the min function ignores
    % array elements that are NaN. 
    next_partition = min( remaining_space );
    % This loop repeats for as many drives there are in total, because this
    % loop is building the array of partitions. Even if a drive has no more
    % space, it will have a partition size of 0 for slices that do exist on
    % the other drives.
    for k = 1:dcount
       % This if statement checks to see if the currently examined disk's
       % remaining space is greater than 0. In MATLAB, boolean comparisons 
       % with NaN as an operand will return false. This avoids assigning a
       % partition to a disk that has been removed from the running.
       if remaining_space( k ) > 0
           % The parts array keeps track of all the partitions on all
           % disks, so if the current disk (k) will house this particular
           % partition, we need to note it.
           parts( pcount , k ) = next_partition;
           % Now that we have added another partition to this disk, we need
           % to reduce the amount of available space on disk k.
           remaining_space(k) = remaining_space(k) - next_partition;
           % Now we have adjusted the amount of remaining space on disk k,
           % we need to see if it has enough space for any more partitions.
           % If the remainig space is 0, then obviously no.
           if remaining_space( k ) == 0
               % No other partitions can be added to this disk, so we note
               % that by setting its remaining space to NaN (not a number).
               % This has the added benefit of removing it from
               % consideration by the min function and boolean operators
               remaining_space( k ) = NaN;
               % Now that this disk is out of the running, we need to
               % update the availcount variable to reflect that
               availcount = availcount - 1;
           end % end of remaining_space == 0
       end % end of remaining_space > 0
    end % end of for loop
end % end of while loop
end % end of function

It maximizes storage, while still tolerating single drive failure. For the initial set of four drives, this function produces a list of partitions:
81918 81918 81918 81918
0 41564 41564 41564
0 0 80402 80402

This gives a total of 409,284 bytes - 80 GB more than the referenced post.

In this case, any drive added MUST be 80GB or larger, and any drive replacement must be equal to or larger than the drive it is replacing. This is because vdevs can't shrink. Or rather, the work needed to shrink vdevs is... painful.

[ku-gew]

I would leave all this only to cases where it is REALLY needed. Simplicity is the key to success (and to data integrity