How to add a larger hard drive? Still don't get it...

[marqus]

Hi there,

I am running a freebsd based NAS for some years now. But all my set ups I never touched because of the fear of losing everything

My current system: 9.1.0.1 Sandstorm with 6 Samsung 1TB disks. 5 of which I put into a virtual device called vRaidZ1 and the 6th one I put into a virtual device called vSpare. There is one pool called deeppool with a max capacity of 4.53TB.
I creadted a Dataset called "media".

The NAS is 99% full and I desperately want to add larger drives to the pool, or better to the virtual disk. The system should just recognize the new disk an enlarge the pool size. I do not want to change all disks at once of course!

I would like to start with one 3TB disk which I bought new and just put it on the Sata connector where now lets say drive dev0 is connected to. A simple switching when the system is shut down.

Hopefully, the zfs filesystem can handle the new disk and add the 2 more TB to my NAS.

Is this possible at all? Is there a way to change disks to larger ones. One after another. I wouldnt mind rebuilding time, if it only would work.

Is there a howto for dummies?

thanks a lot,
Marqus

[ChriZathens]

You can change all your disks with larger ones, one by one.
But you will get the extra space only after changing all of them
To put simply:

• Raidz 6x1TB HDDs =~5TB
• Raidz 5x3TB + 1x1TB HDDs =~5TB (still)
• Raidz 6x3TB HDDs =~15TB

[marqus]

Oh my goodness...do you really mean it? Thats almost a financial desaster...how can anyone handle that scenario. I cannot just go an buy 6 drives, 120 dollars each...
and spending money now for 3TB, which i cannot use for the next months...?no way.

Is this really the only way?
And if so, how exactly is it done?

thx,
Markus

[kenZ71]

Try ebay for used drives? If they pass smart scans you are good to go.

[ChriZathens]

Oh my goodness...do you really mean it? Thats almost a financial desaster...how can anyone handle that scenario. I cannot just go an buy 6 drives, 120 dollars each...
and spending money now for 3TB, which i cannot use for the next months...?no way.

Is this really the only way?
And if so, how exactly is it done?

thx,
Markus

This is the way it is for any raid system. You can either expand the array, or grow it using bigger drives.
in case of zfs, if someone has e.g. a pool with a 3 disk raidz vdev, they can add another raidz vdev to the pool and expand the array. Openmediavault which is based on Linux and uses mdadm, supports adding one more drive and expanding the array in a more affordable way. But all raid systems, either using mdadm or zfs, or even hardware raid, will create an array which consists of the size of the smallest disk multiplied by the number of drives. So you will always need to replace all the drives with bigger ones in order to take advantage of the extra space.
AFAIK the only OS which supports what you ask is Unraid which uses a different logic to offer protection to the array.

P.S. In your case you can try something else: You currently have 6x1TB disks, about 4.5TB of space.
Buy only 3x3TB disks and create a raidz vdev with them and then a new pool. It will offer about 5.5 TB of space.
Copy your data to this new pool, and destroy the old one.
If your six drives are in good condition you can sell them and add some more money to buy 3 more drives.
You can then create another raidz vdev and add it to the existing pool, or create a new one which will offer another 5.5TB of space. OK, still not a cheap solution, but more affordable than buying 6x3TB disks at once...

[marqus]

Hi,

thx for the tip with the 2 times 3 disk scenario Maybe I try this. But why are there only 5,5TB of space using 3x3TB???

I really was under the impression that, when deciding whether to go with Win8 and its storage spaces (which can add disks just as if it were nothing...) or again with Nas4free, that nas4free can handle this as well. I thought that RaidZ is no RAID in its typical sense and does not suffer from the RAID limitations you mentioned above.

Are you 100% sure, that I cannot add just another bigger drive to my vdev and ec´vrything works fine?

thx again,
Markus

[ChriZathens]

Raidz is equivalent to raid5. Hence the disk space I mentioned: The third disk is for parity, so you only use 2x3TB for storage. 2x3TB equals about 5.5TB of usable space
You cannot add a disk to an existing vdev. You can only add a vdev to an existing pool. You can create a vdev consisting of a single 3TB disk and add it to your pool. This way you will gain additional 3TB. But DON'T do it.
Loss of the single disk will result to total loss of your pool and all your data!
For a better understanding of how zfs works, please read this excellent presentation:
http://forums.freenas.org/index.php?thr ... oobs.7775/

[marqus]

Ok, thank you. I will read the ppt file in your link. But again, what does the "replace" function in WebGUI do? Firts thought is, it replaces an existing drive with another / a larger one.

But obviously it doesn't

thx,
M

[ChriZathens]

It replaces a drive with a different one. For example it replaces a faulty drive with a new one.
The replacement drive can be larger, but in order to aquire larger space, all drives must be replaced with bigger ones.

[marqus]

and when does the actual "enlargement" start? Will the system automatically rebuild everything, once the last missing large drive has been added?

[kenZ71]

You can only replace one drive at a time. As each drive is replaced the pool is rebuilt. If you have the ability to copy your data you could copy it off then setup a pool with 2 4TB disks or 3 3TB.

I hope you have backups. I personally fear fire, flood or other similar perils more than a drive failure.

[marqus]

Sorry, I do not get it. The pool rebuilds as each drive is replaced, you say. But the larger space is not available yet. It is available when the LAST drive has been added and the system rebuilds for the LAST time? Thats how I thought it works...
Lets say, I use the replac function within the GUI and replace one of my 5 (actually 5 with a sixth as hotswap, which I will not use any longer in the future...) 1TB with a 3 or 4 TB drive. Then the system does some rebuilding but still using the same size? When will the extra TBytes of space be finally available for me to store data to?
Wahts the trigger, the system knows from "well, now you have more space!"?

thx

[b0ssman]

did i read that right that you have one pool and 1 raidz1 and one single drive in that pool?

in that case if the single drive fails your entire pool is gone!

[ChriZathens]

If this is true (did not get it from the initial post) then, indeed, if you lose that alone drive you lose everything.
I did not realize that you have two vdevs, one as a raidz1 and one with a single disk
I already wrote in a previous post:

You can create a vdev consisting of a single 3TB disk and add it to your pool. This way you will gain additional 3TB. But DON'T do it.
Loss of the single disk will result to total loss of your pool and all your data!

I hope you can create a backup quickly if you don't already have one...

Anyways, to expand the new size after changing all drives with bigger ones, you need to have autoexpand to on (see what is this property for your pool by running "zpool get all poolname"
If it is off you can turn it one by running "zpool set autoexpand=on poolname"
But a lot of people reported that it does not work for them (the autoexpansion)
In that case a zpool online -e poolname will do the trick and you will have the available space (the process is almost instant AFAIK)
But if what b00sman said is true there is no meaning to try anything. You WILL lose all your data eventually with this disk architecture

[marqus]

Hi there,

Yes, I have a pool called deeppool. I have a vDev called vRaidz1 which contains 5x1TB disks. I have another vDev called vSwap in the same Pool containing 1x1TB but configured as HOTSWAP and not as part of my storage...
Yesterday I deleted the vSwap via WebGUI for I needed the drive on a different PC...

The vRaidz1 is still untouched and works fine, still.

There's no indicator that somethings faulty or dangerous. Everything is online, status ok.

Am I on the wrong track completely?

thx

[raulfg3]

a hotswap disk is not part of a pool.

( do not sum in the free size of the pool), so you can detach and do not lose data.

[marqus]

Ok, then we do not have a problem regarding the 2nd vDev...in my case.

What du you mean by "do not sum the free size of the pool?"?
thx,
Markus

[ChriZathens]

He means that the size of the spare is not calculated to the size of your pool.
Please post the result of Disks|ZFS|Pools|Information (or go to Advanced|Execute command, write zpool status and click execute) so that we are sure what we are talking about

[marqus]

$ zpool status
pool: deeppool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
deeppool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ada0.nop ONLINE 0 0 0
ada1.nop ONLINE 0 0 0
ada2.nop ONLINE 0 0 0
ada3.nop ONLINE 0 0 0
ada4 ONLINE 0 0 0
spares
7015092499884191078 FAULTED was /dev/ada4

errors: No known data errors

[marqus]

AAARGH, ok, now I am scared. Yesterday when I just disconnected the hotspare drive ada4, everything seemed to be ok. No, when I look in disks/management it says ada0-ada3 ONLINE and ada5 MISSING :-O
ada5 I accidentally plugged off when I thought it was the HOTSPARE. The System of course was not running, and I immediately plugged it on again. Same cable, same Port!

I then unplugged the ada4 drive as planned.

right after that, everything was fine, all disks were ONLINE and I could access my files. Now I still can access my files, but I see this MISSING status...

[ChriZathens]

Yesterday I deleted the vSwap via WebGUI for I needed the drive on a different PC...

Since I don't use spares (they are useless with Freebsd ATM IMHO) I cannot be 100% sure about your problem. And since your raidz seems to be online I would assume that all is fine and not attempt to give advice to the wrong direction. Perhaps someone experienced with spares can offer help. But can you please explain how you deleted the spare?
I see in the webui there is a spare remove option in Disks|ZFS|Pools|Tools. But as far as I understand it is for devices, meaning disks. You are telling you had a vdev and you removed it... Can you explain more?


I would also try to do something else: Use a usb stick for embedded install and install a fresh copy of the OS.
Then I would try to issue a command like zpool import poolname and see if the pool is imported OK and the data is accessible. I can't recommend you to do the same thing, though. You should not harm anything with this, but keep in mind that if you do it, you do it at your own risk...

[marqus]

Ok, I really need help now. IMHO the problem is that I did sth wrong whilst disconnecting the spare. It looks as if the system does know that there once was a spare called ada4 as you can see here in the last line:
NAME STATE READ WRITE CKSUM
deeppool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ada0.nop ONLINE 0 0 0
ada1.nop ONLINE 0 0 0
ada2.nop ONLINE 0 0 0
ada3.nop ONLINE 0 0 0
ada4 ONLINE 0 0 0
spares
7015092499884191078 FAULTED was /dev/ada4

Now, the prior ada5 has switched to ada4 because the system simply goes by numberrs and after 3 comes 4 and the old 4 is missing, so the old 5 becomes the new 4!

The disk itself is still somewhere listed as ada5...
Also it says:
Total: 4.53T | Used: 3.56T | Free: 2.61G | State: ONLINE

hm, that cannot be good...

I already miss some files I think. When I go under Win8 and choose my "audio" directory -> properties, it says "SIZE 178GB" but "SIZE ON DISK 210GB".

So I already lost stuff.

I went under Disks|ZFS|Pools|Virtual device and chose the vSwap vDev. Then I just clicked the red cross. Thats all I did and I think that killed my files...

I would give alot if only the 5th drive would show up again under disks and not MISSING...

[b0ssman]

windows info with size on disk has nothing to do with you loosing data.

it simply means that you have 178 gb of audio but those 178 gb of data take up 210gb of space on the drive because of overhead in the file system.

[marqus]

Ok, but isnt that way too much overhead. When I remember correctly the "Used" value of this overview line (Total: 4.53T | Used: 3.56T | Free: 2.61G | State: ONLINE
) had been larger before the crash fo my 5th disk.

However...Is there any way or idea to "online" my disk again? How can I turn ada5 to ada4 (or vice versa)?

thx alot,
markus

EDIT: Sorry, I' ve been on the wrong track totally. Of course, it is around 3.6TB of available space. I had 6 disks, one of which was a spare (6-1=5). One of these 5 is parity due to RaidZ1. Thats why 3.6TB is totally fine...

But I really do not want to read MISSING any more...any (!) ideas on your side?

thanks

[marqus]

Ok, so obviously there is no solution for my problem.
Can you please tell me what I have to do exactly, when I want to replace the drives one by another? I unplug disk n and put a new 4TB drive on its place. Do I have to do sth in the webGui BEFORE? AFTER?

Thx alot,
Markus