To begin the story, I have a pool (pool1) of six encrypted 4 TB WD green HDDs in RAIDZ1 (20 TB of space) in XigmaNAS which is always up to date (including ZFS and pool flags). The pool exists since the FreeNAS7 days and has been updated (2 TB HDDs to 4 TB HDDs) once before. The current configuration with the current 4 TB HDDs have been running 24/7/365 for about 3 years without problems off of my system (Gigabyte GA-F2A88XM-D3HP m/b with a A10-7860K CPU and 16 GB memory). pool1 is half full and I want to expand. I found some WD white 8 TB HDDs and have had them in a 4 slot Qnap (TS-431P2) for 6 months without issue. I have now bought 7 (one spare) more for the upgrade and have covered the 3.3V pin on all the HDDs as required to be recognized by my system.
Note: FYI, y'all may bash my choice of HDDs, equipment and raid type but that's what I want so please no bashing though polite conversation on these matters are always appreciated.
I'll do this in steps to be clear:
- Backup all data on the pool to be upgraded. Took a day to copy and verify over 10 TB of data.
- Ensure XigmaNAS is up-to-date. That's me, I always upgrade. This also makes sure your pool is in autoexpend mode since the latest XigmaNAS sets this for you but let's check:
- Go to the XigmaNAS home screen or to Disks>ZFS>Pools>Management. Hit the health status link of pool1 under "State:" or "Health" respectively.
- Under "ZFS Pool Properties" is a list. Find "autoexpand" and it should be set to "on".
- Save the XigmaNAS config file. You never know...
- Shutdown the computer; ensure no scrubs or resilvers or other issues are pending on pool1.
- Note, I have another pool (pool0) in the same system that is seriously effed up and I have been trying to repair it for weeks. It has 7(!) entries under the "replacing" list instead of the normal 2 entries (old and unavailable HDD (vdev) and the new and online HDD (vdev)). It often has "CAM status: ATA Status Error" where HDDs drop out for no discernible reason. It often pops into scrubs in a degraded status and does not finish a resilver. I have replaced SATA cables in droves, moved cables around, placed on-the-spot fans for cooling the SATA card, moved the SATA card into another slot and even replaced the SATA card (Syba SI-PEX40064 PCI-e 1x out, Ziyituod (used to be Ubit) SA3014 PCI-e 1x in). Last night it started resilvering and I hope it works this time. I am resolved to not give up and will fix it but it has no connection to the work on pool1.
- System is powered off. Physically replace one HDD in pool1. pool1 has six HDDs labeled ada4 through ada9, so start anywhere. I progressed through ada8, ada4, ada5, ada6, ada7, ada9 just because ada8 had the most old-age settings in SMART but any order is fine. Let's start with ada4 for this exercise.
- Start system into XigmaNAS (I run embedded). I enter a password through the console for each encrypted HDD in the system upon bootup. Note there is no ada4. At NOP GEOM in the console it takes quite some time because it's looking for ada4 (or in my case pool0 is messed up) but just wait it out (about 1-2 minutes).
- After startup and logging into the XigmaNAS GUI, go to Disk>Management>HDD Management. I see ada4 is different than before and XigmaNAS tells you there are changes because it's smart. I usually clear the config (select check box) and import disks. Accept changes XigmaNAS finds. Now ada4 HDD should be recognized by XigmaNAS.
- Optional: I usually go to the Disk>Encryption>Management screen to "Clear Config and Import Disks" just to make sure XigmaNAS doesn't have any remnants of the old ada4. That's me.
- Time to encrypt the entire ada4 HDD. Since FreeNAS/NAS4Free/XigmaNAS coders will not put a boot option in the GUI, I need PuTTY to enter the encryption command. You could also use the Tools>Execute Command as your Command Line Interface (CLI) but PuTTY shows a little more like when the command finishes. Either way works though. Here are the arguments for the program that encrypts things in FreeBSD, geli (found here):
- I want AES-XTS which is default, needs no extra argument.
- I want a 256 key so use "-l 256".
- I want a sector size of 4096 so "-s 4096".
- It's whole disk encryption and I want it available from bootup so I add "-b". Because this is not an option in the XigmaNAS GUI, geli must be run from the CLI.
- Result is for our ada4 HDD with arguments in any order is:
Code: Select all
geli init -b -s 4096 -l 256 /dev/ada4
- Two prompts ask for the passcode twice which I type in. You could use key files and all sorts of other things but a passcode is fine for me. Note this means the pro is you need to enter a passcode to access the encrypted HDDs at every bootup (it is not an auto-fill method although you can set that up, too) which is safe but the con is you have to enter them with every bootup which is tedious. After about 15 secs it'll remind you where restore info is and resolve back to the CLI.
- At this point ada4 is no longer relevant, it is now ada4.eli that is the HDD to be used for replacement.
- Attach ada4.eli, this can be easily done though the GUI at Disk>Encryption>Tools. Enter in your passcode, of course.
- Same as #8 above, refresh the Encryption list of HDDs. Note the encryption type (AES-XTS) and that ada4 is attached.
- If you are curious you could go to Disk>Encryption>Tools, pick any HDD and select "List" from the pull down menu. In the output look for ada4.eli and you'll see all the details of your encrypted HDD including the boot setting enabled under "Flags:".
- Now to simply replace the old for the new in pool1.
- Go to Disks>ZFS>Pools>Tools>Step 1 in the GUI. Select "Replace a device". Hit "Next".
- Select pool1. Do not select force. Hit "Next".
- Under "Pool Devices" pick the entry that has ada4.eli in it, should also say missing.
- Under "Select Data Device" select ada4 (it should say ada4.eli but don't worry, this is the HDD you want and it is only represented in the system as ada4.eli, an encrypted HDD, from #11 onwards).
- Wrapping up what you've done, in pool1 you are replacing the old, missing ada4.eli with the new, available ada4.eli you just created. Hit "Next"
- It'll take about 30 seconds and you know you succeeded when it tells you so.
- Go to Disks>ZFS>Pools>Information to see a list of all system pools and check out pool1. Under "status:" it says it's resilvering and under "scan:" it will eventually give you time to completion. Believe what it says under "action:" and wait for the resilver to complete before you restart the above steps from #4 on.
- I am currently on the 5th HDD but once it's done, the pool will auto-magically expand from 24 TB (20 TB useable) to 48 TB (40 TB usable). Or so I read everywhere. It happened during the old upgrade and although it's not XigmaNAS that's what happened in my Qnap when I upgraded a four HDD RAID5 pool.
- Good luck.