*New 12.1 series Release:
2020-09-13: XigmaNAS 12.1.0.4.7743 - released

*New 11.4 series Release:
2020-09-12: XigmaNAS 11.4.0.4.7741 - released!


We really need "Your" help on XigmaNAS https://translations.launchpad.net/xigmanas translations. Please help today!

Producing and hosting XigmaNAS costs money. Please consider donating for our project so that we can continue to offer you the best.
We need your support! eg: PAYPAL

[Tutorial]Replacing HDDs in an encrypted pool

Encrypting information and help
Forum rules
Set-Up GuideFAQsForum Rules
Post Reply
User avatar
Earendil
Moderator
Moderator
Posts: 53
Joined: 23 Jun 2012 15:57
Location: near Boston, MA, USA.
Status: Offline

[Tutorial]Replacing HDDs in an encrypted pool

#1

Post by Earendil »

I see little on this forum pertaining to encryption (when I say that I always mean geli). All my pools are encrypted even in my Qnaps. You may think it is for nefarious purposes but no, where I work really enforces security and that turns out to be safe. So I do it at home as well.

To begin the story, I have a pool (pool1) of six encrypted 4 TB WD green HDDs in RAIDZ1 (20 TB of space) in XigmaNAS which is always up to date (including ZFS and pool flags). The pool exists since the FreeNAS7 days and has been updated (2 TB HDDs to 4 TB HDDs) once before. The current configuration with the current 4 TB HDDs have been running 24/7/365 for about 3 years without problems off of my system (Gigabyte GA-F2A88XM-D3HP m/b with a A10-7860K CPU and 16 GB memory). pool1 is half full and I want to expand. I found some WD white 8 TB HDDs and have had them in a 4 slot Qnap (TS-431P2) for 6 months without issue. I have now bought 7 (one spare) more for the upgrade and have covered the 3.3V pin on all the HDDs as required to be recognized by my system.

Note: FYI, y'all may bash my choice of HDDs, equipment and raid type but that's what I want so please no bashing though polite conversation on these matters are always appreciated.

I'll do this in steps to be clear:
  1. Backup all data on the pool to be upgraded. Took a day to copy and verify over 10 TB of data.
  2. Ensure XigmaNAS is up-to-date. That's me, I always upgrade. This also makes sure your pool is in autoexpend mode since the latest XigmaNAS sets this for you but let's check:
    • Go to the XigmaNAS home screen or to Disks>ZFS>Pools>Management. Hit the health status link of pool1 under "State:" or "Health" respectively.
    • Under "ZFS Pool Properties" is a list. Find "autoexpand" and it should be set to "on".
  3. Save the XigmaNAS config file. You never know...
  4. Shutdown the computer; ensure no scrubs or resilvers or other issues are pending on pool1.
    • Note, I have another pool (pool0) in the same system that is seriously effed up and I have been trying to repair it for weeks. It has 7(!) entries under the "replacing" list instead of the normal 2 entries (old and unavailable HDD (vdev) and the new and online HDD (vdev)). It often has "CAM status: ATA Status Error" where HDDs drop out for no discernible reason. It often pops into scrubs in a degraded status and does not finish a resilver. I have replaced SATA cables in droves, moved cables around, placed on-the-spot fans for cooling the SATA card, moved the SATA card into another slot and even replaced the SATA card (Syba SI-PEX40064 PCI-e 1x out, Ziyituod (used to be Ubit) SA3014 PCI-e 1x in). Last night it started resilvering and I hope it works this time. I am resolved to not give up and will fix it but it has no connection to the work on pool1.
  5. System is powered off. Physically replace one HDD in pool1. pool1 has six HDDs labeled ada4 through ada9, so start anywhere. I progressed through ada8, ada4, ada5, ada6, ada7, ada9 just because ada8 had the most old-age settings in SMART but any order is fine. Let's start with ada4 for this exercise.
  6. Start system into XigmaNAS (I run embedded). I enter a password through the console for each encrypted HDD in the system upon bootup. Note there is no ada4. At NOP GEOM in the console it takes quite some time because it's looking for ada4 (or in my case pool0 is messed up) but just wait it out (about 1-2 minutes).
  7. After startup and logging into the XigmaNAS GUI, go to Disk>Management>HDD Management. I see ada4 is different than before and XigmaNAS tells you there are changes because it's smart. I usually clear the config (select check box) and import disks. Accept changes XigmaNAS finds. Now ada4 HDD should be recognized by XigmaNAS.
  8. Optional: I usually go to the Disk>Encryption>Management screen to "Clear Config and Import Disks" just to make sure XigmaNAS doesn't have any remnants of the old ada4. That's me.
  9. Time to encrypt the entire ada4 HDD. Since FreeNAS/NAS4Free/XigmaNAS coders will not put a boot option in the GUI, I need PuTTY to enter the encryption command. You could also use the Tools>Execute Command as your Command Line Interface (CLI) but PuTTY shows a little more like when the command finishes. Either way works though. Here are the arguments for the program that encrypts things in FreeBSD, geli (found here):
    • I want AES-XTS which is default, needs no extra argument.
    • I want a 256 key so use "-l 256".
    • I want a sector size of 4096 so "-s 4096".
    • It's whole disk encryption and I want it available from bootup so I add "-b". Because this is not an option in the XigmaNAS GUI, geli must be run from the CLI.
    • Result is for our ada4 HDD with arguments in any order is:

      Code: Select all

      geli init -b -s 4096 -l 256 /dev/ada4
  10. Two prompts ask for the passcode twice which I type in. You could use key files and all sorts of other things but a passcode is fine for me. Note this means the pro is you need to enter a passcode to access the encrypted HDDs at every bootup (it is not an auto-fill method although you can set that up, too) which is safe but the con is you have to enter them with every bootup which is tedious. After about 15 secs it'll remind you where restore info is and resolve back to the CLI.
  11. At this point ada4 is no longer relevant, it is now ada4.eli that is the HDD to be used for replacement.
  12. Attach ada4.eli, this can be easily done though the GUI at Disk>Encryption>Tools. Enter in your passcode, of course.
  13. Same as #8 above, refresh the Encryption list of HDDs. Note the encryption type (AES-XTS) and that ada4 is attached.
    • If you are curious you could go to Disk>Encryption>Tools, pick any HDD and select "List" from the pull down menu. In the output look for ada4.eli and you'll see all the details of your encrypted HDD including the boot setting enabled under "Flags:".
  14. Now to simply replace the old for the new in pool1.
    • Go to Disks>ZFS>Pools>Tools>Step 1 in the GUI. Select "Replace a device". Hit "Next".
    • Select pool1. Do not select force. Hit "Next".
    • Under "Pool Devices" pick the entry that has ada4.eli in it, should also say missing.
    • Under "Select Data Device" select ada4 (it should say ada4.eli but don't worry, this is the HDD you want and it is only represented in the system as ada4.eli, an encrypted HDD, from #11 onwards).
    • Wrapping up what you've done, in pool1 you are replacing the old, missing ada4.eli with the new, available ada4.eli you just created. Hit "Next"
    • It'll take about 30 seconds and you know you succeeded when it tells you so.
  15. Go to Disks>ZFS>Pools>Information to see a list of all system pools and check out pool1. Under "status:" it says it's resilvering and under "scan:" it will eventually give you time to completion. Believe what it says under "action:" and wait for the resilver to complete before you restart the above steps from #4 on.
  16. I am currently on the 5th HDD but once it's done, the pool will auto-magically expand from 24 TB (20 TB useable) to 48 TB (40 TB usable). Or so I read everywhere. It happened during the old upgrade and although it's not XigmaNAS that's what happened in my Qnap when I upgraded a four HDD RAID5 pool.
  17. Good luck.
Earendil

XigmaNAS server:
-AMD A10-7860K APU
-Gigabyte F2A88XM-D3HP w/16GB RAM
-pool0 - 4x 2 TB WD green HDDs
-pool1 - 6x 8 TB WD white HDDs
-Ziyituod (used to be Ubit) SA3014 PCI-e 1x SATA card
-External Orico USB 3.0 5 bay HDD external enclosure set at RAID 5
--5x 4 TB WD green HDDs
-650W power supply

User avatar
Earendil
Moderator
Moderator
Posts: 53
Joined: 23 Jun 2012 15:57
Location: near Boston, MA, USA.
Status: Offline

Re: [Tutorial]Replacing HDDs in an encrypted pool

#2

Post by Earendil »

Note, I have another pool (pool0) in the same system that is seriously effed up and I have been trying to repair it for weeks. It has 7(!) entries under the "replacing" list instead of the normal 2 entries (old and unavailable HDD (vdev) and the new and online HDD (vdev)). It often has "CAM status: ATA Status Error" where HDDs drop out for no discernible reason. It often pops into scrubs in a degraded status and does not finish a resilver. I have replaced SATA cables in droves, moved cables around, placed on-the-spot fans for cooling the SATA card, moved the SATA card into another slot and even replaced the SATA card (Syba SI-PEX40064 PCI-e 1x out, Ziyituod (used to be Ubit) SA3014 PCI-e 1x in). Last night it started resilvering and I hope it works this time. I am resolved to not give up and will fix it but it has no connection to the work on pool1.
If anyone cares, pool0 never did finish resilvering as I had more errors. The actual pool0 make up is four, encrypted 2 TB HDDs labeled ada0 through ada3 on a 4 port SATA cord. ada3 had errors and I was trying to resilver. Either ada3 or ada2 would get timeout errors or geli errors (I would see them in the console in real time or in the logs) and thus fall out of the pool in the middle of a resilver. I currently have 8(!!) entries under "replacing" in a Disks>ZFS>Pools>Information list of pool0 information.

As I mentioned I replaced lots for ada3 and ada2 and also tried three different 2 TB HDDs in place of ada3. The power supply is only a year old and 650W, more than double the power consumption of the system so I know it works. Everything was changed but the m/b and power cables. Turns out I had a Molex to Molex Y cable and then Molex to SATA power connector Y cable in series between the power supply and those two HDDs. This was my setup for years. I replaced all of pool0 SATA power connectors with a Molex to 4 SATA power connectors splitter with the SATA power connectors in series (think daisy chain, looks like this). I already am using the same splitter to power three of my pool1 HDDs (each set of 3 HDDs in the pool1's 6 HDDs are separated for air flow and the splitter's fourth SATA power connector cannot span the gap). After the restart it started resilvering pool0. After running for hours and no errors, the resilvering worked, all the "replacing" entries went away and pool0 is fine.

And as for pool1, I have a 48 TB pool. Cool! :ugeek:
Earendil

XigmaNAS server:
-AMD A10-7860K APU
-Gigabyte F2A88XM-D3HP w/16GB RAM
-pool0 - 4x 2 TB WD green HDDs
-pool1 - 6x 8 TB WD white HDDs
-Ziyituod (used to be Ubit) SA3014 PCI-e 1x SATA card
-External Orico USB 3.0 5 bay HDD external enclosure set at RAID 5
--5x 4 TB WD green HDDs
-650W power supply

Post Reply

Return to “Encryption”