Sorry we lost some posts because of database errors!

*New 12.1 series Release:
2020-09-01: XigmaNAS 12.1.0.4.7728 - released

*New 11.4 series Release:
2020-08-27: XigmaNAS 11.4.0.4.7718 - released!


We really need "Your" help on XigmaNAS https://translations.launchpad.net/xigmanas translations. Please help today!

Producing and hosting XigmaNAS costs money. Please consider donating for our project so that we can continue to offer you the best.
We need your support! eg: PAYPAL

How can I clone the RootonZFS Disk?

New installs of XigmaNAS.
Forum rules
Set-Up GuideFAQsForum Rules
Post Reply
User avatar
Viper780
Starter
Starter
Posts: 45
Joined: 23 Jun 2012 14:54
Location: Vienna / Austria / Europe
Contact:
Status: Offline

How can I clone the RootonZFS Disk?

#1

Post by Viper780 »

Hi there

I've installed Xigmanas as RootonZFS on a SSD, to use the extra Space I also put there a "data" ZFS dataset where I've installed several Jails.
Now the SSD slowly dies and I wan't to move the whole Disk to another SSD.

Can I simply boot a live Linux, copy the whole Disk via dd and boot from the new SSD?
How can I enlarge the "data" dataset? (The new SSD has 4 times the space )

User avatar
JoseMR
Hardware & Software Guru
Hardware & Software Guru
Posts: 1064
Joined: 16 Apr 2014 04:15
Location: PR
Contact:
Status: Offline

Re: How can I clone the RootonZFS Disk?

#2

Post by JoseMR »

Viper780 wrote:
02 Aug 2020 23:54
Hi there

I've installed Xigmanas as RootonZFS on a SSD, to use the extra Space I also put there a "data" ZFS dataset where I've installed several Jails.
Now the SSD slowly dies and I wan't to move the whole Disk to another SSD.

Can I simply boot a live Linux, copy the whole Disk via dd and boot from the new SSD?
How can I enlarge the "data" dataset? (The new SSD has 4 times the space )

Honestly forget about the Linux stuff here. ;)

You just need to learn Boot Environments and ZFS data replication(send/recv), simply Export the latest Boot Environment to a safe location then re-install the ssd as single or even as a mirror setup then Import back the Boot Environment.

As for the jails in the DATA partition/dataset, you need to save those manually, but if you are using Bastille Jail Manager, you can export them to a safe location the import the jails later on.

As for enlarging the dataset, simply install into the new SSD and use the space you want, you can later add the becked-up data tho this dataset by extracting or replicating the ZFS dataset.

Regards
System: FreeBSD 12 RootOnZFS Mirror, MB: Supermicro X8SI6-F, Xeon X3450, 16GB DDR3 ECC RDIMMs.
XigmaNAS RootOnZFS
Addons at GitHub
BastilleBSD
Boot Environments Intro
Resources Home Page

User avatar
Viper780
Starter
Starter
Posts: 45
Joined: 23 Jun 2012 14:54
Location: Vienna / Austria / Europe
Contact:
Status: Offline

Re: How can I clone the RootonZFS Disk?

#3

Post by Viper780 »

I can also use dd from frebsd or a live xigmanas stick.

Do you have a good "How to" or something similar for the Boot Environments? a bit lost and overwhelmed on this topic (read the whole day different stuff and still do't know where to start)
I'm using Bastille (and love it) but my backup and export won't work (don't know why, don't see any errors in the log).

The way you describe needs 4 independent Disks! (the old SSD, which hopfully still boots, a safe Data storage which is large enough, the new Disk and a USB drive to install the new disk)

So there is no simple cloning like every other OS supports?

User avatar
JoseMR
Hardware & Software Guru
Hardware & Software Guru
Posts: 1064
Joined: 16 Apr 2014 04:15
Location: PR
Contact:
Status: Offline

Re: How can I clone the RootonZFS Disk?

#4

Post by JoseMR »

Viper780 wrote:
03 Aug 2020 17:53
I can also use dd from frebsd or a live xigmanas stick.

Do you have a good "How to" or something similar for the Boot Environments? a bit lost and overwhelmed on this topic (read the whole day different stuff and still do't know where to start)
I'm using Bastille (and love it) but my backup and export won't work (don't know why, don't see any errors in the log).

The way you describe needs 4 independent Disks! (the old SSD, which hopfully still boots, a safe Data storage which is large enough, the new Disk and a USB drive to install the new disk)

So there is no simple cloning like every other OS supports?

Hello, you can of course use `dd` command to dump your SSD to an image, then use it again to write the image back to the new SSD, while this may be ok for cloning the OS 1:1 on identical drives, I personally never tried it for cloning a GPT/ZFS drive to a larger one, probably you may get GEOM warnings etc. but again I have never tried, I could test in this during the week as I'm already curious about too and let you know, though during the week.

As for a "good How-To" on FreeBSD Boot Environments, the internet is overloaded on this already and the FreeBSD forums as well, however if you want the "easy go" route you can install the Extension Boot Environments Manager(recommended) or the less user friendly CLI/TUI Boot Environments Manager(advanced), all you need is to export the latest/running BE to a safe/permanent DATA location/pool/mountpoint etc., then after install XigmaNAS on the new SSD as you normally do, simply install again one of the previously mentioned Boot Environments utilities and import the Boot Environments and restart.

At that point the system will boot from the last state with all your scrips and the third parties packages already installed, however since you opted to use a separate dataset rather than simply using all the available disk for the OS + jails it will complicate things a little more since you need to backup said jails either manually of by the built-in export/import from the jail manager in question.

Also I don't know what you meant that Bastille Jail Manager backup/restore function does not work for you, can you please elaborate a bit more on what makes you suspect it is not working?

P.S. I will strongly advice to install jails the same way as you install your critical DATA, on a separate ZFS pool, the user dataset option on the RootOnZFS installer menu is simply for non critical replaceable data, if this is used for critical Data, just like in Embedded install Data partition, the user must backup this then restore later on, I would rather use and store the jails in the OS "/usr/local/jails" personally on RootOnZFS, though Bastille Jail Manager was designed to enforce the user to always place their jails in a separate data location as previously denoted, however if this is a thin-client like NAS with hardware limited to a single disk then I understand on why you just want to clone it.

And last but not least be aware if this is a dying HDD/SSD with several bad sectors and garbage/corrupted data, you will also restore this to the new SSD as the `dd` command is a 1:1 data clone(sector by sector).

Regards
System: FreeBSD 12 RootOnZFS Mirror, MB: Supermicro X8SI6-F, Xeon X3450, 16GB DDR3 ECC RDIMMs.
XigmaNAS RootOnZFS
Addons at GitHub
BastilleBSD
Boot Environments Intro
Resources Home Page

User avatar
Viper780
Starter
Starter
Posts: 45
Joined: 23 Jun 2012 14:54
Location: Vienna / Austria / Europe
Contact:
Status: Offline

Re: How can I clone the RootonZFS Disk?

#5

Post by Viper780 »

I can say it works with dd when you repair the GPT header
I've used this forum post as instruction https://forums.freebsd.org/threads/new- ... ost-419742

As to Bastille and my Backup problem
there was just an error on the webinterface when i tried backups - I've wanted to investigate further and ask you for advice.
But playing with zfs send and recv I lost all my jails and "data". Nothing critical. But it will take me another week to set everything up as it was before.

actually this is an older self built workstation with 9x4TB HDD for storage and a 256GB SSD for Xigmanas, the Jails (download stuff, unifi controller, homeautomation, grafana,...) few Databases and the intranet webserver.
I just upgraded the SSD to 1TB and basically it is faster and consumes less power when I just use the SSD.
Thinking about using a 500GB NVME SSD just for the Jails and Database (initially was planed for ZIL and L2ARC but I guess my usecase doesn't benefit from it)

User avatar
JoseMR
Hardware & Software Guru
Hardware & Software Guru
Posts: 1064
Joined: 16 Apr 2014 04:15
Location: PR
Contact:
Status: Offline

Re: How can I clone the RootonZFS Disk?

#6

Post by JoseMR »

Viper780 wrote:
03 Aug 2020 21:56
I can say it works with dd when you repair the GPT header
I've used this forum post as instruction https://forums.freebsd.org/threads/new- ... ost-419742

As to Bastille and my Backup problem
there was just an error on the webinterface when i tried backups - I've wanted to investigate further and ask you for advice.
But playing with zfs send and recv I lost all my jails and "data". Nothing critical. But it will take me another week to set everything up as it was before.

actually this is an older self built workstation with 9x4TB HDD for storage and a 256GB SSD for Xigmanas, the Jails (download stuff, unifi controller, homeautomation, grafana,...) few Databases and the intranet webserver.
I just upgraded the SSD to 1TB and basically it is faster and consumes less power when I just use the SSD.
Thinking about using a 500GB NVME SSD just for the Jails and Database (initially was planed for ZIL and L2ARC but I guess my usecase doesn't benefit from it)

Hi, if it works with `dd` and after repair the GPT header and ZFS auto expand, it is great despite this method may take some time on large drives.

As for the Bastille backup error, I'm sure that your issue may be related to this issue/solution posted HERE.

Also you are completely right about ZIL and L2ARC, it is actually somewhat useless in a simple home NAS, while somewhat useful for SOHO servers, NFS, iSCSI, VM's setups etc, but almost a requirement for the enterprise, if this SSD is 1TB and you have only a single disk, either you can actually use all the space available for the OS + Jails to keep the big Data pool from constant access thus reducing power consumption or even setting copies=2 may be useful in big single disk setups though I haven't added this option in the RootOnZFS installer but I'm considering it. :idea:

I think I will play with an easy dataset export/import script here to see how useful this may be, something similar to exporting/importing Boot Environments, it would be cool to take a recent dataset/snap and export/import with a simple TUI/GUI dialog keyboard/mouse clicks. :?:

Regards
System: FreeBSD 12 RootOnZFS Mirror, MB: Supermicro X8SI6-F, Xeon X3450, 16GB DDR3 ECC RDIMMs.
XigmaNAS RootOnZFS
Addons at GitHub
BastilleBSD
Boot Environments Intro
Resources Home Page

User avatar
Viper780
Starter
Starter
Posts: 45
Joined: 23 Jun 2012 14:54
Location: Vienna / Austria / Europe
Contact:
Status: Offline

Re: How can I clone the RootonZFS Disk?

#7

Post by Viper780 »

dd'ing the 250GB SSD to the 1TB took about 7min (was 2/3 full, but makes no difference to dd)

L2ARC and ZIL doesn't make so much sense after SSD are getting cheaper every day. Everything I/O heavy would be put on a SSD ( and just mirror it). I'm working with a (more or less) large hoster, and only the cheapest webspaces are on HDD with 1-2 SSD as caches.

As for bastille, no the path was fine, I guess it was a problem with ZFS - which couldn't be activated (but was on) - I'll take more care this time

Thank you for helping and putting such nice features in Xigmanas

was-armandh
experienced User
experienced User
Posts: 111
Joined: 19 Nov 2018 11:30
Location: St Louis County MO USA
Status: Offline

Re: How can I clone the RootonZFS Disk?

#8

Post by was-armandh »

I have had the best luck cloning bootable stuff with clone-zilla [included with g-parted free down load]
I recommend starting over fresh when one does not know the extent of the failures.
by using this free advise, you, your heirs, etc
absolve, save, and hold harmless the advisor.
CAVEAT EMPTOR
YMMV

francoisv
NewUser
NewUser
Posts: 4
Joined: 25 Aug 2020 23:13
Status: Offline

Re: How can I clone the RootonZFS Disk?

#9

Post by francoisv »

Hello,

Lost posts about dd and gnu ddrescue here.
Briefly, in case of bad/problematic drive you should try ddrescue because it reports bad sectors.
Someone aked if ddrescue was still working with SSDs ?
I emailed the gddrescue team and here follows their answer.

---
> There's a talk on Xigmanas forums (custom FreeBSD OS NAS) about SSD with
> ddrescue and dd:
> https://www.xigmanas.com/forums/viewtop ... 15&t=15441
> (see #10 & 11).

I get an "Internal server error" when trying to access that page.

> Does ddrescue work on SSD & NVMe?

Ddrescue should work with any device for which the kernel provides a block device name.

> I've done clones of SSDs with ddrescue but they were brand new, so can't
> say for sure if ddrescue detects bad cells on them to try to rescue them.

Ddrescue does not "detect bad cells". It is the device driver the one that detects them and returns a "read error" to ddrescue.

> Also, is there a way, on SSD or NVMe, to get the files' names lying in
> bad sectors with ddrescue?

The same methods used for hard discs should work because they are performed on the copy. See http://www.gnu.org/software/ddrescue/ma ... #Fill-mode


Best regards,
Antonio.
---

Regards

User avatar
JoseMR
Hardware & Software Guru
Hardware & Software Guru
Posts: 1064
Joined: 16 Apr 2014 04:15
Location: PR
Contact:
Status: Offline

Re: How can I clone the RootonZFS Disk?

#10

Post by JoseMR »

Hello, while `dd` might work for cloning RootOnZFS disks, there is absolutely no need to get tru that hassle when replacing a disk.

I would advise to simply use the built-in ZFS Boot Environments, export the current working BE to a local/remote data storage/share, then you can later import it to a single drive or even to the ZFS RAID of choice during the initial XigmaNAS installation, thats it.

If there is additional user created datasets in the RootOnZFS disk, i.e. below "/mnt" simply backup that data with your favorite backup solution, i.e. tar, rsync, zfs send/recv etc to a local data storage, you can later sync back the data after the Boot Environment import, though I would suggest to avoid storing Critical Data on boot drives regardless of the filesystem of choice, User data should be keep separate from boot whenever possible.


An introduction to Boot Environments by Sławomir Wojciech Wojtczak(vermaden).

Boot Environments GUI Manager can be found HERE
Boot Environments CLI/TUI Manager Tool can be found HERE

Regards
System: FreeBSD 12 RootOnZFS Mirror, MB: Supermicro X8SI6-F, Xeon X3450, 16GB DDR3 ECC RDIMMs.
XigmaNAS RootOnZFS
Addons at GitHub
BastilleBSD
Boot Environments Intro
Resources Home Page

Post Reply

Return to “Install XigmaNAS”