*New 11.3 series Release:
2019-10-19: XigmaNAS 11.3.0.4.7014 - released

*New 12.0 series Release:
2019-10-05: XigmaNAS 12.0.0.4.6928 - released!

*New 11.2 series Release:
2019-09-23: XigmaNAS 11.2.0.4.6881 - released!

We really need "Your" help on XigmaNAS https://translations.launchpad.net/xigmanas translations. Please help today!

Producing and hosting XigmaNAS costs money. Please consider donating for our project so that we can continue to offer you the best.
We need your support! eg: PAYPAL

rsync creates copies instead of links in incremental backup script

Synchronize files & directories to/from NAS4Free with minimal data transfer.
Forum rules
Set-Up GuideFAQsForum Rules
Post Reply
Sparkster
NewUser
NewUser
Posts: 1
Joined: 08 Jun 2017 19:16
Status: Offline

rsync creates copies instead of links in incremental backup script

#1

Post by Sparkster » 08 Jun 2017 19:52

Hi guys, I hope you can help me with this.

I created a script to backup some of my stuff into an external hard drive. This hard drive has an NTFS file system, so I ensured I could create links by creating a file, hard-linking it to a different file in a different directory and checking the size of both directories at the same time (it's the same size as the size of each one of the directories separately, so the link works in NTFS).

After checking it, I created a script to perform an incremental backup via rsync. This script reads the input data from a text file, which consists on 3 lines per directory:
1 - Upper directory of the directory to save
2 - Name of the directory to save
3 - Name of the directory where it is going to be saved in the external hard drive.

As an example, if the backup disk is mounted in /mnt/backup and I want to save my directory /home/multimedia in the directory /mnt/backup/backup.X/saved_stuff, the 3 lines in the configuration file would be:
1: /home
2: multimedia
3: saved_stuff

The reason for this weird structure is that I am saving several independent files and folders from different places that might have the same name. This way I can keep them in separated spaces inside the same backup, for instance, /home/sparkster/dir1/backupme and /home/sparkster/dir2/backupme,

After performing the backup for the first time, the copy is /mnt/backup/backup.0/saved_stuff/multimedia.
If I run the backup script again, this backup.0 directory is moved to backup.1 and the new backup.0 directory is created before the actual backup is performed.

After running the backup script for the second time, I noticed it was performing just way too slow. I checked the I/O in my local hard drive and it was very low, but the I/O in my backup hard drive was rather high, which should not happen, since the second backup was an exact copy of the first one. I checked the size of the 2 backups together and it was twice the size of one single backup, which means it's copying the files from the backup disk to the same backup disk, instead of linking them.

The line where I run the backup is this one:

Code: Select all

rsync -aH --delete --link-dest=/mnt/backup/incremental_backup/backup.1/$destination_directory/"${directory_name}" $upper_directory/"${directory_name}" /mnt/backup/incremental_backup/backup.0/$destination_directory/"${directory_name}"
I will also include a simplified version of the whole script in case it helps. It might be a bit messy, so sorry about that. It's divided into 3 parts:
1 - Mounts the backup hard drive and performs checks for the hard drive and the directories to be saved
2 - Swaps all backup directories one position forward in order to save the latest backup in backup.0
3 - Creates the directory backup.0 and performs the actual backup

Code: Select all

backup_script_directory=/home/sparkster/programas/scripts/backup
backup_mountpoint=/mnt/backup
backup_directory_name="incremental_backup"
backup_directory=$backup_mountpoint/$backup_directory_name
backup_disk_label=Seagate\ Backup\ Plus\ Drive
date=`date +%Y-%m-%d--%H%M%S`
directories_file=$backup_script_directory/directorios.txt
directories_number=0
files_number=0
total_bar_dots=50

#MONTA EL DISCO DE BACKUP EVITANDO LOS MENSAJES DE ERROR
echo -n "Montando disco de backup... "
mount -L "${backup_disk_label}" $backup_mountpoint 2>/dev/null

#COMPRUEBA SI EL DISCO ESTÁ MONTADO, Y SI NO, SALE
if ! grep -qs $backup_mountpoint /proc/mounts ; then
    echo "Error. Consultar $backup_script_directory/log.txt"
    echo $date" - Disco de backup no disponible" >> $backup_script_directory/log.txt
    exit
fi

echo "OK"

if ! [ -d $backup_directory ] ; then
    echo "Creando directorio general de backup"
    mkdir $backup_directory
fi

echo -n "Comprobando directorios a salvar... "
counter=0
#LEE EL FICHERO DE DIRECTORIOS DE 3 EN 3 LINEAS
#PARA COMPROBAR QUE LOS DIRECTORIOS EXISTEN:
#   - Directorio padre
#   - Fichero a salvar
#   - Directorio de destino
while IFS='' read -r line || [[ -n "$line" ]]; do
    ((counter++))
    
    if [[ $counter = 1 ]] ; then
        upper_directory=$line
    fi
    
    if [[ $counter = 2 ]] ; then
        directory_name=$line
    fi
        
    if [[ $counter = 3 ]] ; then
        
        #REINICIA EL CONTADOR. EN ESTE PUNTO TIENE TODA LA INFO PARA SALVAR UN DIR
        ((counter=0))
        destination_directory=$line
    
         if ! [ -d $upper_directory/"${directory_name}" ] ; then 
            echo "Error. Consultar $backup_script_directory/log.txt"
            echo $date" - El directorio a salvar $upper_directory/"${directory_name}" no existe" >> $backup_script_directory/log.txt
            umount $backup_mountpoint
            exit
        fi
        
        ((directories_number++))
    fi
done < $directories_file

echo "OK"
echo -n "Desplazando backups anteriores... "

#DESPLAZA LOS DIRECTORIOS DE BACKUP INCREMENTAL
rm -rf $backup_directory/backup.9
mv $backup_directory/backup.8 $backup_directory/backup.9
mv $backup_directory/backup.7 $backup_directory/backup.8
mv $backup_directory/backup.6 $backup_directory/backup.7
mv $backup_directory/backup.5 $backup_directory/backup.6
mv $backup_directory/backup.4 $backup_directory/backup.5
mv $backup_directory/backup.3 $backup_directory/backup.4
mv $backup_directory/backup.2 $backup_directory/backup.3
mv $backup_directory/backup.1 $backup_directory/backup.2
mv $backup_directory/backup.0 $backup_directory/backup.1

echo "OK"
echo -n "Creando nuevo directorio de backup... "

mkdir $backup_directory/backup.0
echo "" > $backup_directory/backup.0/date-$date.txt

echo "OK"
echo "Salvando directorios"

counter=0
#LEE EL FICHERO DE DIRECTORIOS DE 3 EN 3 LINEAS
#PARA REALIZAR EL BACKUP:
#   - Directorio padre
#   - Directorio a salvar
#   - Directorio de destino
while IFS='' read -r line || [[ -n "$line" ]]; do
    ((counter++))
    
    if [[ $counter = 1 ]] ; then
        upper_directory=$line
    fi
    
    if [[ $counter = 2 ]] ; then
        directory_name=$line
    fi
        
    if [[ $counter = 3 ]] ; then
        
        #REINICIA EL CONTADOR. EN ESTE PUNTO TIENE TODA LA INFO PARA SALVAR UN DIR
        ((counter=0))
        destination_directory=$line
        
        if ! [ -d $backup_directory/backup.0/$destination_directory ] ; then 
            mkdir $backup_directory/backup.0/$destination_directory
        fi
        
        rsync -aH --delete --link-dest=$backup_directory/backup.1/$destination_directory/"${directory_name}" $upper_directory/"${directory_name}" $backup_directory/backup.0/$destination_directory/"${directory_name}" 2>>$backup_script_directory/log.txt
    fi
done < $directories_file

echo "Backup finalizado. Limpiando cache."
echo $date" - Backup finalizado" >> $backup_script_directory/log.txt

echo -n "Desmontando disco de backup... "

umount $backup_mountpoint

echo "OK"
I might be using a wrong command for rsync or maybe it's because of my directories structure or the filesystem in the backups disk. As I said, when creating links to files with ln file1 file2, it works properly and the new file is just a link and not a new file, so I'm not sure where the problem is. Any help would be appreciated.

Thank you guys.

Post Reply

Return to “RSYNC”