Introduction to rsync

rsync is a data image backup tool under linux system. Using the fast incremental backup tool Remote Sync, you can synchronize remotely, support local replication, or synchronize with other SSH and rsync hosts. Synchronize massive small files

rsync feature

rsync supports many features:

• You can mirror and save the entire directory tree and file system
• It is easy to maintain the permissions, time, soft and hard links of the original files
• Installation without special permission
• Fast: rsync will copy all the contents during the first synchronization, but only the modified files will be transferred next time. rsync can compress and decompress data, so it can use less bandwidth
• Security: you can use scp, ssh and other methods to transfer files. Of course, you can also connect directly through socket [socket] (IP + port number)
• Support anonymous transmission to facilitate website mirroring

ssh authentication protocol based on rsync

Before using rsync command to synchronize system files, log in to remote host for authentication. There are two protocols used in the authentication process:

• ssh protocol
• rsync protocol

The rsync server does not need to start the daemon process of rsync. You can directly synchronize files with rsync by obtaining the user name and password of remote host
The rsync server does not need to start the daemon process, so the configuration file / etc/rsyncd.conf is not needed

The principle of ssh authentication protocol is the same as that of scp. If you don't want to enter a password during synchronization, use ssh keygen - t RSA to open the channel

//This method omits - e ssh by default, which is equivalent to the following:
rsync -avz /SRC -e ssh root@192.168.126.128:/DEST 
    -a  //The file host changes and the timestamp remains unchanged
    -z  //Compressed data transmission
 
//When the port needs to be modified, we can:
rsync -avz /SRC -e "ssh -p2222" root@192.168.126.128:/DEST  
//The ssh protocol port is modified. The default is 22

[root@129 ~]# yum -y install rsync
[root@128 ~]# rsync -avz anaconda-ks.cfg root@192.168.126.150:/tmp/
The authenticity of host '192.168.126.150 (192.168.126.150)' can't be established.
ECDSA key fingerprint is SHA256:R1sHsPUKGqzvhsHbbdaEr0NcNxutf4OEUT3JuAss6m4.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.126.150' (ECDSA) to the list of known hosts.
root@192.168.126.150's password: 
sending incremental file list
anaconda-ks.cfg

sent 903 bytes  received 35 bytes  170.55 bytes/sec
total size is 1,396  speedup is 1.49

[root@129 ~]# ls /tmp/
anaconda-ks.cfg

rsync command

//There are three common command formats for Rsync:
    1,rsync [OPTION]... SRC DEST      // Locally, it is equivalent to copying files

1)Copy local files. When SRC and DES Path information does not contain a single colon":"This mode of operation is started when the separator is.
[root@128 ~]# rsync -avz anaconda-ks.cfg abc / / synchronize files with different names locally
sending incremental file list

sent 53 bytes  received 12 bytes  130.00 bytes/sec
total size is 1,396  speedup is 21.48
[root@128 ~]# ll
-rw-------. 1 root root 1396 3 January 2021 abc
-rw-------. 1 root root 1396 3 January 2021 anaconda-ks.cfg

[root@128 ~]# rsync -avz abc /tmp / / / synchronize files with the same name locally
sending incremental file list
abc

sent 891 bytes  received 35 bytes  1,852.00 bytes/sec
total size is 1,396  speedup is 1.51
[root@128 ~]# ll abc /tmp/abc 
-rw-------. 1 root root 1396 3 January 2021 abc
-rw-------. 1 root root 1396 3 January 2021 /tmp/abc

    2,rsync [OPTION]... SRC [USER@]HOST:DEST    // Put the local files on the remote host

2)Using a remote shell program(as rsh,ssh)To copy the content of the local machine to the remote machine. When DST Path address packet \
With single colon":"Start this mode when the delimiter.
[root@128 ~]# rsync -avz anaconda-ks.cfg root@192.168.126.150:/tmp/
The authenticity of host '192.168.126.150 (192.168.126.150)' can't be established.
ECDSA key fingerprint is SHA256:R1sHsPUKGqzvhsHbbdaEr0NcNxutf4OEUT3JuAss6m4.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.126.150' (ECDSA) to the list of known hosts.
root@192.168.126.150's password: 
sending incremental file list
anaconda-ks.cfg

sent 903 bytes  received 35 bytes  170.55 bytes/sec
total size is 1,396  speedup is 1.49

[root@129 ~]# ls /tmp/
anaconda-ks.cfg

    3,rsync [OPTION]... [USER@]HOST:SRC DEST   // Put the files on the remote host locally
    
3)Using a remote shell program(as rsh,ssh)To copy the content of the remote machine to the local machine. When SRC Address path \
Contains a single colon":"Start this mode when the delimiter.
[root@128 ~]# ls 
Public video document music  initial-setup-ks.cfg
 Template picture download desktop
[root@128 ~]# rsync -avz  root@192.168.126.150 : / TMP / Anaconda ks.cfg. / / synchronize files
root@192.168.126.150's password: 
receiving incremental file list
anaconda-ks.cfg

sent 43 bytes  received 903 bytes  378.40 bytes/sec
total size is 1,396  speedup is 1.48
[root@128 ~]# ls
 Public video document music  anaconda-ks.cfg
 Template picture download desktop  initial-setup-ks.cfg

//Synchronize directory
[root@129 ~]# ls /opt/

[root@128 ~]# rsync -avz /tmp root@192.168.126.150:/opt/   
root@192.168.126.150's password: 
sending incremental file list
tmp/
tmp/abc
tmp/.ICE-unix/
tmp/.ICE-unix/1780
tmp/.ICE-unix/2182
tmp/.X11-unix/
tmp/.X11-unix/X0
tmp/.X11-unix/X1024
tmp/.esd-0/
tmp/.esd-0/socket

[root@129 ~]# ls /opt/
tmp

//rsync common options:
    -a, --archive       //file
    -v, --verbose       //Verbose mode
    -q, --quiet         //silent mode 
    -r, --recursive     //recursion
    -p, --perms         //Keep the original permission attribute
    -z, --compress      //Compress during transmission to save bandwidth and speed up transmission
    --delete            //Deletions made on the source server are also synchronized on the target server

[root@128 ~]# ls /tmp/

[root@128 ~]# rsync -avz --delete /tmp root@192.168.126.150:/opt/
root@192.168.126.150's password: 
sending incremental file list
deleting tmp/abc

sent 1,088 bytes  received 46 bytes  2,268.00 bytes/sec
total size is 0  speedup is 0.00

[root@129 ~]# ls /opt/tmp/

rsync+inotify

Compared with the traditional cp and tar backup methods, rsync has the advantages of high security, fast backup and supporting incremental backup. rsync can solve the data backup requirements with low real-time requirements, such as regularly backing up the file server data to the remote server, regularly mirroring the local disk, etc.

With the continuous expansion of the scale of the application system, there are better requirements for the security and reliability of data. rsync has gradually exposed many deficiencies in the high-end business system. First, when rsync synchronizes data, it needs to scan all files for comparison and differential transmission. If the number of files reaches the order of millions or even tens of millions, scanning all files will be very time-consuming. And what is changing is often a small part of it, which is a very inefficient way. Secondly, rsync can't monitor and synchronize data in real time. Although it can trigger synchronization through the linux daemon, there must be a time difference between the two trigger actions, which may lead to inconsistency between the server and client data and can't completely recover the data in case of application failure. For the above reasons, the rsync+inotify combination appears!

Inotify is a powerful, fine-grained and asynchronous file system event monitoring mechanism. Since 2.6.13, the linux kernel has added inotify support. Inotify can monitor various subtle events such as addition, deletion, modification and movement in the file system. Using this kernel interface, third-party software can monitor various changes of files in the file system, Inotify tools is such a third-party software.

As mentioned earlier, rsync can realize triggered file synchronization, but triggered by crontab daemon, the synchronized data will be different from the actual data. inotify can monitor various changes in the file system and trigger rsync synchronization when there is any change in the file, which just solves the real-time problem of synchronized data.
Environmental description:

Requirements:

Synchronize the / etc directory on the source server to / tmp / on the target server in real time
Do the following on the target server:

//Turn off firewall and SELINUX
[root@129 ~]# systemctl disabled --now firewalld.service
[root@129 ~]# getenforce 0
Disabled

//Install rsync server software
[root@129 ~]# yum -y install rsync
[root@129 ~]# yum -y install rsync-daemon

//Set rsyncd.conf configuration file
[root@129 ~]# cat /etc/rsyncd.conf 
log file = /var/log/rsyncd.log    # Log file location. This file will be generated automatically after rsync is started. There is no need to create it in advance
pidfile = /var/run/rsyncd.pid     # Storage location of pid files
lock file = /var/run/rsync.lock   # Lock files that support the max connections parameter
secrets file = /etc/rsync.pass    # User authentication profile, which stores user name and password, must be created manually

[etc_from_client]     # Custom sync name
path = /WJJ/          # rsync server data storage path, and client data will be synchronized to this directory
comment = sync etc from client
uid = root        # Set rsync running permission to root
gid = root        # Set rsync running permission to root
port = 873        # Default port
ignore errors     # Indicates that an error has occurred. Ignore the error
use chroot = no       # The default value is true and modified to no. the soft connection backup of directory files is added
read only = no    # Set the rsync server to read / write permission
list = no     # The rsync server resource list is not displayed
max connections = 200     # maximum connection
timeout = 600     # Set timeout
auth users = admin        # Multiple user names for data synchronization can be set, separated by commas in English

//Create user authentication file
[root@129 ~]# Echo 'admin: 123456' > / etc / rsync.pass / / admin is a virtual fake user
[root@129 ~]# cat /etc/rsync.pass
admin:123456

//set files permissions
[root@129 ~]# chmod 600 /etc/rsync*

//Start the rsync service and set the startup self startup
[root@129 ~]# systemctl enable --now rsyncd
Created symlink /etc/systemd/system/multi-user.target.wants/rsyncd.service → /usr/lib/systemd/system/rsyncd.service.
[root@129 ~]# ss -antl
State   Recv-Q  Send-Q   Local Address:Port     Peer Address:Port  
LISTEN  0       5              0.0.0.0:873           0.0.0.0:*

Do the following on the source server:

//Turn off firewall and SELINUX
[root@128 ~]# systemctl disabled --now firewalld.service
[root@128 ~]# getenforce 0
Disabled

//Configure yum source
 Check the system version and ensure that the system version and epel Version matching
[root@128 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.2 (Ootpa)
If the result is Red Hat Enterprise Linux release 8.***，You need to select epel Version 8.

get into epel Official website https://dl.fedoraproject.org/pub/epel, download the EPEL installation package of the corresponding version
[root@128 ~]#  wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
--2021-10-11 18:41:58--  https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
 Resolving host dl.fedoraproject.org (dl.fedoraproject.org)... 38.145.60.23, 38.145.60.24, 38.145.60.22
 on connection dl.fedoraproject.org (dl.fedoraproject.org)|38.145.60.23|:443... Connected.
Issued HTTP Request, waiting for response... 200 OK
 Length: 23644 (23K) [application/x-rpm]
Saving to: "epel-release-latest-8.noarch.rpm"

epel-release-lat 100%[=========>]  23.09K  62.0KB/s  Time 0.4s    

2021-10-11 18:42:00 (62.0 KB/s) - Saved“ epel-release-latest-8.noarch.rpm" [23644/23644])

Execute the following command to install the related epel(This example uses epel (version 8)
[root@128 ~]# rpm -vih epel-release-latest-8.noarch.rpm
 Warning: epel-release-latest-8.noarch.rpm: head V4 RSA/SHA256 Signature, secret key ID 2f86d6a1: NOKEY
Verifying...                          ################################# [100%]
In preparation...                          ################################# [100%]
Upgrading/install...
   1:epel-release-8-13.el8            ################################# [100%]
[root@128 ~]# ls /etc/yum.repos.d/
epel-modular.repo     epel-testing-modular.repo  wjj.repo
epel-playground.repo  epel-testing.repo
epel.repo             redhat.repo

//To install rsync server software, you only need to install, do not start, and do not need to configure
[root@128 ~]# yum -y install rsync

//Create authentication password file
[root@128 ~]# echo '123456' > /etc/rsync.pass

//Set file permissions. Only the file owner has read and write permissions
[root@128 ~]# chmod 600 /etc/rsync.pass

//Create a test directory on the source server, and then run the following command on the source server
[root@128 ~]# mkdir /runtime

[root@128 ~]# rsync -avH --port 873 --progress --delete /runtime admin@192.168.47.129::etc_from_client --password-file=/etc/rsync.pass
sending incremental file list

sent 86 bytes  received 21 bytes  214.00 bytes/sec
total size is 0  speedup is 0.00

//After running, check on the target server. There is a runtime directory under the / WJJ directory, indicating that the data synchronization is successful
[root@129 WJJ]# ls 
runtime

//Install inotify tools tool to trigger rsync for synchronization in real time
//Installing inotify tools
[root@128 ~]# yum -y install inotify-tools

//Writing synchronization script is the most important step. Please be careful. Let the script automatically detect the directory we set\
//File changes, and then execute the rsync command to synchronize it to our server
[root@128 ~]# mkdir /scripts
[root@128 ~]# cd /scripts/
[root@128 scripts]# cat inotify.sh 
host=192.168.126.150  
src=/runtime     
des=etc_from_client   
password=/etc/rsync.pass   
user=admin        
inotifywait=/usr/bin/inotifywait

$inotifywait -mrq --timefmt '%Y%m%d %H:%M' --format '%T %w%f%e' -e modify,delete,create,attrib $src \
| while read files;do
    rsync -avzP --delete  --timeout=100 --password-file=${password} $src $user@$host::$des
    echo "${files} was rsynced" >>/WJJ/rsync.log 2>&1     //0 STDIN refers to standard input; 1 STDOUT refers to standard output; 3 STDERR refers to standard error
done

[root@128 scripts]# chmod +x inotify.sh

[root@129 ~]# rm -rf /WJJ/*

//Manual trigger
[root@128 ~]# touch /runtime/abc

[root@128 scripts]# ./inotify.sh

[root@129 ~]# ll /WJJ/runtime/
Total consumption 0
-rw-r--r--. 1 root root 0 10 November 20:11 abc
-rw-r--r--. 1 root root 0 10 November 18:51 test

[root@128 ~]# /Scripts / inotify.sh & / / run in the background
[1] 217021
[root@128 ~]# ps -ef|grep inotify
root      217022  217021  0 20:16 pts/1    00:00:00 /usr/bin/inotifywait -mrq --timefmt %Y%m%d %H:%M --format %T %w%f%e -e modify,delete,create,attrib /runtime
root      218019    3998  0 20:16 pts/1    00:00:00 grep --color=auto inotify
//As you can see from the log, we generated a test file and added content to it

Set script startup to start automatically:

[root@128 ~]# ll /etc/rc.local 
lrwxrwxrwx. 1 root root 13 3 June 24, 2020 /etc/rc.local -> rc.d/rc.local    //The last file to execute after system startup

[root@128 ~]# chmod +x /etc/rc.d/rc.local 
[root@128 ~]# ll /etc/rc.local 
lrwxrwxrwx. 1 root root 13 3 June 24, 2020 /etc/rc.local -> rc.d/rc.local

[root@128 ~]# vim /etc/rc.d/rc.local
nohup /scripts/inotify.sh &    //Add this line and nohup to get away from the terminal
touch /var/lock/subsys/local

[root@128 ~]# reboot

[root@128 ~]# ps -ef|grep inotify
root        1064       1  0 04:46 ?        00:00:00 /bin/sh /scripts/inotify.sh
root        1073    1064  0 04:46 ?        00:00:00 /usr/bin/inotifywait -mrq --timefmt %Y%m%d %H:%M --format %T %w%f%e -e modify,delete,create,attrib /runtime
root        1074    1064  0 04:46 ?        00:00:00 /bin/sh /scripts/inotify.sh
root        7375    1614  0 04:47 pts/2    00:00:00 grep --color=auto inotify