Introduction to rsync
rsync is a data image backup tool under linux system. Using the fast incremental backup tool Remote Sync, you can synchronize remotely, support local replication, or synchronize with other SSH and rsync hosts. Synchronize massive small files
rsync feature
rsync supports many features:
- You can mirror and save the entire directory tree and file system
- It is easy to maintain the permissions, time, soft and hard links of the original files
- Installation without special permission
- Fast: rsync will copy all the contents during the first synchronization, but only the modified files will be transferred next time. rsync can compress and decompress data, so it can use less bandwidth
- Security: you can use scp, ssh and other methods to transfer files. Of course, you can also connect directly through socket [socket] (IP + port number)
- Support anonymous transmission to facilitate website mirroring
ssh authentication protocol based on rsync
Before using rsync command to synchronize system files, log in to remote host for authentication. There are two protocols used in the authentication process:
- ssh protocol
- rsync protocol
The rsync server does not need to start the daemon process of rsync. You can directly synchronize files with rsync by obtaining the user name and password of remote host
The rsync server does not need to start the daemon process, so the configuration file / etc/rsyncd.conf is not needed
The principle of ssh authentication protocol is the same as that of scp. If you don't want to enter a password during synchronization, use ssh keygen - t RSA to open the channel
//This method omits - e ssh by default, which is equivalent to the following: rsync -avz /SRC -e ssh root@192.168.126.128:/DEST -a //The file host changes and the timestamp remains unchanged -z //Compressed data transmission //When the port needs to be modified, we can: rsync -avz /SRC -e "ssh -p2222" root@192.168.126.128:/DEST //The ssh protocol port is modified. The default is 22
[root@129 ~]# yum -y install rsync [root@128 ~]# rsync -avz anaconda-ks.cfg root@192.168.126.150:/tmp/ The authenticity of host '192.168.126.150 (192.168.126.150)' can't be established. ECDSA key fingerprint is SHA256:R1sHsPUKGqzvhsHbbdaEr0NcNxutf4OEUT3JuAss6m4. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '192.168.126.150' (ECDSA) to the list of known hosts. root@192.168.126.150's password: sending incremental file list anaconda-ks.cfg sent 903 bytes received 35 bytes 170.55 bytes/sec total size is 1,396 speedup is 1.49 [root@129 ~]# ls /tmp/ anaconda-ks.cfg
rsync command
//There are three common command formats for Rsync: 1,rsync [OPTION]... SRC DEST // Locally, it is equivalent to copying files 1)Copy local files. When SRC and DES Path information does not contain a single colon":"This mode of operation is started when the separator is. [root@128 ~]# rsync -avz anaconda-ks.cfg abc / / synchronize files with different names locally sending incremental file list sent 53 bytes received 12 bytes 130.00 bytes/sec total size is 1,396 speedup is 21.48 [root@128 ~]# ll -rw-------. 1 root root 1396 3 January 2021 abc -rw-------. 1 root root 1396 3 January 2021 anaconda-ks.cfg [root@128 ~]# rsync -avz abc /tmp / / / synchronize files with the same name locally sending incremental file list abc sent 891 bytes received 35 bytes 1,852.00 bytes/sec total size is 1,396 speedup is 1.51 [root@128 ~]# ll abc /tmp/abc -rw-------. 1 root root 1396 3 January 2021 abc -rw-------. 1 root root 1396 3 January 2021 /tmp/abc 2,rsync [OPTION]... SRC [USER@]HOST:DEST // Put the local files on the remote host 2)Using a remote shell program(as rsh,ssh)To copy the content of the local machine to the remote machine. When DST Path address packet \ With single colon":"Start this mode when the delimiter. [root@128 ~]# rsync -avz anaconda-ks.cfg root@192.168.126.150:/tmp/ The authenticity of host '192.168.126.150 (192.168.126.150)' can't be established. ECDSA key fingerprint is SHA256:R1sHsPUKGqzvhsHbbdaEr0NcNxutf4OEUT3JuAss6m4. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '192.168.126.150' (ECDSA) to the list of known hosts. root@192.168.126.150's password: sending incremental file list anaconda-ks.cfg sent 903 bytes received 35 bytes 170.55 bytes/sec total size is 1,396 speedup is 1.49 [root@129 ~]# ls /tmp/ anaconda-ks.cfg 3,rsync [OPTION]... [USER@]HOST:SRC DEST // Put the files on the remote host locally 3)Using a remote shell program(as rsh,ssh)To copy the content of the remote machine to the local machine. When SRC Address path \ Contains a single colon":"Start this mode when the delimiter. [root@128 ~]# ls Public video document music initial-setup-ks.cfg Template picture download desktop [root@128 ~]# rsync -avz root@192.168.126.150 : / TMP / Anaconda ks.cfg. / / synchronize files root@192.168.126.150's password: receiving incremental file list anaconda-ks.cfg sent 43 bytes received 903 bytes 378.40 bytes/sec total size is 1,396 speedup is 1.48 [root@128 ~]# ls Public video document music anaconda-ks.cfg Template picture download desktop initial-setup-ks.cfg //Synchronize directory [root@129 ~]# ls /opt/ [root@128 ~]# rsync -avz /tmp root@192.168.126.150:/opt/ root@192.168.126.150's password: sending incremental file list tmp/ tmp/abc tmp/.ICE-unix/ tmp/.ICE-unix/1780 tmp/.ICE-unix/2182 tmp/.X11-unix/ tmp/.X11-unix/X0 tmp/.X11-unix/X1024 tmp/.esd-0/ tmp/.esd-0/socket [root@129 ~]# ls /opt/ tmp //rsync common options: -a, --archive //file -v, --verbose //Verbose mode -q, --quiet //silent mode -r, --recursive //recursion -p, --perms //Keep the original permission attribute -z, --compress //Compress during transmission to save bandwidth and speed up transmission --delete //Deletions made on the source server are also synchronized on the target server [root@128 ~]# ls /tmp/ [root@128 ~]# rsync -avz --delete /tmp root@192.168.126.150:/opt/ root@192.168.126.150's password: sending incremental file list deleting tmp/abc sent 1,088 bytes received 46 bytes 2,268.00 bytes/sec total size is 0 speedup is 0.00 [root@129 ~]# ls /opt/tmp/
rsync+inotify
Compared with the traditional cp and tar backup methods, rsync has the advantages of high security, fast backup and supporting incremental backup. rsync can solve the data backup requirements with low real-time requirements, such as regularly backing up the file server data to the remote server, regularly mirroring the local disk, etc.
With the continuous expansion of the scale of the application system, there are better requirements for the security and reliability of data. rsync has gradually exposed many deficiencies in the high-end business system. First, when rsync synchronizes data, it needs to scan all files for comparison and differential transmission. If the number of files reaches the order of millions or even tens of millions, scanning all files will be very time-consuming. And what is changing is often a small part of it, which is a very inefficient way. Secondly, rsync can't monitor and synchronize data in real time. Although it can trigger synchronization through the linux daemon, there must be a time difference between the two trigger actions, which may lead to inconsistency between the server and client data and can't completely recover the data in case of application failure. For the above reasons, the rsync+inotify combination appears!
Inotify is a powerful, fine-grained and asynchronous file system event monitoring mechanism. Since 2.6.13, the linux kernel has added inotify support. Inotify can monitor various subtle events such as addition, deletion, modification and movement in the file system. Using this kernel interface, third-party software can monitor various changes of files in the file system, Inotify tools is such a third-party software.
As mentioned earlier, rsync can realize triggered file synchronization, but triggered by crontab daemon, the synchronized data will be different from the actual data. inotify can monitor various changes in the file system and trigger rsync synchronization when there is any change in the file, which just solves the real-time problem of synchronized data.
Environmental description:
Server type | IP address | application | operating system |
---|---|---|---|
Source server | 192.168.126.128 | Rsync inotify tools script | centos7/redhat7 |
Target server | 192.168.126.150 | rsync | centos7/redhat7 |
Requirements:
Synchronize the / etc directory on the source server to / tmp / on the target server in real time
Do the following on the target server:
//Turn off firewall and SELINUX [root@129 ~]# systemctl disabled --now firewalld.service [root@129 ~]# getenforce 0 Disabled //Install rsync server software [root@129 ~]# yum -y install rsync [root@129 ~]# yum -y install rsync-daemon //Set rsyncd.conf configuration file [root@129 ~]# cat /etc/rsyncd.conf log file = /var/log/rsyncd.log # Log file location. This file will be generated automatically after rsync is started. There is no need to create it in advance pidfile = /var/run/rsyncd.pid # Storage location of pid files lock file = /var/run/rsync.lock # Lock files that support the max connections parameter secrets file = /etc/rsync.pass # User authentication profile, which stores user name and password, must be created manually [etc_from_client] # Custom sync name path = /WJJ/ # rsync server data storage path, and client data will be synchronized to this directory comment = sync etc from client uid = root # Set rsync running permission to root gid = root # Set rsync running permission to root port = 873 # Default port ignore errors # Indicates that an error has occurred. Ignore the error use chroot = no # The default value is true and modified to no. the soft connection backup of directory files is added read only = no # Set the rsync server to read / write permission list = no # The rsync server resource list is not displayed max connections = 200 # maximum connection timeout = 600 # Set timeout auth users = admin # Multiple user names for data synchronization can be set, separated by commas in English //Create user authentication file [root@129 ~]# Echo 'admin: 123456' > / etc / rsync.pass / / admin is a virtual fake user [root@129 ~]# cat /etc/rsync.pass admin:123456 //set files permissions [root@129 ~]# chmod 600 /etc/rsync* //Start the rsync service and set the startup self startup [root@129 ~]# systemctl enable --now rsyncd Created symlink /etc/systemd/system/multi-user.target.wants/rsyncd.service → /usr/lib/systemd/system/rsyncd.service. [root@129 ~]# ss -antl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 5 0.0.0.0:873 0.0.0.0:*
Do the following on the source server:
//Turn off firewall and SELINUX [root@128 ~]# systemctl disabled --now firewalld.service [root@128 ~]# getenforce 0 Disabled //Configure yum source Check the system version and ensure that the system version and epel Version matching [root@128 ~]# cat /etc/redhat-release Red Hat Enterprise Linux release 8.2 (Ootpa) If the result is Red Hat Enterprise Linux release 8.***,You need to select epel Version 8. get into epel Official website https://dl.fedoraproject.org/pub/epel, download the EPEL installation package of the corresponding version [root@128 ~]# wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm --2021-10-11 18:41:58-- https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm Resolving host dl.fedoraproject.org (dl.fedoraproject.org)... 38.145.60.23, 38.145.60.24, 38.145.60.22 on connection dl.fedoraproject.org (dl.fedoraproject.org)|38.145.60.23|:443... Connected. Issued HTTP Request, waiting for response... 200 OK Length: 23644 (23K) [application/x-rpm] Saving to: "epel-release-latest-8.noarch.rpm" epel-release-lat 100%[=========>] 23.09K 62.0KB/s Time 0.4s 2021-10-11 18:42:00 (62.0 KB/s) - Saved“ epel-release-latest-8.noarch.rpm" [23644/23644]) Execute the following command to install the related epel(This example uses epel (version 8) [root@128 ~]# rpm -vih epel-release-latest-8.noarch.rpm Warning: epel-release-latest-8.noarch.rpm: head V4 RSA/SHA256 Signature, secret key ID 2f86d6a1: NOKEY Verifying... ################################# [100%] In preparation... ################################# [100%] Upgrading/install... 1:epel-release-8-13.el8 ################################# [100%] [root@128 ~]# ls /etc/yum.repos.d/ epel-modular.repo epel-testing-modular.repo wjj.repo epel-playground.repo epel-testing.repo epel.repo redhat.repo //To install rsync server software, you only need to install, do not start, and do not need to configure [root@128 ~]# yum -y install rsync //Create authentication password file [root@128 ~]# echo '123456' > /etc/rsync.pass //Set file permissions. Only the file owner has read and write permissions [root@128 ~]# chmod 600 /etc/rsync.pass //Create a test directory on the source server, and then run the following command on the source server [root@128 ~]# mkdir /runtime [root@128 ~]# rsync -avH --port 873 --progress --delete /runtime admin@192.168.47.129::etc_from_client --password-file=/etc/rsync.pass sending incremental file list sent 86 bytes received 21 bytes 214.00 bytes/sec total size is 0 speedup is 0.00 //After running, check on the target server. There is a runtime directory under the / WJJ directory, indicating that the data synchronization is successful [root@129 WJJ]# ls runtime //Install inotify tools tool to trigger rsync for synchronization in real time //Installing inotify tools [root@128 ~]# yum -y install inotify-tools //Writing synchronization script is the most important step. Please be careful. Let the script automatically detect the directory we set\ //File changes, and then execute the rsync command to synchronize it to our server [root@128 ~]# mkdir /scripts [root@128 ~]# cd /scripts/ [root@128 scripts]# cat inotify.sh host=192.168.126.150 src=/runtime des=etc_from_client password=/etc/rsync.pass user=admin inotifywait=/usr/bin/inotifywait $inotifywait -mrq --timefmt '%Y%m%d %H:%M' --format '%T %w%f%e' -e modify,delete,create,attrib $src \ | while read files;do rsync -avzP --delete --timeout=100 --password-file=${password} $src $user@$host::$des echo "${files} was rsynced" >>/WJJ/rsync.log 2>&1 //0 STDIN refers to standard input; 1 STDOUT refers to standard output; 3 STDERR refers to standard error done [root@128 scripts]# chmod +x inotify.sh [root@129 ~]# rm -rf /WJJ/* //Manual trigger [root@128 ~]# touch /runtime/abc [root@128 scripts]# ./inotify.sh [root@129 ~]# ll /WJJ/runtime/ Total consumption 0 -rw-r--r--. 1 root root 0 10 November 20:11 abc -rw-r--r--. 1 root root 0 10 November 18:51 test [root@128 ~]# /Scripts / inotify.sh & / / run in the background [1] 217021 [root@128 ~]# ps -ef|grep inotify root 217022 217021 0 20:16 pts/1 00:00:00 /usr/bin/inotifywait -mrq --timefmt %Y%m%d %H:%M --format %T %w%f%e -e modify,delete,create,attrib /runtime root 218019 3998 0 20:16 pts/1 00:00:00 grep --color=auto inotify //As you can see from the log, we generated a test file and added content to it
Set script startup to start automatically:
[root@128 ~]# ll /etc/rc.local lrwxrwxrwx. 1 root root 13 3 June 24, 2020 /etc/rc.local -> rc.d/rc.local //The last file to execute after system startup [root@128 ~]# chmod +x /etc/rc.d/rc.local [root@128 ~]# ll /etc/rc.local lrwxrwxrwx. 1 root root 13 3 June 24, 2020 /etc/rc.local -> rc.d/rc.local [root@128 ~]# vim /etc/rc.d/rc.local nohup /scripts/inotify.sh & //Add this line and nohup to get away from the terminal touch /var/lock/subsys/local [root@128 ~]# reboot [root@128 ~]# ps -ef|grep inotify root 1064 1 0 04:46 ? 00:00:00 /bin/sh /scripts/inotify.sh root 1073 1064 0 04:46 ? 00:00:00 /usr/bin/inotifywait -mrq --timefmt %Y%m%d %H:%M --format %T %w%f%e -e modify,delete,create,attrib /runtime root 1074 1064 0 04:46 ? 00:00:00 /bin/sh /scripts/inotify.sh root 7375 1614 0 04:47 pts/2 00:00:00 grep --color=auto inotify