Python 3.9 automatically downloads alicloud RDS on a regular basis and backs it up locally

1, Business application demands

  • The method described in this paper is practical in the same environment after real business practice.
  • Background note: the business system is deployed in Alibaba cloud, but it can be downloaded to the local server every day for database backup. The previous practice was to log in to Alibaba cloud by a specially assigned person every day, first obtain the download link manually, and then download it locally with Xunlei. Because it is a pure manual operation, there is a problem that it is often forgotten and not in time when the affairs are complicated. And often download during working hours, occupying the company's network bandwidth and affecting normal business use. Therefore, the appeal is to realize regular automatic download and avoid working hours.
  • Our method refers to Alibaba cloud's technical blog, but there are some improvements and instantiation of the operation process, which can be used for reference by friends with similar demands.
    Reference technology blog on Alibaba cloud
    Specific address: https://help.aliyun.com/knowledge_detail/55515.html?spm=5176.10695662.1996646101.searchclickresult.43bf3e75uTcoXB
  • The environment of the method described in this paper is Windows 10 and the prepared Python 3.9 environment.

2, Install aliyun Python SDK core

Execute PIP3 install aliyun Python SDK core in the cmd command window

E:\python\pm_py>pip3 install aliyun-python-sdk-core

A message like "Successfully installed aliyun-python-sdk-core-2.13.35" appears, indicating that the installation is successful.

3, Install aliyun Python SDK RDS

Execute PIP3 install aliyun Python SDK RDS in the cmd command window

E:\python\pm_py>pip3 install aliyun-python-sdk-rds

A message like "Successfully installed aliyun-python-sdk-rds-2.5.11" appears, indicating that the installation is successful.

4, Run get_rds_backup.py script file

Run get with reference to the following standard commands_ rds_ Backup.py script file.

python get_rds_backup.py [$RDS_ID] [$Access_Key_ID] [$Access_Key_Secret] [$Backup_Dir]

explain:
[RDS_ID]: the ID of the target RDS instance, which can be viewed on the basic information page of the RDS instance. For example, "rm-hp38xx2i07".
[Access_Key_ID]: the key ID of the RAM user. If you do not already have a RAM user, see creating a RAM user.
[Access_Key_Secret]: the key of the RAM user.
[Backup_Dir]: the directory where the backup is expected to be saved. Make sure there is enough disk space left to avoid download failure.
The backup of the previous day is downloaded by default. If you need to modify the time range, you can modify the corresponding starttime and endtime variable values in the script.

matters needing attention:

4.1 RDS_ Acquisition of ID

Log in to alicloud and click Copy in the instance list.

4.2 Access_Key_ID Access_Key_Secret and get

After logging in to alicloud, click image processing on the user account in the upper right corner, and then click AccessKey management to see it.

4.3 get_rds_backup.py changes

Alibaba technology blog_ rds_ Backup.py is applicable to Python 2.7, while our Python environment is 3.9. The downloaded documents need to be modified. Specifically
1) The print statement is enclosed in parentheses.
2) The import package needs to be replaced. Import urlib2 becomes import urlib.error and import urlib.request in Python 3.
3) The method to obtain the file length is provided by bak_size = int(jsload ['totalbackupsize']) becomes bak_size = int(jsload[‘TotalBackupSize’])
The code reference after completion of specific repair is as follows. Download link:

# zhanghc python learning files
# Development time: 12:48, September 15, 2021
# !/usr/bin/env python

import datetime
import json
import sys
import urllib.error
import urllib.request
from aliyunsdkcore import client
from aliyunsdkrds.request.v20140815.DescribeBackupsRequest import DescribeBackupsRequest

if (len(sys.argv) != 5):
	print ('use help:')
	print ('''
		get_rds_backup.py rm-xxxxxxx  xxxxxxx xxxxxxxxx  /mnt/
		get_rds_backup.py  RDS example ID   key  secret  Backup save location
		key: Access Key ID
		secret:Access Key Secret 
		Yesterday's backup is downloaded by default, and the time can be modified, corresponding to the variables in the script starttime and endtime
		''')
	sys.exit(0)

key = sys.argv[2]
key_secret = sys.argv[3]
rds_id = sys.argv[1]
back_path = sys.argv[4]
#back_path = "/mnt/"
yesterday = str(datetime.date.today() + datetime.timedelta(days=-1))
today = str(datetime.date.today())
starttime = yesterday + "T00:00Z"
print (yesterday,today)
endtime = today + "T00:00Z"
#key = ''
#key_secret = ''
region = "cn-qingdao"
#rds_id = ['rmmhlc,'rm-mc']


def download_rds_backfile(instanceid):

        clt = client.AcsClient(key, key_secret, region)
        req_bakup = DescribeBackupsRequest()
        req_bakup.set_DBInstanceId(instanceid)
        req_bakup.set_accept_format('json')
        req_bakup.set_StartTime(starttime)
        req_bakup.set_EndTime(endtime)
        backup = clt.do_action_with_exception(req_bakup)
        #  print (backup)
        jsload = json.loads(backup)
        num = jsload["PageRecordCount"]
        print ("backfiles:" + str(num))
        i = 0
        while i < num:

                bak_url = jsload["Items"]["Backup"][i]["BackupDownloadURL"]
                bak_host = jsload["Items"]["Backup"][i]["HostInstanceID"]
                bak_id = jsload["Items"]["Backup"][i]["BackupId"]
                print ("BackupId:" + str(bak_id), "HostInstanceID:" + str(bak_host), "downloadurl:" + bak_url)
                save_name = back_path + bak_url.split('?')[0].split('/')[-1]
                u = urllib.request.urlopen(bak_url)
                # u = urllib.request.urlopen(bak_url).read()
                f_header = u.info()
                # bak_size = int(f_header.getheaders("Content-Length")[0])
                bak_size = int(jsload['TotalBackupSize'])
                print ("backup file size: %s M ,fime nema: %s" % (bak_size / 1024 / 1024, save_name))

                with open(save_name, "wb") as f:

                        file_size_dl = 0
                        block_sz = 8192
                        j = 1;
                        while True:
                                buffer = u.read(block_sz)
                                if not buffer:
                                        break

                                file_size_dl += len(buffer)
                                f.write(buffer)
                                status = r"%10d  [%3.2f%%]" % (file_size_dl, file_size_dl * 100. / bak_size)
                                # status = status + chr(8) * (len(status) + 1)
                                if ((file_size_dl / bak_size) >= j): # Judge the progress and output the progress information once every 1% of the progress.
                                        print (status)
                                        j = j + 1
                i = i + 1
                print ("download complet!")
download_rds_backfile(rds_id)

4.4 test whether it can be downloaded normally under CMD

Enter the following command in the cmd command window:

 python get_rds_backup.py rm-m5ek8sk8s433uzhd7q1 LTAI5tLkk8sYUEb89qC BZaqix98DtWpsXkjlloqhf3ti E:\

If BackupId:1*****9 HostInstanceID information can be output, the program can be executed normally.

5, Create bat script file

With a text editor, enter the following similar content and save it as a bat file. The > > baklog.txt in the last line is to output the execution results to a TXT file and record the execution process.

E:
cd \python\pm_py
python get_rds_backup.py rm-m5ksk8sk8s1 LTAI4k8sk8sk8s9qC BZaqik8sk8sk8sti E:\ >> baklog.txt

6, Create a Windows scheduled task

Find and open the task scheduler in the windows start program.

Create a new scheduled task



In the planned task list, you can see the basic information of the planned task, such as last run time, next run time, etc.

Tags: Python Database rds

Posted on Sat, 18 Sep 2021 07:04:11 -0400 by pck76