The guidelines are divided into three steps.
- Upload compressed package from local to platform personal data
- Download the platform personal data to the instance
- Conduct training and automatically upload results and shut down
The operation process is as follows:
Upload personal data locally
First, the training data is typed locally zip,tar.gz Compressed packages in common formats. Then download and install locally oss command line tool , use this tool to upload the compressed package to the platform Personal data Yes.
The Windows system is used here as the local environment. For other systems, please refer to oss command line tool Install.
download OSS (Windows) Executable file. When the download is complete, the oss_windows_x86_64.exe Change the file name to oss.exe. Open CMD (command prompt) or PowerShell under Windows.
use cd Command switch to oss.exe Directory, here oss.exe Saved in D:\Download Down. In this case, the PowerShell terminal is used for execution.
PS C:\Users\windows> cd D:\Download PS D:\Download>
Execute in this directory .\ oss This command can be executed. You need to log in before uploading. The login account is the account name and password of Hengyuan cloud platform, not an instance. The mobile phone number Chinese name is called. If it is the mobile phone number in Chinese mainland, it needs to add the country number with +.
# Execute. \ oss login PS D:\Download> .\oss login Username:139******** Password:*********** 139******** login successfully!
After successful login, the file transfer operation will be carried out, and the locally printed compressed package will be uploaded to Personal data Yes. Note that the uploaded file extension only supports the common compressed package format. The path of the compressed package here is D:\Datasets\MNIST.zip.
# Execute. \ oss cp D:\Datasets\MNIST.zip oss:// PS D:\Download> .\oss cp D:\Datasets\MNIST.zip oss:// Start at 2021-10-22 06:21:48.2061924 +0000 UTC [-------------------------------------] 100.00% 52.34MB/s 76.55MB/76.55MB 1.53s Upload successfully, 76.55MB, n/a, D:\Datasets\MNIST.zip --> oss://MNIST.zip, cost [7020], status [200], request id [0000017CA6A9BE01901395D26CE9A228]
On the platform after successful upload Personal data The file can be viewed on the page.
Instance Download Personal Data
After an instance is created or started on the platform, the JupyterLab Enter the terminal or SSH client Connect the terminal.
It is also used in the terminal oss Log in first. The login account is the account name and password of Hengyuan cloud platform, not an instance. The mobile phone number Chinese name is called. If it is the mobile phone number in Chinese mainland, it needs to add the country number with +.
# Execute oss login ~# oss login Username:139******** Password:*********** 139******** login successfully!
After successful login, download the file and download the file just uploaded to / hy-tmp Medium (available for models with shared storage) / hy-nas).
# Execute oss cp oss://MNIST.zip /hy-tmp ~# oss cp oss://MNIST.zip /hy-tmp Start at 2021-10-22 06:37:17.227649376 +0000 UTC [-------------------------------------] 100.00% 62.85MB/s 76.55MB/76.55MB 1.42s Waiting to rename temporary file... Download successfully, 76.55MB, n/a, oss://MNIST.zip --> /hy-tmp/MNIST.zip, cost [2422], status [206], request id [0000017CA6B7D6419012DB7767D11959]
After the download is successful, enter the download path and decompress the compressed package.
# Execute CD / HY TMP to enter this directory ~# cd /hy-tmp # Execute unzip -q MNIST.zip to unzip the package /hy-tmp# unzip -q MNIST.zip
Shut down after performing training and automatically uploading results
The data already exists in the instance, and the next step is the training stage. have access to JupyterLab or VSCode And other tools. Note that the data path referenced in the code needs to be changed to the data path within the instance. In the above example, the / hy-tmp. Long training tasks need to pass Tmux Run in the background so that you can turn off your local computer without interrupting your training.
Pay as you go instances can upload results to personal data after training is completed through the following operations, and then shut down to terminate billing. This operation can save the use cost of pay as you go and avoid the damage caused by the occupation of the machine / hy-tmp Your data cannot be viewed and downloaded.
The operation of the sample script requires the following preconditions, and some operations are adjusted according to actual needs.
- The results of the training are / hy-tmp/result in
- In the instance, the platform account, mobile number and password need to be used in advance oss login Successful login
stay / root Create a directory using vim upload.sh script.
cd vim upload.sh
The script is as follows, using vim Editor, entering Enter the insert mode, paste down the entire script content, and then enter
preservation.
#!/bin/bash set -e cd /hy-tmp # Package name file="result-$(date "+%Y%m%d-%H%M%S").zip" # Make the result directory into a zip package zip -q -r "${file}" result # Upload to the backup folder in your personal data through oss oss cp "${file}" oss://backup/ rm -f "${file}" # Shutdown after successful transmission shutdown
Add execution permission to the script.
chmod u+x upload.sh
This script implements the / hy-tmp/result Make it into a zip package, and then upload this file to your personal data. Execute the shutdown operation after successful execution.
The last call in the training task code. / root/upload.sh The script can be run. It is recommended to test it when it is used for the first time. If referenced in Python code:
import os os.system('/root/upload.sh')
It can be used locally after the training execution is completed and shut down oss The command line tool downloads the packaged result file. When the instance machine is occupied and cannot be started, you can also create a new instance on other machines and start it through oss Download the data to the new instance. Specific about oss Relevant operation reference documents oss command tool.
# Log in to the Hengyuan cloud account and use the account name and password of Hengyuan cloud. The account name is the mobile phone number # If the mobile phone number is in Chinese mainland, it is necessary to add a zone code with +. ~# oss login Username:139******** Password:*********** 139******** login successfully! # View files and folders, - s means that only file names are displayed ~# oss ls -s oss:// Folder list: oss:// oss://backup/ Object list: oss://backup/result-20211018-164323.zip # Download the result-20211018-164323.zip file in personal data to the current directory ~# oss cp oss://backup/result-20211018-164323.zip . Download successfully, 310B, n/a, oss://backup/result-20211018-164323.zip --> /root/result-20211018-164323.zip, cost [57], status [200], request id [0000017C92929EF49014BE16738685B7]