The guidelines are divided into three steps.

• Upload compressed package from local to platform personal data
• Download the platform personal data to the instance
• Conduct training and automatically upload results and shut down

The operation process is as follows:

Upload personal data locally  

First, the training data is typed locally   zip,tar.gz   Compressed packages in common formats. Then download and install locally   oss command line tool , use this tool to upload the compressed package to the platform Personal data Yes.  

The Windows system is used here as the local environment. For other systems, please refer to   oss command line tool   Install.

download   OSS (Windows)   Executable file. When the download is complete, the   oss_windows_x86_64.exe   Change the file name to   oss.exe. Open CMD (command prompt) or PowerShell under Windows.

use   cd   Command switch to   oss.exe   Directory, here   oss.exe   Saved in   D:\Download   Down. In this case, the PowerShell terminal is used for execution.

PS C:\Users\windows> cd D:\Download
PS D:\Download>

Execute in this directory  .\ oss   This command can be executed. You need to log in before uploading. The login account is the account name and password of Hengyuan cloud platform, not an instance. The mobile phone number Chinese name is called. If it is the mobile phone number in Chinese mainland, it needs to add the country number with +.  

# Execute. \ oss login
PS D:\Download> .\oss login
Username:139********
Password:***********
139******** login successfully!

After successful login, the file transfer operation will be carried out, and the locally printed compressed package will be uploaded to Personal data Yes. Note that the uploaded file extension only supports the common compressed package format. The path of the compressed package here is   D:\Datasets\MNIST.zip.

# Execute. \ oss cp D:\Datasets\MNIST.zip oss://
PS D:\Download> .\oss cp D:\Datasets\MNIST.zip oss://

Start at 2021-10-22 06:21:48.2061924 +0000 UTC

[-------------------------------------] 100.00% 52.34MB/s 76.55MB/76.55MB 1.53s

Upload successfully, 76.55MB, n/a, D:\Datasets\MNIST.zip --> oss://MNIST.zip, cost [7020], status [200], request id [0000017CA6A9BE01901395D26CE9A228]

On the platform after successful upload Personal data The file can be viewed on the page.

Instance Download Personal Data

After an instance is created or started on the platform, the   JupyterLab   Enter the terminal or   SSH client Connect the terminal.

It is also used in the terminal   oss   Log in first. The login account is the account name and password of Hengyuan cloud platform, not an instance. The mobile phone number Chinese name is called. If it is the mobile phone number in Chinese mainland, it needs to add the country number with +.

# Execute oss login
~# oss login
Username:139********
Password:***********
139******** login successfully!

  After successful login, download the file and download the file just uploaded to  / hy-tmp   Medium (available for models with shared storage)  / hy-nas).

# Execute oss cp oss://MNIST.zip /hy-tmp
~# oss cp oss://MNIST.zip /hy-tmp
Start at 2021-10-22 06:37:17.227649376 +0000 UTC

[-------------------------------------] 100.00% 62.85MB/s 76.55MB/76.55MB 1.42s
Waiting to rename temporary file...

Download successfully, 76.55MB, n/a, oss://MNIST.zip --> /hy-tmp/MNIST.zip, cost [2422], status [206], request id [0000017CA6B7D6419012DB7767D11959]

After the download is successful, enter the download path and decompress the compressed package.

# Execute CD / HY TMP to enter this directory
~# cd /hy-tmp
# Execute unzip -q MNIST.zip to unzip the package
/hy-tmp# unzip -q MNIST.zip

Shut down after performing training and automatically uploading results

The data already exists in the instance, and the next step is the training stage. have access to   JupyterLab   or   VSCode   And other tools. Note that the data path referenced in the code needs to be changed to the data path within the instance. In the above example, the  / hy-tmp. Long training tasks need to pass   Tmux   Run in the background so that you can turn off your local computer without interrupting your training.

Pay as you go instances can upload results to personal data after training is completed through the following operations, and then shut down to terminate billing. This operation can save the use cost of pay as you go and avoid the damage caused by the occupation of the machine  / hy-tmp   Your data cannot be viewed and downloaded.

The operation of the sample script requires the following preconditions, and some operations are adjusted according to actual needs.

• The results of the training are  / hy-tmp/result   in
• In the instance, the platform account, mobile number and password need to be used in advance   oss login   Successful login

stay  / root   Create a directory using vim   upload.sh   script.

cd
vim upload.sh

The script is as follows, using   vim   Editor, entering    Enter the insert mode, paste down the entire script content, and then enter  preservation.   

#!/bin/bash
set -e

cd /hy-tmp
# Package name
file="result-$(date "+%Y%m%d-%H%M%S").zip"
# Make the result directory into a zip package
zip -q -r "${file}" result
# Upload to the backup folder in your personal data through oss
oss cp "${file}" oss://backup/
rm -f "${file}"

# Shutdown after successful transmission
shutdown

Add execution permission to the script.

chmod u+x upload.sh

This script implements the  / hy-tmp/result   Make it into a zip package, and then upload this file to your personal data. Execute the shutdown operation after successful execution.

The last call in the training task code.  / root/upload.sh   The script can be run. It is recommended to test it when it is used for the first time. If referenced in Python code:

import os
os.system('/root/upload.sh')

It can be used locally after the training execution is completed and shut down   oss   The command line tool downloads the packaged result file. When the instance machine is occupied and cannot be started, you can also create a new instance on other machines and start it through   oss   Download the data to the new instance. Specific about   oss   Relevant operation reference documents   oss command tool.

# Log in to the Hengyuan cloud account and use the account name and password of Hengyuan cloud. The account name is the mobile phone number
# If the mobile phone number is in Chinese mainland, it is necessary to add a zone code with +.
~# oss login
Username:139********
Password:***********
139******** login successfully!

# View files and folders, - s means that only file names are displayed
~# oss ls -s oss://
Folder list:
oss://
oss://backup/

Object list:
oss://backup/result-20211018-164323.zip

# Download the result-20211018-164323.zip file in personal data to the current directory
~# oss cp oss://backup/result-20211018-164323.zip .
Download successfully, 310B, n/a, oss://backup/result-20211018-164323.zip --> /root/result-20211018-164323.zip, cost [57], status [200], request id [0000017C92929EF49014BE16738685B7]