Browser directly uploads files to Cloud Storage, bypassing the maximum 32M limit of App Engine Request

With the increase of traffic, GCP App Engine will automatically allocate more resources to the application, but the automatic allocation of resources is still constrained by some thresholds, one of which is that the request body sent to the application cannot be greater than 32M. For the demand of uploading large files, this restriction makes the applications that set the EndPoint of the file upload service on the App Engine unable to process the request normally.

Considering that App Engine does not allow applications to operate on local storage, and the files we upload will not be saved locally, but in Cloud Storage, File Store and other places.

For example, Cloud Storage provides Signed URLs Access to resources in Storage (without any constraints other than expiration time). In this way, the content type needs to be used as a part of the signature when signing. However, for a form of mutlipart/formdata type, the content tyep is in the form of mutlipart/formdata -- boundaryString, which is generated dynamically. This results in Unable to sign file upload request.

In this questions You can also see that someone has given a solution, which is to use the post-object#policydocument.

See the following example:

<form action="https://storage.googleapis.com/bucket-name/file-name" method="post" enctype="multipart/form-data">
    <input type="hidden" name="GoogleAccessId" value="1234567890123@developer.gserviceaccount.com">
    <input type="hidden" name="policy" value="eyJleHBpcmF0aW9uIjogIjIwMTAtMDYtMTZUMTE6MTE6MTFaIiwNCiAi">
    <input type="hidden" name="signature" value="BSAMPLEaASAMPLE6SAMPLE+SAMPPLEqSAMPLEPSAMPLE+SAMPLEgSAMPL">
    <input type="hidden" name="success_action_status" value="200">
    <input type="file" name="file">
    <input type="submit" value="Upload">
</form>

The policy field contains an upload rule indicating the user needs to follow, i.e. policy document. After base64 encoding, fill in the policy field.

// policy document
{
  "expiration": "2010-06-16T11:11:11Z",
  "conditions": [
    ["content-length-range", 0, 1073741824],
    ["eq", "$success_action_status", "200"]
  ]
}

The content length range above indicates that the user's upload file size is limited to 0-1gb. In addition, there are several other rules that can be added. Note that this method only supports one file upload at a time. The signature field of the form is the policy document after signature, which needs to be signed with service account key. The signature algorithm is SHA256withRSA. Here is the implementation of go by the way:

import (
	"crypto"
	"crypto/rand"
	"crypto/rsa"
	"crypto/sha256"
	"crypto/x509"
	"encoding/base64"
	"encoding/json"
	"encoding/pem"
)

func signPolicy(policy string) (string, error) {
	secretKey, err := loadSecretKey()
	if err != nil {
		return "", err
	}

	// sign base64ed policy document using RSA with SHA-256 using a secret key
	d := sha256SumMessage(base64.StdEncoding.EncodeToString([]byte(policy)))
	messageDigest, err := rsa.SignPKCS1v15(rand.Reader, secretKey, crypto.SHA256, d)
	if err != nil {
		return "", err
	}

	return base64.StdEncoding.EncodeToString(messageDigest), nil
}

func sha256SumMessage(msg string) []byte {
	h := sha256.New()
	h.Write([]byte(msg))
	d := h.Sum(nil)
	return d
}


func loadSecretKey() (priKey *rsa.PrivateKey, err error) {
	// RSA private key, extract from service account key json file, private_key filed
	blockPri, _ := pem.Decode([]byte(`-----BEGIN PRIVATE KEY-----
-----END PRIVATE KEY-----
`))

	// may returns a *rsa.PrivateKey, a *ecdsa.PrivateKey, or a ed25519.PrivateKey
	// see doc here: https://golang.org/src/crypto/x509/pkcs8.go
	prkI, err := x509.ParsePKCS8PrivateKey(blockPri.Bytes)
	if err != nil {
		return nil, err
	}

	return prkI.(*rsa.PrivateKey), err
}

In GCP IAM & admin#Service account You can manually create the RSA key for the App Engine default service account, and choose to download the private key to the local in the format of json file. The file format is as follows:

{
  "type": "service_account",
  "project_id": "***",
  "private_key_id": "***",
  "private_key": "-----BEGIN PRIVATE KEY-----***-----END PRIVATE KEY-----\n",
  "client_email": "***",
  "client_id": "***",
  "auth_uri": "***",
  "token_uri": "***",
  "auth_provider_x509_cert_url": "***",
  "client_x509_cert_url": "***"
}

The value of private key is RSA private key. It should be noted that it is not safe to put private keys directly in the code. The next thing we need to do is to find a safe place to save and replace these private keys regularly. KMS It's an option.

108 original articles published, 45 praised, 90000 visitors+
Private letter follow

Tags: encoding JSON

Posted on Thu, 16 Jan 2020 02:39:30 -0500 by fusioneko