Use Docker and Golang to get started with WebAssembly quickly

This article will talk about how to quickly get started with WebAssembly using Docker and Golang. I will describe it from the browser scenario and the "general application" scenario respectively. If you still linger in front of the web assembly, maybe this article will help you all.

Write in front

From the point of view that browsers began to support Web WebAssembly functions in an experimental way in 2017, the wind of using non JavaScript to complete computing in browsers has been blowing for five years. However, it has been nearly three years that Wasm ecology has really developed.

The change of the general environment has provided a broader market for audio and video, cloud computing and the Internet of things in the industry ecology, as well as a higher pursuit of cost reduction and efficiency improvement. If the C position in Wasm ecology is Mozilla, after the Mozilla layoff incident last year, they quickly established the Rust foundation to ensure that the Rust development team can operate independently and stably, protect the sustainable development of Rust and surrounding projects, and provide soil for ecology, which can be said to be advantageous.

The time and place are favorable, only for people.

The economic environment at home and abroad has changed unprecedentedly. Without the temptation of a lot of external capital, we can feel the vigorous development of infrastructure technology in recent years. There are all kinds of excellent engineers who are gradually shifting their attention from "business" to "technology". At present, with the rapid development and promotion of its first-class citizen Rust, Wasm Kingdom has attracted the attention of many other language ecology and well-known commercial companies. As for when it will break out, I personally think it is only a matter of time.

However, it should be noted that * * no technology will be a silver bullet. The effect of twice the result with half the effort can be achieved only by putting the technology in the applicable scenario** So which scenarios are suitable for web assembly?

For convenience of writing, the web assembly will be referred to as Wasm for short.

Applicable scenarios & advantages

Let's take a look at the scenarios in which the industry has publicly indicated that it has been used in the past three years:

If we summarize the above scenarios, we can see that WebAssembly still has great advantages in browser side, cloud computing and embedded:

  • Expand existing business capabilities (cloud and browser) with a relatively low amount of code
  • Make full use of the computing power of the client to save the bandwidth of ECs and the cost of computing resources (browser side)
  • Take advantage of wasm's advantages in high-performance computing to solve the execution efficiency of complex computing (visualization and general computing scenarios)
  • Rapid reuse of other language technologies and rapid iteration of products (cloud, browser and embedded) with the idea of container
  • Use more popular, easy to develop and maintain, or fit your team's language for product iteration (embedded)
  • Encryption of front-end sensitive content (browser side)

Easy start: web assembly in browser

Step by step, let's start with the simplest scenario: browser.

Environmental preparation

If you don't want to mess with the local development environment of golang, we can use Docker to quickly create a running environment:

docker run --rm -it -v `pwd`/code:/app -p 8012:8012 golang:1.17.3-buster bash

Here, we map the local code directory to the / app directory in the container, and connect the local port and port 8012 in the container for subsequent use.

Then, initialize the item in the terminal console of the container after the command is executed:

cd /app
go mod init soulteary.com/wasm-demo/v2

Then, use your preferred method (in the container or in the local IDE) to create a golang program file, such as main.go:

package main

import "fmt"

func main() {
	fmt.Println("Everything will start here")
}

After completion, execute go run main.go in the container console. Not surprisingly, you will see the text output result of "everything will start here".

Because the scenario we want to demonstrate includes the front end, we also need to have a simple Web server. Continue to write a simple Web server using golang.

package main

import (
	"log"
	"net/http"
)

func main() {
	log.Fatal(http.ListenAndServe(":8012", http.FileServer(http.Dir("."))))
}

Save the above memory as server.go. When we execute it, it will take the local as the root directory of the server and provide Web services to visitors.

In this scenario, engineers generally have several problems:

  • How to get a Wasm program
  • How to run this program in the browser;
  • How to make JavaScript in the browser call the export function of WASM. (functions in Golang program)
  • And how to further optimize the performance of the whole program

In the process of "Show You The Code", we will answer the above questions in turn.

Create a WebAssembly program from Golang

There are two general ways to "turn" a Golang program into a WebAssembly:

  • Compile using the Golang native compiler.
  • Compile using the TinyGo compiler.

Golang's "native compiler scheme" is very applicable. It is suitable for B-end products that are developed at the initial stage of the project, or do not care about the size of the compiled product and the first distribution time of the program. If you are willing to invest time in product volume cutting, you can also get good results. The build command is generally similar to GOOS=js GOARCH=wasm go build -o YOUR_MODULE_NAME.wasm., the construction product needs to cooperate with Golang wasm_exec.js.

Compared with the former, TinyGo's compilation results are smaller and can be used in embedded scenarios (more than 60 kinds of MCU are officially supported), cloud computing scenarios supporting Wasi interface, and Web scenarios mentioned in this text section. After GZip compression, your program is not even as big as a picture. The build command is similar to the native TinyGo build -- no debug - o your_ MODULE_ Name.wasm - target Wasi. The difference is that in addition to supporting wasm as the construction result, it supports communication of general Wasi, which is convenient for you to reuse multi terminal functions. (this capability is similar to Docker in terms of distribution mode, and it is similar to the front-end and back-end code reuse I practiced in Taobao team when Node first appeared.)

Let's take the native mode as an example. Based on the contents in the "environment preparation" section, we can compile wasm with the following command:

GOOS=js GOARCH=wasm go build -o module.wasm main.go

Next, copy the "JS Bridge" provided by Golang to the project root directory.

cp "$(go env GOROOT)/misc/wasm/wasm_exec.js" .

Then, write a landing page so that it can load the JS Bridge above, automatically download our compiled wasm program, and automatically execute after the program download is completed:

<html>
<head>
  <meta charset="utf-8" />
  <title>Go wasm</title>
</head>
<body>
  <script src="wasm_exec.js"></script>

  <script>
    if (!WebAssembly.instantiateStreaming) {
      // polyfill
      WebAssembly.instantiateStreaming = async (resp, importObject) => {
        const source = await (await resp).arrayBuffer();
        return await WebAssembly.instantiate(source, importObject);
      };
    }

    const go = new Go();

    let mod, inst;

    WebAssembly.instantiateStreaming(fetch("module.wasm"), go.importObject).then(
      async (result) => {
        mod = result.module;
        inst = result.instance;

        await go.run(inst);
        inst = await WebAssembly.instantiate(mod, go.importObject); // reset instance
      }
    );

  </script>
</body>
</html>

When everything is ready, we execute go run server.go. When you visit localhost:8012 in the browser, you can see that the console outputs the long lost string above: "everything will start here".

Create API interfaces that can interact with JS

Let's take a basic MD5 calculation as an example to expand the story of this section: suppose we need to make JavaScript in the browser call the MD5 calculation function in Golang.

First adjust the "main.go" file in the "environment preparation" section to complete the basic calculation.

package main

import (
	"crypto/md5"
	"fmt"
)

func main() {
	fmt.Println("Everything will start here")
	fmt.Println(CalcMd5("The result you want to calculate"))
}

func CalcMd5(src string) string {
	return fmt.Sprintf("%x", md5.Sum([]byte(src)))
}

Run the program with go run main.go, and you can see the following results:

Everything will start here
849d1b972ec01975a9d1e16f804fec94

Then, adjust the syntax of the above program and declare the new function CalcMd5 as a Wasm export function accessible to JS.

package main

import (
	"crypto/md5"
	"fmt"
	"syscall/js"
)

func main() {
	fmt.Println("Everything will start here")

	wait := make(chan struct{}, 0)
	js.Global().Set("CalcMd5", js.FuncOf(CalcMd5))
	<-wait
}

func CalcMd5(this js.Value, p []js.Value) interface{} {
	ret := fmt.Sprintf("%x", md5.Sum([]byte(p[0].String())))
	return js.ValueOf(ret)
}

Execute GOOS=js GOARCH=wasm go build -o module.wasm main.go to compile and build the module. Then execute go run server.go again. In the console of the browser, we can call the function CalcMd5 just created in Golang to calculate MD5 through JS.

If your project does not use cgo, you can consider directly using TinyGo to replace the compiler, and the compiled default product will be reduced to a size that surprises you. ( Code example of TinyGo , the discussion on TinyGo is detailed below and will not be described again)

To use TinyGo, you need to adjust the previous JS Bridge to the version of TinyGo.

cp "$(tinygo env TINYGOROOT)/targets/wasm_exec.js" .

Continue to use tinygo build -- no debug - O module.tiny.wasm - target wasm main.go to build a small wasm program. After the program is built, we can see that the optimization result is very obvious. Even after further compression, the file size is only about 64kb:

du -hs *
148K	module.tiny.wasm
2.0M	module.wasm
// 64K	module.tiny.wasm.gz

Of course, the program created by using Go is not just creating an interface that allows JS to call, but also calling JS API in browser environment in Go, or directly manipulate browser BOM API in Go to change the rendering and behavior of pages in the entire browser.

TinyGo abnormal error report and repair

When using TinyGo version of the program in the browser console, there may be some abnormal errors, such as "syscall/js.finalizeRef not implemented". For the solution, refer to the solution in GitHub for wasm_exec.js file Make a patch.

Advanced operation: use web assembly as a container to build a general WASI program

In the last decade, many languages have put forward the grand goal of "write once, run anywhere", among which Node.js has practiced the idea of language isomorphism. However, in the container era, we find that heterogeneous technology stacks are also very popular, as long as your application interfaces can be standardized, the communication efficiency is high enough, and the loss cost in the calculation process is small.

We can regard the existing container technology as a high latitude lightweight application running environment with OS and program running dependencies, while WASI applications are "flexible containers" with finer granularity : it can be integrated and used by many languages in any environment, parsed and executed in a more low-level manner during execution, or used as a lightweight sandbox. In the past, we threw the program in the container and implemented "code as infrastructure" by displaying the declaration , the arrival of Wasm/ WASI will make our program have the capability of component containerization, especially the capability expansion for cross stack and heterogeneous scenarios.

Environmental preparation

In order to experience the "generality" of Wasm program, we will write a Wasm program and call it using three different running environments: browser, Node and Golang. In order to quickly develop and verify, I have prepared a simple container environment:

FROM golang:1.17.3-buster
# Pre preparation
RUN sed -i -e "s/deb\.debian\.org/mirrors\.tuna\.tsinghua\.edu\.cn/" /etc/apt/sources.list && \
    sed -i -e "s/security\.debian\.org/mirrors\.tuna\.tsinghua\.edu\.cn/" /etc/apt/sources.list && \
    apt-get update && apt-get install -y && \
    rm /bin/sh && ln -s /bin/bash /bin/sh && \
    echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
# Prepare Go environment
RUN go env -w GO111MODULE=on
RUN go env -w  GOPROXY=https://goproxy.cn,direct
RUN curl -L https://github.com/tinygo-org/tinygo/releases/download/v0.21.0/tinygo_0.21.0_amd64.deb -o tinygo_0.21.0_amd64.deb && \
    dpkg -i tinygo_0.21.0_amd64.deb
# Installing wasmer
RUN curl https://get.wasmer.io -sSfL | sh
# Prepare Node environment
RUN curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash
ENV NVM_DIR /root/.nvm
ENV NVM_NODEJS_ORG_MIRROR "https://npm.taobao.org/mirrors/node"
RUN . $NVM_DIR/nvm.sh && nvm install node

WORKDIR /app

Save the above contents as Dockerfile, and then use docker build - it wasm dev env. To perform the build. Considering the simplicity of use, we can also write a container orchestration configuration:

version: '3'

services:

  wasm-dev-env:
    image: wasm-dev-env
    volumes:
      - ./app:/app
    command: tail -f /etc/hosts
    ports: 
      - 8081:8081
      - 8082:8082
      - 8083:8083

Save the above content as docker-compose.yml, and then use docker-compose up -d to start the container. After the container is started, use docker composition exec wasm dev env bash to enter our development environment. Let's take a look at the versions of various components first.

# go version
go version go1.17.3 linux/amd64

# node --version
v17.1.0

# wasmer --version
wasmer 2.0.0

#tinygo version
tinygo version 0.21.0 linux/amd64 (using go version go1.17.3 and LLVM version 11.0.0)

Write general WASI program

In consideration of practicality and interest, I will compile an open source Go software into WASI program to call programs in other languages. The open source project I choose can convert ordinary text into ASCII ART https://github.com/common-nighthawk/go-figure

First, initialize the project directory.

mkdir /app/wasm
cd /app/wasm
go mod init soulteary.com/wasm-demo/v2

Next, create a program called fun.go:

package main

import (
	"github.com/common-nighthawk/go-figure"
)

func main() {}

//export HelloWorld
func HelloWorld() {
	myFigure := figure.NewFigure("Hello World", "", true)
	myFigure.Print()
}

Then use TinyGo build -- no debug - O module.wasm - wasm ABI = generic - target = Wasi fun.go to compile the program. However, due to the compatibility of TinyGo's current fs module, our compilation will fail:

# github.com/common-nighthawk/go-figure
../../go/pkg/mod/github.com/common-nighthawk/go-figure@v0.0.0-20210622060536-734e95fb86be/bindata.go:3606:11: MkdirAll not declared by package os
../../go/pkg/mod/github.com/common-nighthawk/go-figure@v0.0.0-20210622060536-734e95fb86be/bindata.go:3614:11: Chtimes not declared by package os

Considering that we do not need the ability to customize external resources in the original project, we can directly adjust the dependent files with errors and delete the API methods not supported in TinyGo. After the adjustment, compile again and you will see that we can get the WASI program we need soon.

du -hs *
4.0K	funny.go
4.0K	go.mod
4.0K	go.sum
704K	module.wasm

Run the WASI standard web assembly program in the Node

There are two schemes to run Wasm in Node.js. One is to directly run the traditional Wasm program by using the WebAssembly object in Node, and the other is to run the Wasm program by using the WASI interface.

Although the second scheme is still in the experimental state in the Node and needs to be enabled with parameters, it is the future standard after all. The second method is still recommended here.

const { readFileSync } = require('fs');
const { WASI } = require('wasi');
const { argv, env } = require('process');

(async function () {
    const wasi = new WASI({ args: argv, env });
    const importObject = { wasi_snapshot_preview1: wasi.wasiImport, /** or: wasi_unstable: wasi.wasiImport **/ };
    const wasm = await WebAssembly.compile(readFileSync("./module.wasm"));
    const instance = await WebAssembly.instantiate(wasm, importObject);
    wasi.start(instance);

    const { HelloWorld } = instance.exports;
    HelloWorld();
}());

Save the above content as index.js, then use node --experimental-wasi-unstable-preview1 index.js to execute the program. We can see that we successfully invoked Wasm program compiled by Go in Node.js and exported "artistic word".

(node:15307) ExperimentalWarning: WASI is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
  _   _          _   _            __        __                 _       _
 | | | |   ___  | | | |   ___     \ \      / /   ___    _ __  | |   __| |
 | |_| |  / _ \ | | | |  / _ \     \ \ /\ / /   / _ \  | '__| | |  / _` |
 |  _  | |  __/ | | | | | (_) |     \ V  V /   | (_) | | |    | | | (_| |
 |_| |_|  \___| |_| |_|  \___/       \_/\_/     \___/  |_|    |_|  \__,_|

Run the WASI standard web assembly program in the browser

One scheme has been mentioned earlier. Next, let's try the second operation scheme. First, prepare the project directory and initialize the project.

mkdir /app/js-app
cd /app/js-app
npm init-y
npm i parcel parcel-bundler @wasmer/wasi @wasmer/wasmfs @wasmer/wasm-transformer --registry=https://registry.npmmirror.com
mkdir dist
cp module.wasm dist/

After the above command is executed, we will add a field in package.json in the project directory to minimize unnecessary compatibility conversion (you can adjust it according to your actual situation):

  "browserslist": [
    "last 1 Chrome versions"
  ],

Create a landing page for display, index.html:

<html>
  <body>
    <script src="./index.js" type="module"></script>
  </body>
</html>

Then, create our core script, index.js:

import { WASI } from '@wasmer/wasi/lib'
import browserBindings from '@wasmer/wasi/lib/bindings/browser'
import { WasmFs } from '@wasmer/wasmfs'

const wasmFilePath = '/module.wasm'
const wasmFs = new WasmFs()

let wasi = new WASI({
  args: [wasmFilePath],
  env: {},
  bindings: {
    ...browserBindings,
    fs: wasmFs.fs
  }
})

const startWasiTask =
  async pathToWasmFile => {
    let response = await fetch(pathToWasmFile)
    let wasmBytes = new Uint8Array(await response.arrayBuffer())

    let wasmModule = await WebAssembly.compile(wasmBytes);
    let instance = await WebAssembly.instantiate(wasmModule, {
      ...wasi.getImports(wasmModule)
    });

    wasi.start(instance)
    instance.exports.HelloWorld()

    let stdout = await wasmFs.getStdOut()
    document.write(`<p>Standard Output:</p><pre>${stdout}</pre>`)
  }

startWasiTask(wasmFilePath)

After the files are ready to continue, use. / node_modules/.bin/parcel index.html --port=8081 start the service, visit localhost:8081 in the browser, and you will see the output of calling Wasm program:

Standard Output:

  _   _          _   _            __        __                 _       _
 | | | |   ___  | | | |   ___     \ \      / /   ___    _ __  | |   __| |
 | |_| |  / _ \ | | | |  / _ \     \ \ /\ / /   / _ \  | '__| | |  / _` |
 |  _  | |  __/ | | | | | (_) |     \ V  V /   | (_) | | |    | | | (_| |
 |_| |_|  \___| |_| |_|  \___/       \_/\_/     \___/  |_|    |_|  \__,_|

Run WASI standard web assembly in Go program

It's actually a little challenging to run Wasm with WASI standard interface written by Golang in Golang. In general, you may encounter the following problems:

  • First, you can't build a standard Wasm program that supports WASI standard through Golang's compiler for the time being.
  • Secondly, if you use the popular wasmer go or other runtime to try to execute standard WASI programs, you may encounter some errors caused by compatibility problems, such as ` ` Missing import: `wasi_snapshot_preview1`.`fd_write```.
  • Finally, if you are lucky enough to fix these problems, you will find that without WAT text format file, you need to manually complete the export function of Wasm program to use the program normally.

With regard to the above questions, In the answer of wasmer go maintainer As mentioned, the maintainers of wasmer go project have suggested more than once that we use TinyGo instead of the default Golang compiler for generating WASI standard programs. If you want to use wasmer go to do this, you will encounter Some problems At present, the maintainer does not consider improving in this direction and recommends us to use it https://github.com/go-wasm-adapter/go-wasm This project is to "run once" the JS Bridge code that plays a bridging role in the browser in the Go code and "level up" the running environment. Or consider using https://github.com/mattn/gowasmer For the project of TinyGo, the "leveling" operation is carried out for the products of TinyGo.

Recalling what was mentioned at the beginning of the article, various cloud service gateways continue to support WASM to expand capabilities, while traifik, which we are familiar with before, adopts a scheme similar to Nginx and uses another more cumbersome scheme provided by the official team Golang based SDK , and then use Agreed manner Dynamically load these homogeneous applications locally or remotely. Most of the reasons for adopting this technical route may be due to the above practical problems.

However, 2021 is coming to an end. Will this problem still be a problem?

In fact, as early as the middle of this year, wasmer go can run Wasm through WASI, but the official project lacks a usable example. After some attempts, I solved this problem. Let's play with me.

First create the project directory and perform some initialization operations:

mkdir /app/go-app
cd /app/go-app/
go mod init soulteary.com/go-app/v2
cp /app/wasm/module.wasm .

Next, install the latest version of wasmer go project Runtime:

go get github.com/mattn/gowasmer

Then, write a simple Golang program to load the Wasm program and execute it:

package main

import (
	"fmt"
	"io/ioutil"

	wasmer "github.com/wasmerio/wasmer-go/wasmer"
)

func main() {
	wasmBytes, _ := ioutil.ReadFile("module.wasm")

	store := wasmer.NewStore(wasmer.NewEngine())
	module, _ := wasmer.NewModule(store, wasmBytes)

	wasiEnv, _ := wasmer.NewWasiStateBuilder("wasi-program").
		// Choose according to your actual situation
		// Argument("--foo").
		// Environment("ABC", "DEF").
		// MapDirectory("./", ".").
		Finalize()
	importObject, err := wasiEnv.GenerateImportObject(store, module)
	check(err)

	instance, err := wasmer.NewInstance(module, importObject)
	check(err)

	start, err := instance.Exports.GetWasiStartFunction()
	check(err)
	start()

	HelloWorld, err := instance.Exports.GetFunction("HelloWorld")
	check(err)
	result, _ := HelloWorld()
	fmt.Println(result)
}

func check(e error) {
	if e != nil {
		panic(e)
	}
}

Save the above content as main.go, and then execute go run main.go. Not surprisingly, you will see results similar to the following:

  _   _          _   _            __        __                 _       _
 | | | |   ___  | | | |   ___     \ \      / /   ___    _ __  | |   __| |
 | |_| |  / _ \ | | | |  / _ \     \ \ /\ / /   / _ \  | '__| | |  / _` |
 |  _  | |  __/ | | | | | (_) |     \ V  V /   | (_) | | |    | | | (_| |
 |_| |_|  \___| |_| |_|  \___/       \_/\_/     \___/  |_|    |_|  \__,_|
<nil>

Running WASI standard programs in other languages

If you have a need to run WASI programs in other languages, you can pay attention to it https://github.com/wasmerio This project. Or refer to the practices of commercial companies or teams mentioned above to peel the required code from their open source projects.

Considering that specific scenario problems need specific analysis, I won't do it here. If necessary, I will write another article to talk about the integration and use of Wasm programs in other technology stacks and application ecology.

last

At present, the ecology of wasm field needs to be improved, like the front-end ecology ten years ago** Wasm, like the front-end technology, appears and develops not to replace anyone, but to make one more way to solve things and make the existing multi technology products more efficient** I always believe that there will be a group of people who will improve its ecology like the old front-end ten years ago.

I hope this article can help you who are unable to start before this technology, and open a new door for you.

A gentleman meets his friends with literature and complements benevolence with friends.

–EOF

We have a small tossing group, which gathers hundreds of little friends who like tossing.

Without advertising, we will talk about some problems in software and hardware, HomeLab and programming, and occasionally share some information about the technology salon in the group.

Friends who like to toss are welcome to scan the code to add friends. (if you add a friend, please note your real name and indicate the source and purpose, otherwise it will not pass the review)

About tossing into the group

If you think the content is practical, you are welcome to like it and share it with your friends. Thank you here.

This article uses the "signature 4.0 International (CC BY 4.0)" license agreement. You are welcome to reprint, modify or use it again, but you need to indicate the source. Signature 4.0 International (CC BY 4.0)

Author: Su Yang

Created on: November 21, 2021
Statistics: 9448 words
Reading time: 19 minutes
Link to this article: https://soulteary.com/2021/11/21/use-docker-and-golang-to-quickly-get-started-with-webassembly.html

Tags: Go Docker WebAssembly

Posted on Mon, 22 Nov 2021 01:22:34 -0500 by nickman013