tinygo-org
WASM program does not respond when repeatedly creating big data slices
I am new to Go and want to use wasm to calculate the md5 of a file, but I found that I can only calculate a 45Mb file twice, and then it will be stopped on this line. dst := make([]byte, args[0].Get("length").Int())I have to refresh the browser to make it work again. Is the memory usage high or what is the reason? I do not know. Is there a way to release it?
I compiled the same code with Go's built-in WASM and it worked well.
My tinygo version:
tinygo version 0.19.0 darwin/amd64 (using go version go1.16.5 and LLVM version 11.0.0)
Here is the my code:
main.go:
package main
import (
"crypto/md5"
"fmt"
"syscall/js"
)
func getMD5() js.Func {
return js.FuncOf(func(this js.Value, args []js.Value) interface{} {
uint8Array := js.Global().Get("Uint8Array")
uint8ClampedArray := js.Global().Get("Uint8ClampedArray")
if len(args) < 1 || !(args[0].InstanceOf(uint8Array) || args[0].InstanceOf(uint8ClampedArray)) {
return js.Undefined()
}
//stop on this line;
dst := make([]byte, args[0].Get("length").Int())
js.CopyBytesToGo(dst, args[0])
if dst == nil {
return js.Undefined()
}
return fmt.Sprintf("%x", md5.Sum(dst))
})
}
func main() {
js.Global().Set("getMd5", getMD5())
select {}
}
index.html:
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8" />
<title>Hello World - Go</title>
</head>
<body>
<input
type="file"
id="j-file"
>
<button id="j-cal">btn</button>
<script src="./wasm_exec.js"></script>
<script
type="module"
src="./index.js"
></script>
</body>
</html>
index.js
(async function () {
const go = new Go()
let app
if (WebAssembly.instantiateStreaming) {
app = await WebAssembly.instantiateStreaming(fetch("app.wasm"), go.importObject)
} else {
const res = await fetch("app.wasm");
app = await WebAssembly.instantiate(await res.arrayBuffer(), go.importObject)
}
const { instance } = app;
go.run(instance)
const $input = document.getElementById("j-file")
const $btn = document.getElementById("j-cal")
$btn.addEventListener("click", function () {
const file = $input.files[0] || new Blob([]);
const fileReader = new FileReader()
fileReader.onload = function () {
console.log((window.getMd5 || function () { })(new Uint8Array(this.result)))
}
fileReader.readAsArrayBuffer(file)
})
})()
build command:
tinygo build -o ./demo/app.wasm -target wasm .
What is the error that you get in the JavaScript console (developer tools)?
There is no error, but the process is getting blocked.
What is the error that you get in the JavaScript console (developer tools)?
Sorry to copy and paste the code with issues, I have corrected it. But when I calculate the MD5 of a file larger than 45MB for the second time without refreshing the page, it is still blocked.
@Mrooze-zeng have you found a solution? Currently experiencing same problem while repeatedly processing large files (~9MB)
I might suggest using a tinygo //export function to do the MD5 instead of re-buffering. Since the memory is shared, you can use an offset/len pair to update the hasher. I think you can find some WebAssembly hash functions that do similarly even if their source isn't TinyGo. Ex. https://github.com/Daninet/hash-wasm
