feat: add basic WASI `net.Listener` via `sock_accept`

[rvolosatovs]

trafficstars

Description

Add support for sock_accept recently added to WASI https://github.com/WebAssembly/WASI/pull/458

The implementation still lacks polling support, but that is something best left for another PR, since that would be a magnitudes bigger change.

Relevant documentation on flags can be found here https://github.com/WebAssembly/WASI/blob/main/phases/snapshot/docs.md#fdflags

I followed the upstream Go implementation as much as possible with minimal changes

Testing

Server

Save the following program to main.go in your working directory:

package main

import (
	"fmt"
	"io"
	"log"
	"net"
	"os"
	"strconv"
	"time"
)

func handle(conn net.Conn) error {
	defer conn.Close()

	var b [128]byte
	n, err := conn.Read(b[:])
	if err != nil {
		if err == io.EOF {
			return nil
		}
		return fmt.Errorf("failed to read from connection: %s", err)
	}
	log.Printf("Read '%s'\n", b[:n])

	res := fmt.Sprintf("Hello, %s", b[:n])
	n, err = conn.Write([]byte(res))
	if err != nil {
		return fmt.Errorf("failed to write to connection: %s", err)
	}
	log.Printf("Wrote '%s'\n", res[:n])
	return nil
}

func run() error {
	fds, err := strconv.Atoi(os.Getenv("FD_COUNT"))
	if err != nil {
		return fmt.Errorf("failed to parse FD_COUNT: %w", err)
	}
	if fds != 4 {
		return fmt.Errorf("FD_COUNT must be 4, got %d", fds)
	}

	ln, err := net.FileListener(os.NewFile(uintptr(3), "socket"))
	if err != nil {
		return fmt.Errorf("failed to listen on fd 3: %w", err)
	}

	for {
		log.Println("Waiting for connection...")
		conn, err := ln.Accept()
		if err != nil {
			if err, ok := err.(*os.SyscallError); ok && err.Timeout() {
				log.Println("Accept timed out, retrying in a second...")
				time.Sleep(time.Second)
				continue
			}
			return fmt.Errorf("failed to accept connection: %w", err)
		}
		log.Println("Accepted connection")

		if err := handle(conn); err != nil {
			return fmt.Errorf("failed to handle connection: %w", err)
		}
		log.Println("---")
	}
}

func init() {
	log.SetOutput(os.Stderr)
	log.SetFlags(0)
}

func main() {
	if err := run(); err != nil {
		log.Fatalf("Failed to run: %s", err)
	}
}

Compile with:

$ tinygo build -target wasi main.go

Wasmtime

Works on wasmtime 0.38.0:

$ wasmtime run --env FD_COUNT=4 --tcplisten 127.0.0.1:9000 main.wasm
Waiting for connection...
Accept timed out, retrying in a millisecond...
Waiting for connection...
Accept timed out, retrying in a millisecond...
Waiting for connection...
Accept timed out, retrying in a millisecond...
Waiting for connection...
Accepted connection
Read 'test
'
Wrote 'Hello, test
'
---

Enarx

Works on https://github.com/enarx/enarx with the following config:

[[files]]
kind = "stdin"

[[files]]
kind = "stdout"

[[files]]
kind = "stderr"

[[files]]
kind = "listen"
prot = "tcp"
port = 9000

For example, on Linux:

$ curl -sL -o enarx https://github.com/enarx/enarx/releases/download/v0.6.0/enarx-x86_64-unknown-linux-musl
$ echo 26823747c19c21fc7c5a7a6d44337d731735c780044b9da90365e63aae15f68d enarx | sha256sum -c
$ chmod +x enarx
$ ./enarx run --wasmcfgfile Enarx.toml main.wasm
Waiting for connection...
Accepted connection
Read 'test
'
Wrote 'Hello, test
'
---

Client

Start in a terminal window:

while :; do sleep 1 && echo test | nc 127.0.0.1 9000; done
Hello, test

[deadprogram]

Hello @rvolosatovs could you please rebase against the latest dev branch so this can be more easily reviewed?

cc/ @dkegel-fastly @dgryski @aykevl

[npmccallum]

@rvolosatovs Can you rebase this?

[rvolosatovs]

@deadprogram apologies for the delay, I was on vacation right after we spoke and then had a few very busy weeks. I've rebased, updated the implementation and added an example for testing the PR is ready for review, PTAL! cc @dkegel-fastly @dgryski @aykevl

There is a CI build failure on MacOS, but it does not seem to be caused by this PR - I may be wrong however

[npmccallum]

@deadprogram Are you able to review this?

[rvolosatovs]

I've extracted some of the required changes into smaller scoped PRs in hopes of a simpler review. @dkegel-fastly @dgryski @aykevl @deadprogram

This PR is now blocked by: https://github.com/tinygo-org/tinygo/pull/3051 https://github.com/tinygo-org/tinygo/pull/3050 https://github.com/tinygo-org/tinygo/pull/3052 https://github.com/tinygo-org/tinygo/pull/3053 https://github.com/tinygo-org/tinygo/pull/3054

[rvolosatovs]

@deadprogram would further splitting this PR into smaller scoped chunks assist review?

[rvolosatovs]

Filed #3061 , marking as draft again

[codefromthecrypt]

@rvolosatovs interesting (esp that the snapshot01 version was updated after the fact!).

I was curious the motivation of this? Is this a proof of concept or are folks making raw socket listeners? Reason I ask is often there are higher level libs that need some changes to support even if sock_accept works.

Finally, I noticed in the past due to lack of description in the spec sometimes you need to look at another impl to see how they handle things. Did you notice how this is implemented in rust or clang?? Ex the resulting wasm (can inspect backwards via wasm2wat) looks similar between here and rust or C of the same?

Cheers!

[rvolosatovs]

@rvolosatovs interesting (esp that the snapshot01 version was updated after the fact!).

I was curious the motivation of this? Is this a proof of concept or are folks making raw socket listeners? Reason I ask is often there are higher level libs that need some changes to support even if sock_accept works.

The motivation is to be able to accept a connection from within a WebAssembly module via WASI. In Wasmtime and Enarx this works by calling sock_accept on pre-opened file descriptors

Finally, I noticed in the past due to lack of description in the spec sometimes you need to look at another impl to see how they handle things. Did you notice how this is implemented in rust or clang?? Ex the resulting wasm (can inspect backwards via wasm2wat) looks similar between here and rust or C of the same?

Both Rust (https://github.com/rust-lang/rust/pull/93158) and C (https://github.com/WebAssembly/wasi-libc/pull/287) support was added by @haraldh, who we work together with at @profianinc. We're making sure to have all implementations consistent.

Note, that big difference of sock_accept from accept family, is that the peer address cannot be determined (at least for now) as part of the "accept".

[hlubek]

I played around with this PR and tried to use it to start an HTTP server with net/http over the FileListener. Basically my WASM/WASI code looks like this (following the example from this PR):

func run() error {
	ln, err := net.FileListener(os.NewFile(uintptr(3), "socket"))
	if err != nil {
		return fmt.Errorf("failed to listen on fd 3: %w", err)
	}

	mux := http.NewServeMux()
	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintf(w, "Hello, %s", r.URL.Path[1:])
	})

	srv := &http.Server{
		Handler: mux,
	}

	err = srv.Serve(ln)
	if err != nil {
		return fmt.Errorf("failed to serve: %w", err)
	}

	return nil
}

It seems to work after I did the following changes:

• Added a dummy LocalAddr and RemoteAddr, otherwise the Go HTTP implementation had a nil dereference issue
• Implement a very naive polling via checking for syscall.EAGAIN and sleeping for a few milliseconds (in Read, Write and Accept):

  func (c *netFD) Read(b []byte) (int, error) {
    var (
    	n   int
    	err error
    )
    for {
    	n, err = c.File.Read(b)
    	if errors.Is(err, syscall.EAGAIN) {
    		time.Sleep(30 * time.Millisecond)
    		continue
    	}
    	break
    }
    return n, err
  }
  

Doing this I could run the server via wasmtime run --tcplisten 127.0.0.1:9000 server.wasm and a Curl succeeded. 😁

What would be a better way to perform the polling? Is poll_oneoff supposed to work for sock_accept and reading / writing from the file descriptor?

I could imagine that the Listener returned by FileListener should not poll by default but just return EAGAIN / a syscall error indicating a timeout and the user code could wrap the listener to control the polling behaviour.

It would be really great to have some kind of support of this in TinyGo, so web services with Go could be run in WASM/WASI.

[codefromthecrypt]

@hlubek if were me I would probably scour GitHub for anyone using wasi-libc in C and see how they are doing it. https://github.com/WebAssembly/wasi-libc/blob/63e4489d01ad0262d995c6d9a5f1a1bab719c917/libc-bottom-half/sources/accept.c then compare polling approaches. I think these network functions are more used in rust for various docker demos.

[chriso]

We're proposing this be added to Go (with GOOS=wasip1). See https://go-review.googlesource.com/c/go/+/493358. It'll use the native netpoll facility so you won't have to manually handle EAGAIN.

[chriso]

This landed in Go via https://github.com/golang/go/commit/a17de43e (and prerequisites https://github.com/golang/go/commit/41893389 and https://github.com/golang/go/commit/c5c21845)

[rvolosatovs]

This landed in Go via https://github.com/golang/go/commit/a17de43e (and prerequisites https://github.com/golang/go/commit/41893389 and https://github.com/golang/go/commit/c5c21845)

Yes, it finally happened! Closing

[codefromthecrypt]

@deadprogram I think this issue is not closed in TinyGo, even if it is solved for Go. There will be users who can't use Go, yet due to lack of exports. Should this be re-opened as an issue until closed? or this codebase repurposed to complete regardless of what Go does?