Full Go "net" package port, WIP

[scottfeldman]

trafficstars

[This is a resurrection of #4187, which I accidentally closed by deleting the fork it was based on.]

This PR is WIP to port the full Go "net" package to TinyGo.

With this PR, I can compile and link a simple example:

package main

import (
        "fmt"
        "net"
)

func main() {
        conn, err := net.Dial("tcp", "localhost")
        if err != nil {
                fmt.Println(err)
        }
        conn.Close()
}

tinygo build -target nano-rp2040 -tags netgo main.go

Notes:

• This PR uses Go's full "net", "net/http", and "crypto/tls" packages, unmodified.
• Need to compile with -tags netgo to pick Go's DNS resolver.
• We no longer need the repo/submodule tinygo-org/net.
• None of the network drivers are hooked up; we'll need to redefine the network driver interfaces at the syscall level.
• I've copied in the stock Go src/syscall/ files for linux,arm. This defines all of the syscall types and data structures for linux,arm. We can copy in other arch files as needed. I'm only testing with nano-rp2040.
• There are new files in sys/syscall:
  • src/syscall/env_tinygo.go
  • src/syscall/exec_tinygo.go // can go away if issue # 1 is addressed
  • src/syscall/netlink_tinygo.go // can go away if issue # 1 is addressed
  • src/syscall/syscall_tinygo.go
• syscall_tinygo.go has the stubbed out syscalls. These all println("not implemented") and return EOPNOTSUPP.

The idea is this: with the netdev work in the last release, we truncated the "net", "net/http", and "crypto/tls" packages and inserted our own stubs to call into the network wifi drivers. With this PR, we use the full Go packages, but this time the insertion point is at the syscall level. Syscall is now where we define the interfaces to the network stack, and the network stack calls into the network drivers. It makes sense...if we were a full OS, syscall is where we'd have our OS-specific code.

Issues:

Issue # 1

Import cycle with "sync" package. Unfortunately, syscall package imports "sync" in some places, which causes an import cycle:

package tinygo.org/x/drivers/examples/net/http-get
       imports bytes
       imports io
       imports sync
       imports internal/task
       imports runtime/interrupt
       imports device/arm
       imports syscall
       imports sync: import cycle not allowed
(See src/syscall/netlink_linux.go for an example).

I've worked around this issue by stubbing out any imports of "sync", but that's not a workable solution in the long term. We'll need "sync" in the implementation of the stubbed out syscalls for the network stack and drivers. So we'll need to revisit "sync" so as to not import "syscall". Is it possible?

[leongross]

@scottfeldman could you maybe elaborate on the process of how the board-specific drivers are abstracted here? As my understanding of tinygos network driver model goes, drivers are represented as the a combination of Netdever and Netlinker interfaces. But I cannot find any reference to that in the inserted syscall code, so how are the bare metal board specifics picked up here?

And to enable network support for linux platforms, couldn't the functions in syscall_tinygo be linked against the according and already existing interfaces provided bey the unix package?

[scottfeldman]

@scottfeldman could you maybe elaborate on the process of how the board-specific drivers are abstracted here? As my understanding of tinygos network driver model goes, drivers are represented as the a combination of Netdever and Netlinker interfaces. But I cannot find any reference to that in the inserted syscall code, so how are the bare metal board specifics picked up here?

@leongross your're right, the netdev/netlink interfaces are the current TinyGo driver model for embedded net devices. In this PR, these interfaces will be replaced with a TBD interface at the syscall level, since all the "net" package calls ultimately resolve into syscalls. The next step in this PR is to discover which syscalls are needed by "net" (and crypto/tls), and let those define the interface to the device drivers. The goal is to support both raw-MAC devices (i.e. Pico-W) as well as devices with an embedded stack (i.e. wifinina, rtl8720n). Some ASCII pics:

raw-MAC stack:

your app
"net" package
-------------           <-- syscall interface TBD
network stack
device driver
-------------           <-- hw interface
device

embedded stack:

your app
"net" package
-------------           <-- syscall interface TBD
device driver
-------------           <-- embedded fw interface
embedded network stack
-------------           <-- hw interface
device

And to enable network support for linux platforms, couldn't the functions in syscall_tinygo be linked against the according and already existing interfaces provided bey the unix package?

Enabling full OS support for "net" wasn't the goal of this PR, but it seems we could make it work by linking in the OS syscalls when compiling against a full OS, bypassing the driver interface mentioned above. I suspect the work to do this is around loader/goroot.go and some built tag magic.

[ydnar]

Import cycle with "sync" package. Unfortunately, syscall package imports "sync" in some places, which causes an import cycle:

Can you work around this with //go:linkname?

[scottfeldman]

Import cycle with "sync" package. Unfortunately, syscall package imports "sync" in some places, which causes an import cycle:

Can you work around this with //go:linkname?

Yes!

I say that with excitement as I discovered that work-around last week and it's working great. So Issue #1 is not an issue.

[scottfeldman]

Update: I'm not ready to post commits, but I do have the full "net" pkg calling into wifinina driver via a custom TinyGo syscall interface. syscalls in the interface so far:

src/syscall/system.go

//go:build tinygo

package syscall

type systemer interface {
        Socket(domain, typ, proto int) (fd int, err error)
        CloseOnExec(fd int)
        SetNonblock(fd int, nonblocking bool) (err error)
        SetsockoptInt(fd, level, opt int, value int) (err error)
        Connect(fd int, ip []byte, port uint16) (err error)
        Write(fd int, buf []byte) (n int, err error)
        Read(fd int, buf []byte) (n int, err error)
}

Wifinina implements this interface. So far I have net.Dial("tcp", "foobar.com") attempting to connect. Since I'm compiling with -tags netgo, the Go DNS client will attempt to resolve "foobar.com" by opening a UDP socket on 127.0.0.1:53. So the first socket to open is the UDP socket. I'm working thru intercepting the Reads and Writes to fake a DNS server response to resolve "foobar.com". I'll have more details on this DNS business when I commit, but that's where I'm at right now.

The net.Dial() test app needs greater than -stack-size=16KB and less than -stack-size=32KB. I haven't figured out the minimum, but 32KB is good so far. The test image is ~350K flash, 8k ram.

[scottfeldman]

Update: I am making some commits to capture where I'm at so far. Not done, but I now have the first test (examples/net/tcpclient) working with wifinina using the full "net" pkg.

$ tinygo flash -monitor -tags netgo -target nano-rp2040 -size short -stack-size 32KB -ldflags="-X 'main.ssid=test' -X 'main.pass=testtest'" ./examples/net/tcpclient/

   code    data     bss |   flash     ram
 297888    5280    4232 |  303168    9512
Connected to /dev/ttyACM0. Press Ctrl-C to exit.

Tinygo ESP32 Wifi network device driver (WiFiNINA)

Driver version           : 0.27.0
ESP32 firmware version   : 1.4.8
MAC address              : 34:94:54:26:a7:cc

Connecting to Wifi SSID 'test'...CONNECTED

DHCP-assigned IP         : 10.0.0.113
DHCP-assigned subnet     : 255.255.255.0
DHCP-assigned gateway    : 10.0.0.1

---------------
Dialing TCP connection 
Sending data 

Wrote 133780 bytes in 3548 ms
 
Disconnecting TCP...
---------------
Dialing TCP connection 
Sending data 

[deadprogram]

This is extremely exciting @scottfeldman please let us know how we can help out!

[scottfeldman]

This is extremely exciting @scottfeldman please let us know how we can help out!

Thank you. I'm not sure how to break this up and share, but here's my short list of what still needs to be done:

0. need to create another PR in drivers to capture the wifinina changes so far...
1. get tcp/udp examples/net tests working with wifinina (nano-rp2040)
2. get "crypto/tls" examples working with wifinina
3. get all examples/net tests working with rtl8720n (wioterminal)
4. don't forget about espat
5. replace netdev/netlink interfaces with something cleaner, that also works with...
6. get seqs/cyw43 working under Systemer and pass all examples/net tests

I would really like help with 5 and 6. Work on those should probably wait until we get thru 1 and 2, just to prove the new full "net" pkg solution is going to work, especially the "crypto/tls" part.

[scottfeldman]

One more comment: I noticed the "net" pkg trying to open system files like /etc/resolve.conf and /etc/hosts for DNS resolution. Opening those files fail, of course, but I do see those calls working their way down to the syscall level so I had a thought: could we put a tinyfs behind these file i/o syscalls?

[scottfeldman]

@deadprogram I need some help with getting "crypto/tls" working...who's my contact for "crypto/tls" for big Go? I'm trying to get examples/net/tlsclient working, and it's doing the full TLS handshake with the server and then failing trying to verify the server certificate:

Connection failed: tls: failed to verify certificate: x509: certificate signed by unknown authority

What is working since last update is DNS resolution and UDP connections. For the TLS connection, I had to first use UDP to get NTP time and then call runtime.AdjustTimeOffset() to set system time, otherwise the server certificate fails due to being out-of-date wrt system time.

Oh, also, since I'm using nano-rp2040, I had to hack "crypto/rand" with this code to provide a custom rand.Reader:

func init() {
        rand.Reader = &reader{}
}

type reader struct{}

func (r *reader) Read(b []byte) (n int, err error) {
        if len(b) == 0 {
                return
        }
        var randomByte uint32
        for i := range b {
                if i%4 == 0 {
                        randomByte, err = machine.GetRNG()
                        if err != nil {
                                return n, err
                        }
                } else {
                        randomByte >>= 8
                }
                b[i] = byte(randomByte)
        }
        return len(b), nil
}

I think I got that code snippet from @deadprogram a while back, and have just been carrying it around with my projects. But perhaps this should get moved into TinyGo somehow? Anyway, without overriding rand.Reader, the program gets a nil-pointer dereference panic. "crypto/tls" uses rand to generate the TLS Client msg for the handshake.

Ok, that's it for now. If someone can help me get passed the server certificate verification, I think full "crypto/tls" is just going to work. From wifinina's perspective, it's just a simple TCP connection, so a lot of the code we had in there to program mbedTLS is no longer needed. Yay, bunch of code deletions!

[scottfeldman]

Ok, I'm passed the server cert validation issue I was having. I created a root CA cert to pass into tls.Dial() by go:embed'ing a PEM file containing the CA certs. Now the validator is happy.

Next problem is OOM:

panic: runtime error at 0x100407ef: out of memory
[tinygo: panic at /usr/local/go/src/crypto/internal/bigmod/nat.go:71:15]

The -print-allocs=. output is overwhelming. Not sure where to start there.

I sprinkled some runtime.GC() calls in the Connect() and Read() driver paths, but still hitting OOM:

panic: runtime error at 0x1003281b: out of memory
[tinygo: panic at /usr/local/go/src/crypto/internal/nistec/p256.go:235:11]

Maybe it's not possible for "crypto/tls" to fit? That would be a bummer.

I don't suppose there is a way to dump annotated memory to see what's in the heap at OOM?

[leongross]

I did some work on enabling the unix.Syscall wrapper for tinygo 1. When the syscal interface for the network stack is done I think we have a good chance to make networking finally work on linux systems.

[deadprogram]

I don't suppose there is a way to dump annotated memory to see what's in the heap at OOM?

I think something like this is possible using gdb?

[leongross]

Ok, I'm passed the server cert validation issue I was having. I created a root CA cert to pass into tls.Dial() by go:embed'ing a PEM file containing the CA certs. Now the validator is happy.

Next problem is OOM:

panic: runtime error at 0x100407ef: out of memory
[tinygo: panic at /usr/local/go/src/crypto/internal/bigmod/nat.go:71:15]

The -print-allocs=. output is overwhelming. Not sure where to start there.

I sprinkled some runtime.GC() calls in the Connect() and Read() driver paths, but still hitting OOM:

panic: runtime error at 0x1003281b: out of memory
[tinygo: panic at /usr/local/go/src/crypto/internal/nistec/p256.go:235:11]

Maybe it's not possible for "crypto/tls" to fit? That would be a bummer.

I don't suppose there is a way to dump annotated memory to see what's in the heap at OOM?

With the new go1.23 release (updated crypto libraries) and the according tinygo release it might be worth the effort to have another look at this.

[leongross]

This PR has been quite inactive in recent history. We at u-root are quite interested in getting this working, so we can make the remaining commands build for the initramfs busybox. If there is anything that you can outsource to me @scottfeldman I would be more than happy to help you with that.

[deadprogram]

@scottfeldman same goes for me, I am willing to help! :smile_cat:

[scottfeldman]

Hi Everyone, sorry for the silence/absence on this issue. I've been away.

My personal motivation for this work was to get pico-w running with the full net and crypto/tls packages and Patricios' cwy43 + seqs TCP/IP stack. I was sucessful in getting TCP/UDP traffic working with the full net package on Arduino rp2040. When I tried TLS traffic, I ran into the OOM issues.

I suspect TLS just by itself is going to take more RAM than we have available on these little embedded processors with ~256KB for both stack and heap. And then we have to add the net package and the seqs package which will need socket buffer space for each socket connection. So adding this all up, it's not going to fit. I wanted to investigate adding SPI PSRAM chip to pico-w to see if we could get 8MB of RAM. But I don't know the upper-limit for TLS (is 8MB enough?), and I don't know what's required from the compiler to map stack/heap to PSRAM. Probably a lot, I suspect.

So that's where I got stuck and gave up.

In any case, @leongross if this work can be used for Linux/u-boot, awesome, please take it over. I was surprised how little actually needs to be implemented at the syscall level to support the full net package. Most of the syscall .go files copied over as-is. It really boils down to a hand full of functions defined in the Systemer interface.

[starquake]

I suspect TLS just by itself is going to take more RAM than we have available

This is way above what I understand so I'm sorry if I say something untrue, but have you tried ECDSA instead of RSA?

[prskr]

Is someone still actively working on this? I fail to put in words how fantastic it would be to have 'full' net and - at least for me - even more fantastic full net/http support in TinyGo because this is something were I constantly hit the wall when I'm exploring WASM based tools like WasmCloud.

As an example: when writing functions, it would be great to be able to use oapi-codegen to have a spec-first approach for the function (making it a lot easier to replace it later, say in another tech stack, because it's WASM!). I tried all(?) - not 100% sure anymore - server types oapi-codegen supports but no matter which one I chose, there was always something missing from net/http and I could not compile it or it would not work at runtime.

Yesterday, I was playing around with a very simple function that should return a random XKCD comic link. I naively tried to use an endpoint that returns an HTTP 302 and forwards to a random comic page. Unfortunately, there seems to be a bug in the WasmCloud host when it comes to HTTP 302. Normally, I'd hack around that, but that was also not possible because the CheckRedirect field of the net/http.Client is missing in TinyGo 😅

Don't get me wrong! I very much appreciate what you all are doing here, I'm just trying to give some examples where it would be awesome to have full net/net/http support for WASM payloads as TinyGo is currently the go-to (no pun intended) solution for WASM with Go.

[tlaurion]

https://github.com/u-root/gobusybox/pull/121 was merged, which depended on this PR.

Really interested in u-root built with tinygo.

[deadprogram]

I am actively looking at this again as well... more soon.