go icon indicating copy to clipboard operation
go copied to clipboard

all: stop using direct syscalls on OpenBSD

Open 4a6f656c opened this issue 4 years ago • 92 comments

Upcoming changes to the OpenBSD kernel will prevent system calls from being made unless they are coming from libc.so (with some exceptions, for example, a static binary). There are also likely to be changes to the APIs used for system calls. As such, the Go runtime (and other packages) need to stop using direct syscalls, rather calling into libc functions instead (as has always been done for Solaris and now also for macOS). This will avoid these issues and allow Go to continue to function correctly on OpenBSD.

A version of Go with openbsd/amd64 being modified to perform all calls via libc is available at:

https://github.com/4a6f656c/go/tree/openbsd-syscall

Further work is still required to convert openbsd/386, openbsd/arm and openbsd/arm64.

4a6f656c avatar Jan 07 '20 17:01 4a6f656c

with some exceptions, for example, a static binary

But Go fits squarely within that exception. Or are you worried about nonstandard build modes?

randall77 avatar Jan 07 '20 17:01 randall77

That exception exists specifically for Go. https://lwn.net/Articles/806863/

FiloSottile avatar Jan 07 '20 19:01 FiloSottile

I think this move to make kernels callable only though a blessed C library is a bad idea, in particular for all users of programming languages that are not C-like. It introduces a performance overhead for every kernel call that could be avoided by direct kernel calls. The security benefits are debatable, fix the kernel, not hide the flaws and incompatibilities behind a C library.

Kernels should do like Linux does, and provide a safe, stable, kernel API, and not try to shift the interface to C libraries. And even then, graphic on Linux is a disaster because they exactly did that, provide most functionality in a C library, not in the kernel itself. We should protest at least once to mr. De Raadt. against this debatable decision.

beoran avatar Jan 08 '20 08:01 beoran

@randall77 - in many cases Go on OpenBSD will generate a dynamically linked executable, unless cgo is disabled:

$ cat n.go
package main

import (
        "fmt"
        "net"
)

func main() {
        fmt.Println(net.IPv4len)
}
$ go build -o n n.go 
$ ldd n              
n:
        Start            End              Type  Open Ref GrpRef Name
        0000000000010000 00000000001d3000 exe   2    0   0      n
        0000000453cfd000 0000000453d3e000 rlib  0    1   0      /usr/lib/libpthread.so.26.1
        000000048614c000 000000048624c000 rlib  0    1   0      /usr/lib/libc.so.95.1
        0000000497c90000 0000000497c90000 ld.so 0    1   0      /usr/libexec/ld.so
$ CGO_ENABLED=0 go build -o n n.go
ldd n
$ ldd n
n:
not a dynamic executable

This would mean that we'd have to drop cgo support and only support statically compiled Go executables.

@FiloSottile you're correct in that there is currently an exception for dynamically linked executables to allow syscalls from both libc.so and the main text segment, so that Go continues to work. However, in addition to this there will always likely be an exception for static binaries, since in that case there is no libc.so and parts of libc.a have likely been linked into the main text segment (otherwise a static binary could not make syscalls).

4a6f656c avatar Jan 08 '20 16:01 4a6f656c

@4a6f656c why does it generate dynamically linked executable in the first place?

ghost avatar Jan 12 '20 01:01 ghost

Change https://golang.org/cl/234381 mentions this issue: runtime, syscall: correct openbsd/arm and openbsd/arm64 syscalls for OpenBSD 6.7

gopherbot avatar May 20 '20 17:05 gopherbot

Moving to backlog milestone.

ianlancetaylor avatar Jun 16 '20 02:06 ianlancetaylor

I would like to work on this. After a brief look, Solaris seems to be a good role model for OpenBSD. What would be the best strategy here? OpenBSD seems to share definitions with the other BSD platforms, which result in conflicting definitions when I try to hook it into mksyscall_libc.pl. Or should I try to do things more like MacOS? Is that an easier way? How should I proceed?

bw-86 avatar Jul 17 '20 13:07 bw-86

I don't think there is any fundamental difference between the Solaris approach and the macOS approach. Following the Solaris approach seems fine if that is easier.

Yes, you will undoubtedly have to shuffle the build tags to separate openbsd from the other BSD systems in some cases.

ianlancetaylor avatar Jul 17 '20 15:07 ianlancetaylor

Well, I started. My WIP commits will be pushed here, if anyone is interested in them: https://github.com/bw-86/go

Currently, nothing is building, of course. I am chasing from one linking error to the next one while imitating what Solaris does. I will notify you when I'm getting somewhere (or when I have questions I cannot answer myself).

EDIT: I changed the default branch of my fork to 4a6f656c's work. My branch is openbsd-syscalls (plural), if anyone wants to see what I tried myself.

bw-86 avatar Jul 18 '20 11:07 bw-86

Does that mean Go will always use dynamic linking or raw syscalls will still be used when linking statically?

I agree with @beoran and iirc OpenBSD will not enforce libc.so stubs if you won't play their security theater, so you don't have to sacrifice both efficiency and actual security.

ghost avatar Jul 18 '20 12:07 ghost

I would like to work on this. After a brief look, Solaris seems to be a good role model for OpenBSD. What would be the best strategy here? OpenBSD seems to share definitions with the other BSD platforms, which result in conflicting definitions when I try to hook it into mksyscall_libc.pl. Or should I try to do things more like MacOS? Is that an easier way? How should I proceed?

@bw-86, the git repo referenced in the first comment contains a full implementation for openbsd/amd64:

https://github.com/4a6f656c/go/tree/openbsd-syscall

I've since rebased and cleaned some aspects of this up, but need to push a new branch.

As noted, there is still work required to complete the migration for openbsd/386, openbsd/arm and openbsd/arm64. There are also some issues relating to the linker that impact this work (see https://github.com/golang/go/issues/39257 and https://github.com/golang/go/issues/39256). Aside from the remaining architectures, most of the remaining work is around being able to split this up to merge upstream and I have some concerns about cross-compiling and bootstrapping that I've not yet had time to investigate.

4a6f656c avatar Jul 18 '20 15:07 4a6f656c

Does that mean Go will always use dynamic linking or raw syscalls will still be used when linking statically?

Go will always use dynamic linking (and possibly always external linking) - it would be a significant amount of effort to maintain both libc based syscalls and raw syscalls and switch between to two. Any Go binary on OpenBSD that pulls in the net package is already going to result in a dynamic cgo-based binary (unless cgo is explicitly disabled - see https://github.com/golang/go/issues/36435#issuecomment-572138195).

I agree with @beoran and iirc OpenBSD will not enforce libc.so stubs if you won't play their security theater, so you don't have to sacrifice both efficiency and actual security.

If you do not want OpenBSD's "security theater" then I would suggest not using OpenBSD. Additionally, have you actually measured the performance difference? Both Solaris and macOS already both use libc-style syscalls and IIRC the switch on macOS resulted in a minimal performance impact.

4a6f656c avatar Jul 18 '20 15:07 4a6f656c

Heh, how could I miss that? Well, at least I learned something by trying to do it myself.

@4a6f656c: Can I do something to accelerate this? Should I try to adapt your work to i386? I have a system here were I could develop and test it.

bw-86 avatar Jul 18 '20 15:07 bw-86

@4a6f656c it's not only about performance, it's also about usability and general efficiency which includes security, but what you overengineers do is simply making stuff overcomplicated and that's really bad for security you care about so much. Go is not a C-like language, so please don't mix it with C when possible.

I don't remember the article so I can't find it, but it states that on Solaris just because of libc stubs Go can't manage the stack sanely, it even has to allocate much larger stack than it would without libc stubs just to avoid crashes because it's become hard to scale, so security has actually lowered.

Also dynamic linking itself is a great mistake, it's bad for both security, performance and stability, again it's an overengineered virus.

ghost avatar Jul 18 '20 16:07 ghost

Heh, how could I miss that? Well, at least I learned something by trying to do it myself.

That's always a good thing :)

@4a6f656c: Can I do something to accelerate this? Should I try to adapt your work to i386? I have a system here were I could develop and test it.

Getting it working on openbsd/386 would be great!

I've just pushed an update to that repo that is based on Go -tip.

4a6f656c avatar Jul 18 '20 16:07 4a6f656c

Does that mean Go will always use dynamic linking or raw syscalls will still be used when linking statically?

Go will always use dynamic linking (and possibly always external linking) - it would be a significant amount of effort to maintain both libc based syscalls and raw syscalls and switch between to two. Any Go binary on OpenBSD that pulls in the net package is already going to result in a dynamic cgo-based binary (unless cgo is explicitly disabled - see #36435 (comment)).

Actually, there is another thing that I need to investigate - in many cases we should be able to statically link against libc.a, which would actually allow us to produce static Go binaries in more cases then we currently do on OpenBSD.

4a6f656c avatar Jul 18 '20 16:07 4a6f656c

I can chime in as both an OpenBSD user and someone who works on a project that builds releases for OpenBSD. I am happy with this change but hope that cross compiling to OpenBSD and reproducible builds remain possible. This probably negates requiring the external linker and having libc.a available.

The current behavior for macOS is ideal in this case. When cross compiling, it is still able to create a dynamically linked executable, and does so reproducibly, without depending on the external macOS linker.

$ uname
OpenBSD
$ cat importnet.go                                                             
package main

import _ "net"

func main() {}
$ CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build importnet.go 
$ echo put importnet | sftp machost
Connected to machost.
sftp> put importnet
Uploading importnet to /Users/jrick/importnet
importnet                                     100% 2047KB   2.6MB/s   00:00    
$ ssh machost './importnet && otool -L importnet'
importnet:
        /usr/lib/libSystem.B.dylib (compatibility version 0.0.0, current version 0.0.0)
        /System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 0.0.0, current version 0.0.0)
        /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 0.0.0, current version 0.0.0)

jrick avatar Jul 18 '20 18:07 jrick

I have to admit defeat for the moment. I pushed some minor work to my repository; feel free to take it. I will have to learn some more assembly and more about the calling conventions of Go and OpenBSD before I can tackle this. To be precise, I don't know how to write the syscall functions (syscall6, syscall9, ...).

So, can someone help me with this one? If I had just one of them, I might adapt it for the other use cases.

bw-86 avatar Jul 19 '20 15:07 bw-86

Change https://golang.org/cl/250180 mentions this issue: cmd/dist,cmd/link,runtime: switch openbsd/amd64 to pthreads

gopherbot avatar Aug 24 '20 15:08 gopherbot

Change https://golang.org/cl/250181 mentions this issue: cmd/link/internal/arm64: handle calls to SDYNIMPORT with internal linking

gopherbot avatar Aug 24 '20 15:08 gopherbot

Change https://golang.org/cl/250182 mentions this issue: runtime: switch openbsd/arm64 to pthreads

gopherbot avatar Aug 24 '20 15:08 gopherbot

Change https://golang.org/cl/250577 mentions this issue: runtime: switch openbsd/386 to pthreads

gopherbot avatar Aug 25 '20 17:08 gopherbot

@4a6f656c please read this: https://utcc.utoronto.ca/~cks/space/blog/programming/GoCLibraryAPIIssues and this: https://utcc.utoronto.ca/~cks/space/blog/unix/CLibraryAPIRequiresC

ghost avatar Nov 11 '20 16:11 ghost

@randall77 @ianlancetaylor - I've been informed that OpenBSD will be removing the direct syscall exception for Go binaries in March 2021, prior to the OpenBSD 6.9 release in May 2021. Unfortunately, this means that as things currently stand, Go 1.16 (released February 2021) will not work on OpenBSD 6.9 (released May 2021). Presuming patches land, Go 1.17 would resume working on OpenBSD 6.9, but this would not be released until August 2021. I'm not sure if a freeze exception would be considered to avoid this situation.

4a6f656c avatar Nov 11 '20 17:11 4a6f656c

Here, I agree with @iceiceleo that OpenBSD (as wel as Solaris, Windows, etc) are in the wrong. A kernel should have a stable programming language-neutral direct syscall interface. It's unfortunate that we have to hurry a patch for Go for this.

beoran avatar Nov 11 '20 19:11 beoran

I think it would be reasonable to have a freeze-exception for changes specific to OpenBSD code and build tags. But the code would need to get it quite soon. I see that some CLs were already sent above; are more CLs needed? Is anybody reviewing those CLs?

ianlancetaylor avatar Nov 11 '20 20:11 ianlancetaylor

I think it would be reasonable to have a freeze-exception for changes specific to OpenBSD code and build tags. But the code would need to get it quite soon.

The bulk of it is OpenBSD specific code in runtime and syscall - there may be a few additional linker related parts, but they likely fall under the bug fix category.

I see that some CLs were already sent above; are more CLs needed?

Yes - I had sent the first set and was avoiding flooding the rest of the CLs. I can get things prepped and mailed out.

Is anybody reviewing those CLs?

With the exception of the library version deduplication CL, not as far as I am aware.

4a6f656c avatar Nov 12 '20 17:11 4a6f656c

Change https://golang.org/cl/249978 mentions this issue: cmd/link/internal/ld: dedup shared libraries on openbsd

gopherbot avatar Nov 12 '20 18:11 gopherbot

@4a6f656c also please read this https://en.wikipedia.org/wiki/Return-to-libc_attack and ask theo: "this syscalls through libc enforcements lowered security. what are you trying to protect against?"

ghost avatar Nov 13 '20 11:11 ghost

Folks, please keep it on-topic. This is not the forum to debate the decisions and security of the OpenBSD project, but only Go's compatibility with current and upcoming OpenBSD versions. Thank you.

FiloSottile avatar Nov 14 '20 02:11 FiloSottile

Change https://golang.org/cl/270377 mentions this issue: cmd/link: emit export for SDYNIMPORT with external linking

gopherbot avatar Nov 16 '20 16:11 gopherbot

Change https://golang.org/cl/270378 mentions this issue: runtime: convert openbsd/amd64 locking to pthreads

gopherbot avatar Nov 16 '20 16:11 gopherbot

Change https://golang.org/cl/270380 mentions this issue: runtime,syscall: convert syscall on openbsd/amd64 to libc

gopherbot avatar Nov 16 '20 16:11 gopherbot

Change https://golang.org/cl/270379 mentions this issue: runtime: switch runtime to libc for openbsd/amd64

gopherbot avatar Nov 16 '20 16:11 gopherbot

@ianlancetaylor I've mailed out the full set of changes needed to convert openbsd/amd64 to libc based syscalls - I'll mail changes for the remaining architectures, however they will be somewhat smaller due to code shared with openbsd/amd64.

There are two outstanding linker issues that I do not yet have fixes for - one is https://github.com/golang/go/issues/41121 (arm), the other is what appears to be an incorrect 4 byte offset in 386 PLTs (I'll try to find a reproducer for this).

4a6f656c avatar Nov 16 '20 18:11 4a6f656c

one is #41121 (arm)

This only affects linking very large binaries. It is probably fine to do it later.

cherrymui avatar Nov 16 '20 21:11 cherrymui

one is #41121 (arm)

This only affects linking very large binaries. It is probably fine to do it later.

Agreed, although we'll have to disable some tests in the interim.

4a6f656c avatar Nov 17 '20 17:11 4a6f656c

@FiloSottile then there is no need for double standards, you (google) should simply drop openbsd compatibility like you already don't care in chromium about compatibility with anything besides mainstream platforms, it's openbsd's responsibility to maintain working ports, not yours, and they are doing it great on their own

ghost avatar Nov 17 '20 17:11 ghost

@ianlancetaylor I've mailed out the full set of changes needed to convert openbsd/amd64 to libc based syscalls - I'll mail changes for the remaining architectures, however they will be somewhat smaller due to code shared with openbsd/amd64.

There are two outstanding linker issues that I do not yet have fixes for - one is #41121 (arm), the other is what appears to be an incorrect 4 byte offset in 386 PLTs (I'll try to find a reproducer for this).

@ianlancetaylor - How do we plan on proceeding here? Of the openbsd/amd64 changes mailed out, so far only one has been reviewed (which cannot be landed since it depends on two others that are unreviewed). Should I try to allocate time to getting this over the line, or just presume it is being shelved (with Go 1.16 being broken on OpenBSD 6.9)?

4a6f656c avatar Jan 14 '21 16:01 4a6f656c

I made comments on CL 250180 (+2'd), CL 250182, and CL 250577, which have not been addressed. I did not review further down the stack, because I think they are similar and it may be better to have the comments addressed before I go further. Thanks.

cherrymui avatar Jan 14 '21 17:01 cherrymui

I made comments on CL 250180 (+2'd), CL 250182, and CL 250577, which have not been addressed. I did not review further down the stack, because I think they are similar and it may be better to have the comments addressed before I go further. Thanks.

@cherrymui - thanks for the response - CL 250180 currently depends on CL 259239 and CL 250179, which means it cannot be merged (or has to be revised to accommodate). I'll address the comments on CL 250182 and CL 250577, however they're not part of the openbsd/amd64 conversion (I was originally doing things lockstep across openbsd platforms, but then mailed out all for openbsd/amd64 per the above comments). If we can get CL 270378, CL 270379 and CL 270380 reviewed (in addition to CL 259239 and CL 250179), then we can fully convert openbsd/amd64 and I can loop back and finish the others based on those changes/reviews.

4a6f656c avatar Jan 14 '21 18:01 4a6f656c

Change https://golang.org/cl/285692 mentions this issue: runtime: remove pthread_kill/pthread_self for openbsd

gopherbot avatar Jan 22 '21 08:01 gopherbot

Aside from the remaining architectures, most of the remaining work is around being able to split this up to merge upstream and I have some concerns about cross-compiling and bootstrapping that I've not yet had time to investigate.

I cross compile for OpenBSD on Linux daily these days. Would some testing be helpful? I don't use cgo though.

kevlar700 avatar Jan 23 '21 00:01 kevlar700

Change https://golang.org/cl/286812 mentions this issue: syscall: clean up mkasm related changes

gopherbot avatar Jan 26 '21 14:01 gopherbot

Change https://golang.org/cl/286813 mentions this issue: runtime: convert openbsd/arm64 locking to libc

gopherbot avatar Jan 26 '21 14:01 gopherbot

Change https://golang.org/cl/286814 mentions this issue: runtime: switch runtime to libc for openbsd/arm64

gopherbot avatar Jan 26 '21 15:01 gopherbot

Change https://golang.org/cl/286815 mentions this issue: runtime,syscall: convert syscall on openbsd/arm64 to libc

gopherbot avatar Jan 26 '21 15:01 gopherbot

@4a6f656c There are some CLs landing for this today, but this issue is currently in the Backlog milestone. Is it expected to be landing these changes at this point in the Go 1.16 release cycle, or should this wait for Go 1.17 instead? Thanks.

CC @golang/release.

dmitshur avatar Jan 26 '21 19:01 dmitshur

@4a6f656c @cherrymui CLs 286812, 286813, 286814 have just landed today, and I see CL 286815 from the same stack is still open and in need of review. We're very close to cutting the Go 1.16 Release Candidate 1 (possibly today).

Can you please confirm if these changes need to land for Go 1.16 before the release candidate is cut, or if it's safe to pause here, or if they should be rolled back and re-sent for Go 1.17 instead? The release team is not aware of a freeze exception and this appears to be a possibly risky change to be landing this late in the freeze (maybe it isn't, someone more familiar with the runtime can comment). This issue milestone is "Backlog" (it's possible I missed some discussion about the timing and safety of these changes because this is a long thread, so we need to clarify it now). Thank you.

dmitshur avatar Jan 27 '21 16:01 dmitshur

@dmitshur - the changes were being landed on the basis of earlier discussions in this issue (https://github.com/golang/go/issues/36435#issuecomment-725569737, https://github.com/golang/go/issues/36435#issuecomment-725633241). The changes are by no means risk free and we could roll them back, however that will mean that Go 1.16 will not work on OpenBSD 6.9. As things currently stand, openbsd/amd64 will continue to work (potentially with one fix), as will openbsd/arm64 with CL 286815. The other three architectures (386, arm, mips64) need further changes, which may have to wait for Go 1.17 (although I'll try to at least get them mailed out this week).

4a6f656c avatar Jan 27 '21 16:01 4a6f656c

Aside from the remaining architectures, most of the remaining work is around being able to split this up to merge upstream and I have some concerns about cross-compiling and bootstrapping that I've not yet had time to investigate.

I cross compile for OpenBSD on Linux daily these days. Would some testing be helpful? I don't use cgo though.

@kevlar700 - any testing to ensure that things continue to work as they did previously will certainly help.

4a6f656c avatar Jan 27 '21 16:01 4a6f656c

@dmitshur - I should also add that while not being risk free, the risk will be impact to Go on OpenBSD, rather than on other platforms.

4a6f656c avatar Jan 27 '21 16:01 4a6f656c

Change https://golang.org/cl/287252 mentions this issue: syscall: remove pad from known openbsd syscalls

gopherbot avatar Jan 27 '21 16:01 gopherbot

@4a6f656c Thanks for answering.

We (@golang/release) discussed this with @cherrymui and @ianlancetaylor and are planning to proceed with the RC, not waiting on the outstanding CLs. Please coordinate with @cherrymui on the few remaining CLs (who'll be paying extra attention in CL reviews to make sure they don't affect any other platforms beyond OpenBSD), and hold off on the rest for Go 1.17.

dmitshur avatar Jan 27 '21 17:01 dmitshur

Change https://golang.org/cl/287532 mentions this issue: syscall: generate readlen/writelen for openbsd libc

gopherbot avatar Jan 28 '21 10:01 gopherbot

Change https://golang.org/cl/287634 mentions this issue: doc/go1.16: document that on OpenBSD syscalls are now made through libc

gopherbot avatar Jan 28 '21 17:01 gopherbot

Change https://golang.org/cl/287652 mentions this issue: runtime: switch openbsd/386 locking to libc

gopherbot avatar Jan 28 '21 18:01 gopherbot

Change https://golang.org/cl/287653 mentions this issue: runtime: switch runtime to libc for openbsd/386

gopherbot avatar Jan 28 '21 18:01 gopherbot

Change https://golang.org/cl/287654 mentions this issue: runtime,syscall: convert syscall on openbsd/386 to libc

gopherbot avatar Jan 28 '21 18:01 gopherbot

Thank you @4a6f656c @cherrymui @randall77 @dmitshur and others for the work and reviews for getting OpenBSD 64-bit using libc. @4a6f656c's CL for OpenBSDB/386 is stalled and didn't make it to Go1.16. Shall we move this issue to Go1.17 and early in cycle so that OpenBSD/386 will be completed, and then in the release notes mark it as fully completed for Go1.17.

odeke-em avatar Feb 08 '21 09:02 odeke-em

Thanks @odeke-em. Moved to 1.17. There are still 386, ARM (32-bit), MIPS64.

cherrymui avatar Feb 08 '21 15:02 cherrymui

Change https://golang.org/cl/312229 mentions this issue: syscall: use libc in Exec on openbsd/arm64

gopherbot avatar Apr 21 '21 15:04 gopherbot

@ianlancetaylor @cherrymui is there any way that we are able to get some traction on this before we reach the end of the current development cycle?

The changes for openbsd/386 were sent out during the previous development cycle and are still pending review. I also have openbsd/arm changes that I was holding until the openbsd/386 changes landed (in order to try to reduce the amount of tangle).

4a6f656c avatar Apr 22 '21 09:04 4a6f656c

Reviewing the 386 CL is still on my list. It is not forgotten. I'll do it as soon as I can. The register ABI stuff has been keeping me occupied, but as it is wrapping up I think I'll have some time. Sorry for the delay.

cherrymui avatar Apr 22 '21 14:04 cherrymui

@cherrymui - no problem, thanks for the update.

4a6f656c avatar Apr 22 '21 17:04 4a6f656c

Change https://golang.org/cl/315794 mentions this issue: runtime: skip TestCrashDumpsAllThreads on openbsd/arm

gopherbot avatar Apr 30 '21 20:04 gopherbot

Change https://golang.org/cl/315790 mentions this issue: runtime: switch openbsd/arm to pthreads

gopherbot avatar Apr 30 '21 20:04 gopherbot

Change https://golang.org/cl/315793 mentions this issue: runtime,syscall: convert syscall on openbsd/arm to libc

gopherbot avatar Apr 30 '21 20:04 gopherbot

Change https://golang.org/cl/315791 mentions this issue: runtime: switch openbsd/arm locking to libc

gopherbot avatar Apr 30 '21 20:04 gopherbot

Change https://golang.org/cl/315792 mentions this issue: runtime: switch runtime to libc for openbsd/arm

gopherbot avatar Apr 30 '21 20:04 gopherbot

Change https://golang.org/cl/317919 mentions this issue: runtime,syscall: simplify openbsd related build tags

gopherbot avatar May 09 '21 17:05 gopherbot

@ianlancetaylor @cherrymui - given that misp64 lacks internal linking support for cgo, I took a look at switching openbsd/mips64 to external-only linking, which would allow for it to be converted to libc syscalls. However, adding openbsd/mips64 to cmd/internal/sys/supported.MustLinkExternal results in:

$ GOROOT_BOOTSTRAP=/home/joel/go-openbsd-mips64-bootstrap ./make.bash  
Building Go cmd/dist using /home/joel/go-openbsd-mips64-bootstrap. (devel go1.17-c14ecaca81 Sun May 9 17:07:22 2021 +0000 openbsd/mips64)
Building Go toolchain1 using /home/joel/go-openbsd-mips64-bootstrap.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
warning: unable to find runtime/cgo.a
Building Go toolchain2 using go_bootstrap and Go toolchain1.                                                      
# cmd/asm                                                                                                         
loadinternal: cannot find runtime/cgo                                                                             
# cmd/cgo                                                                                                         
loadinternal: cannot find runtime/cgo
# cmd/compile                                                                                                     
loadinternal: cannot find runtime/cgo                                                                                                                                                                                               
# cmd/link                                                                                                        
loadinternal: cannot find runtime/cgo                                                                             
Building Go toolchain3 using go_bootstrap and Go toolchain2.
# cmd/asm                                                                                                         
loadinternal: cannot find runtime/cgo                                                                             
# cmd/cgo                                                                                                         
loadinternal: cannot find runtime/cgo
# cmd/compile                                                                                                     
loadinternal: cannot find runtime/cgo                                                                             
# cmd/link
loadinternal: cannot find runtime/cgo
Building packages and commands for openbsd/mips64.
# cmd/addr2line
loadinternal: cannot find runtime/cgo
# cmd/api
loadinternal: cannot find runtime/cgo
# cmd/asm
...

Everything appears to build and work correctly (although I've not yet tried switching to libc syscalls) - any ideas on investigating/fixing loadinternal: cannot find runtime/cgo? An interesting point is that running various tests (e.g. go test path) result in the same error, until I clear the Go cache - after that it builds and runs without the error...

4a6f656c avatar May 11 '21 18:05 4a6f656c

This is probably due to #31544.

If the programs can be built correctly and the only issue is the warning prints, that is probably fine. We have that in the past for some mobile platforms as well, which also require external linking. Maybe you want to add a case to cmd/go/internal/load/pkg.go:externalLinkingForced.

cherrymui avatar May 11 '21 18:05 cherrymui

This is probably due to #31544.

If the programs can be built correctly and the only issue is the warning prints, that is probably fine. We have that in the past for some mobile platforms as well, which also require external linking. Maybe you want to add a case to cmd/go/internal/load/pkg.go:externalLinkingForced.

Ah ha! This is likely the bit that grep was failing to find - thanks, trying that now.

4a6f656c avatar May 11 '21 18:05 4a6f656c

Change https://golang.org/cl/334878 mentions this issue: runtime: fix types in openbsd/amd64 and openbsd/arm64 libc assembly

gopherbot avatar Jul 16 '21 17:07 gopherbot

Change https://golang.org/cl/334877 mentions this issue: runtime: include pthread.h in defs_openbsd.go

gopherbot avatar Jul 16 '21 17:07 gopherbot

Change https://golang.org/cl/334881 mentions this issue: syscall: use correct type for TIOCSPGRP/TIOCGPGRP

gopherbot avatar Jul 16 '21 17:07 gopherbot

Change https://golang.org/cl/334879 mentions this issue: runtime: use asmcgocall_no_g when calling sigprocmask on openbsd

gopherbot avatar Jul 16 '21 17:07 gopherbot

Change https://golang.org/cl/334880 mentions this issue: runtime: correct mips64 asmcgocall signal stack behaviour

gopherbot avatar Jul 16 '21 17:07 gopherbot

Change https://golang.org/cl/342510 mentions this issue: cmd,internal: require external linking for openbsd/mips64

gopherbot avatar Aug 16 '21 18:08 gopherbot

Change https://go.dev/cl/415815 mentions this issue: cmd/link: add internal linking support for calling SDYNIMPORT on mips64

gopherbot avatar Jul 03 '22 14:07 gopherbot

Change https://go.dev/cl/415817 mentions this issue: runtime: switch openbsd/mips64 to pthreads

gopherbot avatar Jul 03 '22 14:07 gopherbot

Change https://go.dev/cl/415816 mentions this issue: cmd/internal,cmd/link: add external linking support for SDYNIMPORT calls on mips64

gopherbot avatar Aug 01 '22 06:08 gopherbot

Change https://go.dev/cl/421077 mentions this issue: runtime,syscall: convert syscall on openbsd/mips64 to libc

gopherbot avatar Aug 03 '22 19:08 gopherbot

Change https://go.dev/cl/421076 mentions this issue: runtime: switch runtime to libc for openbsd/mips64

gopherbot avatar Aug 03 '22 19:08 gopherbot

Change https://go.dev/cl/421075 mentions this issue: runtime: switch openbsd/mips64 locking to libc

gopherbot avatar Aug 03 '22 19:08 gopherbot

Change https://go.dev/cl/421796 mentions this issue: unix: convert openbsd/arm64 to direct libc calls

gopherbot avatar Aug 05 '22 20:08 gopherbot

Change https://go.dev/cl/421795 mentions this issue: unix: make mkasm_darwin.go usable with other operating systems

gopherbot avatar Aug 05 '22 20:08 gopherbot

Change https://go.dev/cl/421797 mentions this issue: unix: convert openbsd/amd64 to direct libc calls

gopherbot avatar Aug 06 '22 08:08 gopherbot

Change https://go.dev/cl/421798 mentions this issue: unix: convert openbsd/386 to direct libc calls

gopherbot avatar Aug 06 '22 10:08 gopherbot

Change https://go.dev/cl/421800 mentions this issue: unix: convert openbsd/arm to direct libc calls

gopherbot avatar Aug 06 '22 17:08 gopherbot

Change https://go.dev/cl/425637 mentions this issue: internal/syscall/unix: convert openbsd (except mips64) to direct libc calls

gopherbot avatar Aug 26 '22 08:08 gopherbot

Change https://go.dev/cl/428776 mentions this issue: internal/poll, syscall: convert writev to direct libc call on openbsd (except mips64)

gopherbot avatar Sep 06 '22 22:09 gopherbot