headscale icon indicating copy to clipboard operation
headscale copied to clipboard
verified publisher icon juanfont

[Bug] unaligned 64-bit atomic operation

Open trackmafia opened this issue 8 months ago • 2 comments

Is this a support request?

  • [x] This is not a support request

Is there an existing issue for this?

  • [x] I have searched the existing issues

Current Behavior

fresh install of headscale server, TLS-ALPN-01 SSL enabled. Using Raspberry PI 3, 32bit arm7 install as server. Opnsense 25.1.4 router as client.

tailscale client on opnsense router. node registers but hangs at client. tailscale client never logs in. On headscale server, node is approved but never comes online.

Expected Behavior

after authorizing node, client would log in and join the tailnet

Steps To Reproduce

  1. install fresh server, configure for SSL
  2. install client and try to connect.

Environment

- OS: Raspbian 12 32bit client. Opnsense router on x86 hardware
- Headscale version: 0.25.1 and 0.24.3
- Tailscale version: 1.82

Runtime environment

  • [ ] Headscale is behind a (reverse) proxy
  • [ ] Headscale runs in a container

Debug information

here are the logs:

Mar 30 16:15:35 headscale[16010]: 2025-03-30T16:15:35+01:00 INF ../../../home/runner/work/headscale/headscale/hscontrol/poll.go:634 > node has connected, mapSession: 0x44244b8, chan: 0x4501080 node=J> Mar 30 16:15:35 headscale[16010]: 2025-03-30T16:15:35+01:00 INF ../../../home/runner/work/headscale/headscale/hscontrol/poll.go:634 > node has disconnected, mapSession: 0x44244b8, chan: 0x4501080 nod> Mar 30 16:15:35 headscale[16010]: 2025/03/30 16:15:35 http2: panic serving 1.2.3.4:21191: unaligned 64-bit atomic operation Mar 30 16:15:35 headscale[16010]: goroutine 1294 [running]: Mar 30 16:15:35 headscale[16010]: golang.org/x/net/http2.(*serverConn).runHandler.func1() Mar 30 16:15:35 headscale[16010]: /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/server.go:2468 +0x184 Mar 30 16:15:35 headscale[16010]: panic({0x15eeb80, 0x1b66b18}) Mar 30 16:15:35 headscale[16010]: /nix/store/0b6vsy4fa4i4qpk1011hi6251nwdg5y8-go-1.23.4/share/go/src/runtime/panic.go:785 +0xfc Mar 30 16:15:35 headscale[16010]: internal/runtime/atomic.panicUnaligned() Mar 30 16:15:35 headscale[16010]: /nix/store/0b6vsy4fa4i4qpk1011hi6251nwdg5y8-go-1.23.4/share/go/src/internal/runtime/atomic/unaligned.go:8 +0x24 Mar 30 16:15:35 headscale[16010]: internal/runtime/atomic.Xadd64(0x461d7b4, 0x1) Mar 30 16:15:35 headscale[16010]: /nix/store/0b6vsy4fa4i4qpk1011hi6251nwdg5y8-go-1.23.4/share/go/src/internal/runtime/atomic/atomic_arm.s:265 +0x14 Mar 30 16:15:35 headscale[16010]: github.com/juanfont/headscale/hscontrol/mapper.(*Mapper).marshalMapResponse(0x461d780, {0x73, {0x488f298, 0x4}, 0x1, {{0x68, 0x51, 0xfc, 0x7b, 0x1, ...}}, ...}, ...) Mar 30 16:15:35 headscale[16010]: /home/runner/work/headscale/headscale/hscontrol/mapper/mapper.go:331 +0x3c Mar 30 16:15:35 headscale[16010]: github.com/juanfont/headscale/hscontrol/mapper.(*Mapper).FullMapResponse(0x461d780, {0x73, {0x488f298, 0x4}, 0x1, {{0x68, 0x51, 0xfc, 0x7b, 0x1, ...}}, ...}, ...) Mar 30 16:15:35 headscale[16010]: /home/runner/work/headscale/headscale/hscontrol/mapper/mapper.go:197 +0xfc Mar 30 16:15:35 headscale[16010]: github.com/juanfont/headscale/hscontrol.(*mapSession).serveLongPoll(0x44244b8) Mar 30 16:15:35 headscale[16010]: /home/runner/work/headscale/headscale/hscontrol/poll.go:289 +0x1238 Mar 30 16:15:35 headscale[16010]: github.com/juanfont/headscale/hscontrol.(*noiseServer).NoisePollNetMapHandler(0x46bc100, {0x1b75960, 0x49524a0}, 0x47b86e8) Mar 30 16:15:35 headscale[16010]: /home/runner/work/headscale/headscale/hscontrol/noise.go:225 +0x250 Mar 30 16:15:35 headscale[16010]: net/http.HandlerFunc.ServeHTTP(0x47480c8, {0x1b75960, 0x49524a0}, 0x47b86e8) Mar 30 16:15:35 headscale[16010]: /nix/store/0b6vsy4fa4i4qpk1011hi6251nwdg5y8-go-1.23.4/share/go/src/net/http/server.go:2220 +0x34 Mar 30 16:15:35 headscale[16010]: github.com/juanfont/headscale/hscontrol.prometheusMiddleware.func1({0x1b75960, 0x49524a0}, 0x47b86e8) Mar 30 16:15:35 headscale[16010]: /home/runner/work/headscale/headscale/hscontrol/metrics.go:82 +0x244 Mar 30 16:15:35 headscale[16010]: net/http.HandlerFunc.ServeHTTP(0x4aa8700, {0x1b75960, 0x49524a0}, 0x47b86e8) Mar 30 16:15:35 headscale[16010]: /nix/store/0b6vsy4fa4i4qpk1011hi6251nwdg5y8-go-1.23.4/share/go/src/net/http/server.go:2220 +0x34 Mar 30 16:15:35 headscale[16010]: github.com/gorilla/mux.(*Router).ServeHTTP(0x46fa0c0, {0x1b75960, 0x49524a0}, 0x499c638) Mar 30 16:15:35 headscale[16010]: /home/runner/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:212 +0x1dc Mar 30 16:15:35 headscale[16010]: golang.org/x/net/http2.(*serverConn).runHandler(0x479e788, 0x49524a0, 0x499c638, 0x4911800) Mar 30 16:15:35 headscale[16010]: /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/server.go:2475 +0xd8 Mar 30 16:15:35 headscale[16010]: created by golang.org/x/net/http2.(*serverConn).scheduleHandler in goroutine 216 Mar 30 16:15:35 headscale[16010]: /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/server.go:2409 +0x248

trackmafia avatar Mar 30 '25 17:03 trackmafia

I'm seeing the same thing on 32-bit Arch Linux ARM (armv7l). I have tried:

  • The package from Arch Linux ARM (headscale-0.25.1-1)
  • The ARMv7 binary from the GitHub release for 0.25.1 (https://github.com/juanfont/headscale/releases/download/v0.25.1/headscale_0.25.1_linux_armv7)
  • The ARMv7 binary from the GitHub release for 0.24.3 (https://github.com/juanfont/headscale/releases/download/v0.24.3/headscale_0.24.3_linux_armv7)

I'm running Headscale behind an nginx reverse proxy (however it is using stream + ssl_preread, so the traffic is pretty much untouched by nginx).

muru avatar Apr 12 '25 18:04 muru

same issue; same context. trying with armv6 now.

e: nope, same issue

habnabit avatar Apr 14 '25 20:04 habnabit

This issue is stale because it has been open for 90 days with no activity.

github-actions[bot] avatar Jul 14 '25 02:07 github-actions[bot]

has this bug been identified or patched? i haven’t checked if there’s any newer releases Message ID: @.***>

habnabit avatar Jul 14 '25 02:07 habnabit

No, and I think realistically we will not put any effort into solving this. I have no 32 bit hardware and its a runtime error so we wont be able to catch it in tests.

Happy to take contributions as long as they fit within maintainability and so on, but it will be hard for us to guarantee that we dont break it again in the future.

kradalby avatar Jul 15 '25 09:07 kradalby

The issue was that atomic.AddUint64[0] was used on a uint64 struct member, which happened to be unaliged because of the struct layout.

This code was already removed and in other places the where atomic are used the bug is fixed because the atomic.Int64[1] type is used, which provide the alignment.

  • [0] https://github.com/juanfont/headscale/blob/v0.25.1/hscontrol/mapper/mapper.go#L331
  • [1] https://github.com/juanfont/headscale/blob/0a43aab8f5c876935f84ab9725f0e8b47dffe809/hscontrol/mapper/batcher_lockfree.go#L37

Duncaen avatar Oct 30 '25 21:10 Duncaen