tailscale icon indicating copy to clipboard operation
tailscale copied to clipboard

F-Droid build (1.76.0) crashes

Open linsui opened this issue 1 year ago • 34 comments

What is the issue?

Since 1.76 builds crashes. 1.76.1-t24929f6b6-ga20b1114dd5 apk can be downloaded from https://gitlab.com/fdroid/fdroiddata/-/jobs/8109300998/artifacts/file/tmp/com.tailscale.ipn_308.apk . Could you please take a look? Thanks

Steps to reproduce

No response

Are there any recent changes that introduced the issue?

No response

OS

Android

OS version

No response

Tailscale version

1.76.1-t24929f6b6-ga20b1114dd5

Other software

No response

Bug report

No response

linsui avatar Oct 18 '24 07:10 linsui

Same issue here on CalyxOS Android 14 FP4.

marek22k avatar Oct 18 '24 09:10 marek22k

Same on Graphene OS. Have downgraded back to 1.72.0

a-jackson avatar Oct 18 '24 11:10 a-jackson

How's it crash? Got any logs?

bradfitz avatar Oct 18 '24 13:10 bradfitz

It never opens. I tap the icon and then I just get the android popup for an app crashed. Tailscale log 1f7f3b58cb0b.txt

a-jackson avatar Oct 18 '24 14:10 a-jackson

Timestamp: 2024-10-18 15:27:33.875801175+0800
Process uptime: 1s
Cmdline: com.tailscale.ipn
pid: 16868, tid: 16893, name: m.tailscale.ipn >>> com.tailscale.ipn <<<
uid: 10230
tagged_addr_ctrl: 0000000000000001 (PR_TAGGED_ADDR_ENABLE)
pac_enabled_keys: 000000000000000f (PR_PAC_APIAKEY, PR_PAC_APIBKEY, PR_PAC_APDAKEY, PR_PAC_APDBKEY)
signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
x0 0000000000000000 x1 00000000000041fd x2 0000000000000006 x3 0000000000000008
x4 0000000000000001 x5 0000000000000000 x6 0000000000000000 x7 0000000000000001
x8 0000000000000083 x9 000000000000db7c x10 000000000000000e x11 0000007412873f9c
x12 0000000000000001 x13 0000007411c5c294 x14 0000007411cc4c73 x15 000000000000000c
x16 0000007480e1f9f0 x17 0000007480f16090 x18 0000007411238000 x19 00000000000041e4
x20 0000007480f15f30 x21 0000007412edd8a0 x22 0000000000000013 x23 0000003cb9d69707
x24 8e41dc0be9fcd734 x25 000000be71521dac x26 0000000000000000 x27 0000000000000010
x28 00000040000021c0 x29 0000004000085c58
lr 0000007411d26554 sp 0000004000085c60 pc 0000007411d4ebf8 pst 0000000080001000
1 total frames
backtrace:
#00 pc 00000000004b1bf8 /data/app/~~BLbjkzLsGmwhSGSN4ZkCxw==/com.tailscale.ipn-lmzUtjCpuQmPqYjQEhOMKg==/base.apk (offset 0x2ac000) 

linsui avatar Oct 18 '24 14:10 linsui

panic: binary built with tailscale_go build tag but failed to read build info or find tailscale.toolchain.rev in build info

Seems to be related to #13527

a-jackson avatar Oct 18 '24 14:10 a-jackson

What makes you think that? Any logs? What's "adb logcat" say while you attempt to start it?

bradfitz avatar Oct 18 '24 14:10 bradfitz

My logs are attached to my previous comment. That's line 446 and 451

a-jackson avatar Oct 18 '24 14:10 a-jackson

Ah, thanks. I only saw the other commenter's logs.

I wish f-droid would test their builds before publishing them.

bradfitz avatar Oct 18 '24 14:10 bradfitz

F-Droid does not develop the app, only packages it.

If the upstream APKs work fine and the F-Droid one misbehaves, yes please open an issue on Gitlab.

licaon-kter avatar Oct 18 '24 14:10 licaon-kter

@linsui looking at your https://gitlab.com/fdroid/fdroiddata/-/commit/00d207cd1981c57fb57607ea6d9d156ecc6886c9#281743176537bad7e7758254b7095f352bea0c57_1554_1555 you mean that even if updating the go lib commit it still crashes?

licaon-kter avatar Oct 18 '24 14:10 licaon-kter

I didn't update the commit. It's read from the go.toolchain.rev file.

linsui avatar Oct 18 '24 14:10 linsui

@linsui it sure looks like the problem was that prior to you disabling the build in https://gitlab.com/fdroid/fdroiddata/-/commit/00d207cd1981c57fb57607ea6d9d156ecc6886c9#281743176537bad7e7758254b7095f352bea0c57_1554_1555 , the tailscale-go srclibs hash was wrong and didn't match the repo's declared go.toolchain.rev.

It's not clear why you were bumping the tailscale-go git hash forward in a commit meant to only disable the build, but in any case, you bumped it to the correct value now, so it should be safe to un-disable the build now.

In summary, F-Droid was building with a bad mix of components and recent Tailscale now detects that mismatch and panicked as a result. That panic is meant to catch such build system screw-ups, but no automated or human testing is involved in F-Droid releases, so the mistake shipped to F-Droid users.

bradfitz avatar Oct 18 '24 15:10 bradfitz

was wrong and didn't match the repo's declared go.toolchain.rev.

read the recipe in full: https://gitlab.com/fdroid/fdroiddata/-/blob/00d207cd1981c57fb57607ea6d9d156ecc6886c9/metadata/com.tailscale.ipn.yml#L1543-L1573

the @... is just the clone point, we later checkout the commit based on your source code file

...
    srclibs:
...
      - tailscale-go@bf15628b759344c6fc7763795a405ba65b8be5d7
    prebuild:
...
    build:
      - export TOOLCHAINREV=$(cat ../go.toolchain.rev)
...
      - pushd $$tailscale-go$$
      - git checkout $TOOLCHAINREV
      - cd src
      - ./make.bash
      - popd
...

/LE: I only asked about the bumped commit because I was not expecting any change in the srclibs section, and also because I didn't know by heart the rest of the recipe

licaon-kter avatar Oct 18 '24 15:10 licaon-kter

the build log is here: https://f-droid.org/repo/com.tailscale.ipn_303.log.gz

...
2024-10-15 15:54:06,921 INFO: Running 'build' commands in build/com.tailscale.ipn/android
2024-10-15 15:54:06,921 DEBUG: Directory: build/com.tailscale.ipn/android
2024-10-15 15:54:06,921 DEBUG: > bash -e -u -o pipefail -x -c export TOOLCHAINREV=$(cat ../go.toolchain.rev); export TOOLCHAINDIR=/home/vagrant/build/srclib/tailscale-go; export ANDROID_NDK_ROOT=/opt/android-sdk/ndk/23.1.7779620; export PATH=$TOOLCHAINDIR/bin:$PATH; make -C .. env; pushd /home/vagrant/build/srclib/tailscale-go; git checkout $TOOLCHAINREV; cd src; ./make.bash; popd; make -C .. libtailscale
/etc/bash.bashrc: line 7: PS1: unbound variable
++ cat ../go.toolchain.rev
+ export TOOLCHAINREV=bf15628b759344c6fc7763795a405ba65b8be5d7
+ TOOLCHAINREV=bf15628b759344c6fc7763795a405ba65b8be5d7
+ export TOOLCHAINDIR=/home/vagrant/build/srclib/tailscale-go
+ TOOLCHAINDIR=/home/vagrant/build/srclib/tailscale-go
+ export ANDROID_NDK_ROOT=/opt/android-sdk/ndk/23.1.7779620
+ ANDROID_NDK_ROOT=/opt/android-sdk/ndk/23.1.7779620
+ export PATH=/home/vagrant/build/srclib/tailscale-go/bin:/opt/android-sdk/ndk/23.1.7779620:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/opt/android-sdk/tools:/opt/android-sdk/platform-tools:/opt/gradle/bin
+ PATH=/home/vagrant/build/srclib/tailscale-go/bin:/opt/android-sdk/ndk/23.1.7779620:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/opt/android-sdk/tools:/opt/android-sdk/platform-tools:/opt/gradle/bin
+ make -C .. env
make: Entering directory '/home/vagrant/build/com.tailscale.ipn'
PATH=/home/vagrant/build/com.tailscale.ipn/android/tool:/home/vagrant/build/com.tailscale.ipn/android/android/build/go/bin:/opt/android-sdk/cmdline-tools/latest/bin:/opt/android-sdk/platform-tools:/home/vagrant/build/srclib/tailscale-go/bin:/opt/android-sdk/ndk/23.1.7779620:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/opt/android-sdk/tools:/opt/android-sdk/platform-tools:/opt/gradle/bin
ANDROID_SDK_ROOT=/opt/android-sdk
ANDROID_HOME=/opt/android-sdk
ANDROID_STUDIO_ROOT=
JAVA_HOME=
TOOLCHAINDIR=/home/vagrant/build/srclib/tailscale-go
AVD_IMAGE=system-images;android-33;google_apis;x86_64
make: Leaving directory '/home/vagrant/build/com.tailscale.ipn'
+ pushd /home/vagrant/build/srclib/tailscale-go
~/build/srclib/tailscale-go ~/build/com.tailscale.ipn/android
+ git checkout bf15628b759344c6fc7763795a405ba65b8be5d7
Previous HEAD position was 66fe5734c4 Merge pull request #85 from tailscale/update-go1.22.0-actions
HEAD is now at bf15628b75 [tailscale] runtime/debug: embed Tailscale toolchain git rev
+ cd src
+ ./make.bash
Building Go cmd/dist using /usr/lib/go-1.22. (go1.22.7 linux/amd64)
...

maybe you can find other issues in it @bradfitz ?

licaon-kter avatar Oct 18 '24 15:10 licaon-kter

the @... is just the clone point, we later checkout the commit based on your source code file

How is it guaranteed that the clone point will contain code from the future to be able to git checkout to? In any case, it does appear that the git checkout bf15628b759344c6fc7763795a405ba65b8be5d7 worked.

Do you see that you're both building the tailscale/go toolchain but that it's also then later downloading it again (from our pre-built releases)?

HEAD is now at bf15628b75 [tailscale] runtime/debug: embed Tailscale toolchain git rev
+ cd src
+ ./make.bash
Building Go cmd/dist using /usr/lib/go-1.22. (go1.22.7 linux/amd64)
Building Go toolchain1 using /usr/lib/go-1.22.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for linux/amd64.
---
Installed Go for linux/amd64 in /home/vagrant/build/srclib/tailscale-go
Installed commands in /home/vagrant/build/srclib/tailscale-go/bin
+ popd
~/build/com.tailscale.ipn/android
+ make -C .. libtailscale
make: Entering directory '/home/vagrant/build/com.tailscale.ipn'
mkdir -p android/libs
./tool/go install golang.org/x/mobile/cmd/gobind
go: downloading golang.org/x/mobile v0.0.0-20240806205939-81131f6468ab
go: downloading golang.org/x/tools v0.24.0
go: downloading golang.org/x/sync v0.8.0
go: downloading golang.org/x/mod v0.20.0
./tool/go install golang.org/x/mobile/cmd/gomobile
# Downloading Go toolchain bf15628b759344c6fc7763795a405ba65b8be5d7
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

  2 65.1M    2 1409k    0     0  1859k      0  0:00:35 --:--:--  0:00:35 1859k
 55 65.1M   55 36.4M    0     0  21.0M      0  0:00:03  0:00:01  0:00:02 35.9M
100 65.1M  100 65.1M    0     0  24.6M      0  0:00:02  0:00:02 --:--:-- 33.8M
go: downloading golang.org/x/sys v0.22.0
go: downloading golang.org/x/mod v0.19.0
go: downloading github.com/google/uuid v1.6.0
go: downloading github.com/go-json-experiment/json v0.0.0-20231102232822-2e55bd4e08b0
go: downloading go4.org/mem v0.0.0-20220726221520-4f986261bf13
go: downloading golang.org/x/net v0.27.0

That is, you're not entirely building this from source, as is the F-Droid way. You're still using our pre-built binaries in this recipe, for at least parts.

bradfitz avatar Oct 18 '24 15:10 bradfitz

How is it guaranteed that the clone point will contain code from the future to be able to git checkout to?

The srclib stage clones that repo at HEAD, then checks out that @commit. The commit mentioned in the srclib section is just a point it time that later we don't care about, for this app at least.

licaon-kter avatar Oct 18 '24 16:10 licaon-kter

Gotcha. In any case, it seems like the recipe is using a mix of Go toolchains.

That curl output in there from ./tool/go (https://github.com/tailscale/tailscale-android/blob/main/tool/go#L13-L71) means that TOOLCHAINDIR is not set, so it's falling back to downloading our toolchain from GitHub, not using the one you just built.

bradfitz avatar Oct 18 '24 16:10 bradfitz

https://gitlab.com/fdroid/fdroiddata/-/jobs/8109300998#L6115 But TOOLCHAINDIR is set.

linsui avatar Oct 18 '24 16:10 linsui

I uninstalled Tailscale from F-Droid. If I try to install an earlier version, an error pops out saying "Update version code 254 is older than current 303". If I try to install in Google Play, says "another user has already installed an incompatible version". Is there another workaround at this moment?

chunjiw avatar Oct 18 '24 17:10 chunjiw

@chunjiw looks like the F-Droid version remained installed? Perhaps try rebooting the phone after uninstalling.

We also publish APKs on our website nowadays so give that a try, these are the same builds as the Play Store: https://pkgs.tailscale.com/stable/#android (Even though the APK might work, I recommend switching to the Play Store when you get a chance, so you can get automatic updates which is better for security).

agottardo avatar Oct 18 '24 17:10 agottardo

I tried rebooting the phone, updating F-Droid, restarting F-Droid, nothing works. I also tried directly install APK as downloaded from link you provided (thank you 👍 ) but it says "App not installed as package conflicts with an existing package". Seems like F-Droid does not properly let the system know that this app is uninstalled.

chunjiw avatar Oct 18 '24 17:10 chunjiw

@chunjiw F-Droid does not do installs and uninstalls itself, only the system does this.

Go to Android Settings - Apps - Tailscale - Uninstall to retry

licaon-kter avatar Oct 18 '24 17:10 licaon-kter

Thank you for the suggestion! But I cannot even find Tailscale in Settings - Apps list.

chunjiw avatar Oct 18 '24 17:10 chunjiw

Also experienced crash-on-open from Tailscale after updating to 1.76.0 from the f-droid repository.

I wiped cache, uninstalled, and reinstalled version 1.76.0 from f-droid and got the same crash-on-open behavior.

I uninstalled 1.76.0 and installed 1.72.0 from f-droid (also checked the "skip this version" so it wouldn't automatically upgrade) and now it's back to working fine.

Not sure where to find logs for the crashes.

QantumEntangled avatar Oct 18 '24 18:10 QantumEntangled

@linsui, the curl is coming via the TAILSCALE_VERSION=$(shell ./version/tailscale-version.sh 200) in the Makefile, which sets TS_USE_TOOLCHAIN=1 (it probably shouldn't) which then calls into the tailscale/tailscale repo's build_dish.sh, which does:

go="go"
if [ -n "${TS_USE_TOOLCHAIN:-}" ]; then
        go="./tool/go"
fi

And it's that ./tool/go (our gocross-wrapper.sh) that ends up calling curl.

And of course there's another git clone of the whole tailscale git repo into a temp directory to be able to look at the git history, of course.

This is all way too convoluted 😭

/cc @raggi

bradfitz avatar Oct 18 '24 18:10 bradfitz

@chunjiw where you asked to keep app data on uninstall?

If yes, reinstall same version from F-Droid, uninstall it ...but do not keep data.

Then install 1.72 and skip the update like the user above did

licaon-kter avatar Oct 18 '24 18:10 licaon-kter

where you asked to keep app data on uninstall?

Unfortunately no. It just asks for confirmation then proceeds to uninstall. I'm using Galaxy S24+

chunjiw avatar Oct 18 '24 18:10 chunjiw

Same here on GrapheneOS Android 14, Pixel 8. My logs look a bit different than the ones above so posted here:

https://pastebin.com/ujEgU6yS https://pastebin.com/yqiBTRru

Sidenote: I saw that Google play store has v 1.74 while F-Droid has 1.72 (my previously used version) and the new version 1.76. Weird. Opted for the play store 1.74 until next release is available on F-Droid.

slipperybeluga avatar Oct 19 '24 05:10 slipperybeluga

Has there been any update on resolving this issue?

skywalker478 avatar Oct 19 '24 08:10 skywalker478