Support for offline mode with previously installed bazelisk

[HemangLavana]

Currently if I run bazelisk, when my PC is in offline mode, it fails to run my build/test tasks as it is not able to connect to the internet and validate that it is running the latest version.

$ ./tools/bin/bazelisk-darwin-amd64 help 2019/09/08 08:15:36 could not resolve the version 'latest' to an actual version number: could not get releases from github.com/bazelbuild/bazel: could not download list of Bazel releases from github.com/bazelbuild: could not fetch https://api.github.com/repos/bazelbuild/bazel/releases: Get https://api.github.com/repos/bazelbuild/bazel/releases: dial tcp: lookup api.github.com: no such host $

It would be great if it could simply print a warning message about this and continue with running the bazel build/test tasks. And perhaps provide an option to run bazelisk in offline mode so that it doesn't have to print warning message.

[philwo]

Hi @HemangLavana,

that's a great feature request and something I definitely would like to add. Happy to review PRs for this in case anyone wants to give it a try, otherwise I'll take it on the next time I have time to work on Bazelisk. :)

[ulfjack]

Our CI is failing regularly with a similar error message even with internet access. It seems like api.github.com is unreliable?

This is a concern because GitHub installs bazelisk at least on their Mac workers as bazel, resulting in a bad user experience for projects that use GitHub actions w/ bazel for CI.

[madhavajay]

This is maddening!?

I don't seem to be able to bypass bazelisk and just use my own copy of bazel. https://github.com/OpenMined/SwiftDP/runs/649228808?check_suite_focus=true

curl -LO https://github.com/bazelbuild/bazel/releases/download/3.0.0/bazel-3.0.0-darwin-x86_64
mkdir -p "${GITHUB_WORKSPACE}/bin/"
mv bazel-3.0.0-darwin-x86_64 "${GITHUB_WORKSPACE}/bin/bazel"
chmod +x "${GITHUB_WORKSPACE}/bin/bazel"
${GITHUB_WORKSPACE}/bin/bazel --version
which bazel
set -o pipefail
/usr/local/bin/bazel build //src:SwiftDP --ios_multi_cpus=x86_64,arm64 -apple_bitcode=embedded --copt=-fembed-bitcode

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   636  100   636    0     0   6000      0 --:--:-- --:--:-- --:--:--  6000

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
 39 39.7M   39 15.5M    0     0  12.0M      0  0:00:03  0:00:01  0:00:02 15.8M
100 39.7M  100 39.7M    0     0  17.8M      0  0:00:02  0:00:02 --:--:-- 20.8M
bazel 3.0.0
/usr/local/bin/bazel
2020/05/06 11:01:23 could not resolve the version 'latest' to an actual version number: could not get releases from github.com/bazelbuild/bazel: could not download list of Bazel releases from github.com/bazelbuild: unexpected status code while reading https://api.github.com/repos/bazelbuild/bazel/releases: 403
##[error]Process completed with exit code 1.

Huh? 🤷‍♀️

[philwo]

@madhavajay In your code snippet, you download Bazel 3.0.0 manually using curl and put it into a folder called bin underneath your workspace. Then you go on and don't actually run it, but instead run /usr/local/bin/bazel (which seems to be Bazelisk).

Of course this will run Bazelisk and not the binary you just downloaded. If you want to run your download copy of Bazel, you'll have to run that and not the one in /usr/local/bin.

If you expected that Bazelisk would detect that a file called $workspace/bin/bazel exists, is the correct version and then use that: This feature sounds interesting, but it doesn't exist - is there a reason you assumed that would work?

[philwo]

This is a concern because GitHub installs bazelisk at least on their Mac workers as bazel, resulting in a bad user experience for projects that use GitHub actions w/ bazel for CI.

Oh, wow - they install Bazelisk on their machines? By default? 😃

I agree that their API endpoint reliability is somewhat lacking. I'll make sure that we improve Bazelisk to not rely on it for use cases where we can get the information from a more reliable source.

[madhavajay]

@philwo I hadn't heard of bazelisk till yesterday. :) I love the idea, just wondering if there is a way to override this functionality by say passing it a local binary for instance (or passing it a preferred download url for a specific bazel binary) like this where the local install is just stomping my usage but also failing. I figured out that I wasn't using the paths correctly thanks to your comment, it was late last night when I was just trying to get the build to pass. I haven't used GitHub Workflow before so I didn't realise it seems like that downloaded binary of mine wasn't available in the subsequent task, so for now im just running it in 1 step.

The good news is I have my tests passing for now: https://github.com/OpenMined/SwiftDP/runs/651254212?check_suite_focus=true

Do you know if this is just an issue at github's end and that it should normally work?

[philwo]

@madhavajay Great that you got it working! I haven't played with GitHub Actions and their specific environment yet, but if I do and find out something, we can update the docs here with some guidance.

Regarding overriding the mechanism - if you put your downloaded binary into $workspace/tools/bazel, Bazelisk should detect it and automatically use it. :)

$ cd ~/workspace
$ bazel version
Build label: 3.1.0

$ curl -LO https://github.com/bazelbuild/bazel/releases/download/2.1.0/bazel-2.1.0-darwin-x86_64
$ mv bazel-2.1.0-darwin-x86_64 tools/bazel
$ chmod +x tools/bazel

$ bazel version
Build label: 2.1.0

[fenollp]

For Github Actions one should set the env BAZELISK_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} or rate limitting will hit eventually (it hits sooner on macos workers as there are fewer of them and RL is by IP).

Example usage: https://github.com/fenollp/bazel_upgradable/blob/master/.github/workflows/workflow.yml#L5

[ulfjack]

@fenollp Is that documented anywhere?

[madhavajay]

@philwo thats a great work around I will investigate that. @fenollp that makes more sense. It would be nice to see that in the error message actually e.g.:

 could not download list of Bazel releases from github.com/bazelbuild: ...
Do you have a BAZELISK_GITHUB_TOKEN set?

Or something similar.

[adam3141]

Just wanted to know if this issue is still being looked at because I think this is an issue for bazelisk when the internet may be temporarily unavailable

Notes:

• I have copied bazelisk as bazel and put it in my system path.
• bazelisk is trying to get the latest version

I did a quick test to confirm my suspicions which is shown below.

family@family-desktop:~/Development/stm32$ bazel build //test
INFO: Analyzed target //test:test (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //test:test up-to-date:
  bazel-bin/test/test
INFO: Elapsed time: 0.269s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action

At this point, I switched off the network connection and ran the command again

family@family-desktop:~/Development/stm32$ bazel build //test
2022/05/14 16:51:07 could not resolve the version 'latest' to an actual version number: unable to determine latest version: could not list Bazel versions in GCS bucket: could not list GCS objects at https://www.googleapis.com/storage/v1/b/bazel/o?delimiter=/: could not fetch https://www.googleapis.com/storage/v1/b/bazel/o?delimiter=/: Get "https://www.googleapis.com/storage/v1/b/bazel/o?delimiter=/": dial tcp: lookup www.googleapis.com on 127.0.0.53:53: server misbehaving

If I specify a version by giving USE_BAZEL_VERSION = 4.2.0, then it will use the cached version regardless of my connection status, however I do think it is an issue that bazelisk aborts when there is no network connection when it is trying to use the latest Bazel version.

Perhaps a change in the way the algorithm determines which version of Bazel to run is needed so that a network connectivity issue will not abort a build.

• If the environment variable USE_BAZEL_VERSION is set, it will use the version specified in the value.
• Otherwise, if a .bazeliskrc file exists in the workspace root and contains the USE_BAZEL_VERSION variable, this version will be used.
• Otherwise, if a .bazelversion file exists in the current directory or recursively any parent directory, it will read the file and use the version specified in it.
• Otherwise it will use the official latest Bazel release.
• Otherwise it will use the cached Bazel release.

[guw]

Is there a way to run Bazelisk offline? We also install Bazelisk as default bazel on our developer machines.

[adam3141]

Is there a way to run Bazelisk offline? We also install Bazelisk as default bazel on our developer machines.

There is if you want to peg your Bazel version to a specific version, like I mentioned in above comment. For now though, you're out of luck if you try and use the latest version.

I hope this issue gets fixed in one way or another.

[janeusz2000]

strategic comment to track this PR .... would be great If I had this feature included in my bazelisk instance :)

[fishy]

I hit this issue the other day when I try to use bazel to run unit tests while offline, and I have to temporarily create a .bazelversion file to make it work.

I also didn't realize about the CI issue (we build a base docker image contains bazelisk and caches the latest bazel version when building it so we don't have to download the latest bazel version on every CI run), but that does make a lot of sense for an offline use case.

Considering the two separated issues, I think maybe two environment variables/.bazeliskrc options would makes sense:

1. BAZELISK_FORCE_CACHED_BAZEL for CI use cases: treat the latest locally cached bazel version as latest and bypass github.
2. BAZELISK_ALLOW_OFFLINE: when it cannot talk to github but there are locally cached bazel versions, print a big warning and use the locally cached version as a fallback.