Android testing CI faliing (all platforms)

[c-parsons]

Android Testing CI has been failing for many days, and thus it's been difficult to diagnose issues with it failing with Bazel@HEAD

Here is a failure for Unbuntu 16 this morning using Bazel@HEAD:

(02:09:03) ERROR: missing input file 'external/androidsdk/build-tools/28.0.2/lib/dx.jar', owner: '@androidsdk//:build-tools/28.0.2/lib/dx.jar'
--
  | (02:09:03) ERROR: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/d3e6522be4da9d904ab54b448eb64f5e/external/androidsdk/BUILD.bazel:8:1: @androidsdk//:dx_jar_import: missing input file '@androidsdk//:build-tools/28.0.2/lib/dx.jar'

I think this dependency was updated in a recent commit?

https://buildkite.com/bazel/android-testing

[jin]

It looks like the latest green is still using 28.0.2, whereas this project's HEAD is already using 29.0.2, which no longer exists on Bazel CI.

Perhaps we can also install 28.0.2 build tools on Bazel CI? @philwo

[philwo]

@jin I'll install all 28.x and 29.x build tools releases on our CI machines today, but will this actually fix the issue? I thought we're incompatible with <29.x since https://github.com/bazelbuild/bazel/issues/10079?

[jin]

This is a chicken and egg problem :) the build-tools were updated to 29.0.2, which the Android rules now need (for a specific build scenario), but the latest green commit is still using 28.0.2, which the CI doesn't have, so it's failing with missing input errors. We need 28.0.2 on the CI to at least get past the missing input errors, or maybe manually update the latest green commit to f08dda59e6e1c9dafd312b1b0c0ef1188bd6d9fe?

[philwo]

LGTM, I'll roll out new images with 28.x and 29.x build tools on all platforms tomorrow morning and will retry the failed jobs then. 👍

[jin]

thanks @philwo!

[philwo]

@jin These versions are installed now:

"build-tools;28.0.2"
"build-tools;28.0.3"
"build-tools;29.0.0"
"build-tools;29.0.2"
"build-tools;29.0.3"

However, a rebuild still fails: https://buildkite.com/bazel/android-testing/builds/641

Do we have to revert https://github.com/android/testing-samples/commit/f08dda59e6e1c9dafd312b1b0c0ef1188bd6d9fe because this project isn't compatible with build-tools 29.0.2?

[jin]

@philwo the ubuntu failures are https://github.com/android/testing-samples/issues/303, will be fixed by https://github.com/android/testing-samples/pull/309

I'm thinking of disabling the Windows pipeline. There isn't anyone working on Windows support for Android now.