Android engine test flaking on failure to fetch dependencies

[zanderso]

Feature: Migrate gradle to use --offline mode to not download dependencies dynamically during CI.

In https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20Android%20Debug%20Engine/20211/overview

ERROR: Failed to fetch maven artifact org.robolectric:android-all-instrumented:4.2.2_r1.2-robolectric-r1-i3

This flake is causing Android build/test times on CI to increase by several minutes due to automatic re-tries.

[reidbaker]

We should not be fetching dependencies as part of our CI. I am not familiar with this code but it is something my team can make sure gets fixed.

io.flutter.plugin.localization.LocalizationPluginTest > computePlatformResolvedLocale_selectOnlyLanguageLocale_beforeAndroidN[17] STANDARD_ERROR
    ERROR: Failed to fetch maven artifact org.robolectric:android-all-instrumented:4.2.2_r1.2-robolectric-r1-i3
    java.util.concurrent.ExecutionException: javax.net.ssl.SSLProtocolException: Connection reset
    	at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:552)
    	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:533)
    	at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:90)
    	at org.robolectric.internal.dependency.MavenArtifactFetcher.fetchArtifact(MavenArtifactFetcher.java:97)
    	at org.robolectric.internal.dependency.MavenDependencyResolver.lambda$getLocalArtifactUrls$0(MavenDependencyResolver.java:80)
    	at org.robolectric.internal.dependency.MavenDependencyResolver.whileLocked(MavenDependencyResolver.java:100)
    	at org.robolectric.internal.dependency.MavenDependencyResolver.getLocalArtifactUrls(MavenDependencyResolver.java:75)
    	at org.robolectric.internal.dependency.MavenDependencyResolver.getLocalArtifactUrls(MavenDependencyResolver.java:65)
    	at org.robolectric.internal.dependency.MavenDependencyResolver.getLocalArtifactUrl(MavenDependencyResolver.java:116)

[reidbaker]

Links and resources that might be helpful I think this is the test that failed. https://github.com/flutter/engine/blob/main/shell/platform/android/test/io/flutter/plugin/localization/LocalizationPluginTest.java Luci python file https://flutter.googlesource.com/recipes/+/refs/heads/main/recipes/engine/engine.py

Command that failed to find any mention of localization plugin testing in recipe repo ❯ find . -type f -exec grep 'LocalizationPluginTest' {} \;

[reidbaker]

Main entry point for engine tests: https://github.com/flutter/engine/blob/main/testing/run_tests.py

What I think is the most recent version of the dependencies that are available locally https://chrome-infra-packages.appspot.com/p/flutter/android/robolectric_bundle/+/

Pull request that migrated android to the current dependency support. https://github.com/flutter/engine/pull/17116

[reidbaker]

Current theory android-all-instrumented-4.2.2_r1.2-robolectric-r1-i4.jar is not included in the CIPD cash so gradle is downloading the dependency dynamically. I am working with @ricardoamador to try to prove that if CIPD has an update jar this issue is resolved.

Also note that gradle has an offline mode that will make our tests fail if we do not add the dependency to cipd. We might consider enabling it to see what breaks. https://docs.gradle.org/current/userguide/dynamic_versions.html#sec:controlling_dependency_caching_command_line

[reidbaker]

PR to use offline mode https://github.com/flutter/engine/pull/39465 Failed build https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20Android%20Debug%20Engine/48013/overview

[reidbaker]

cipd create -pkg-def=somefile.yaml Where somefile.yaml has the contents below from a directory that is the current contents of https://chrome-infra-packages.appspot.com/p/flutter/android/robolectric_bundle/+/ with any additional jars that need to be included in a folder named lib

package: flutter/android/robolectric_bundle
description: Dependencies for robolectric android testing
data:
    - dir: lib

[reidbaker]

CIPD has been update with a robolectric package but I am not sure if that will solve the initial flake on its own.

[reidbaker]

Another relevant link https://cs.opensource.google/flutter/engine/+/main:DEPS;l=662

[zanderso]

We can check in instructions for updating CIPD bundles here https://github.com/flutter/engine/tree/main/tools/cipd

[reidbaker]

Met with @christopherfujino here are the notes from our meeting.

• DEPS files are for built dependencies and CIPD is for prebuilt dependencies, ci.yaml is for dependencies that are part of your source (not sure I follow this)
• I am not missing lots of information, finding dependencies is actually this hard and it is not surprising that I have spent hours looking for where dependencies come from.
• Chris says "we agree that we don't want to download things during builds but we do because dependency management is so difficult"
• We looked through the builds and didn't see anyplace where roboletric is downloaded. 3 Ideas for what could be happening. 1. we missed it, and one of these steps explicitly download robolectric 2. robolectric was already on the bot before the build started

3. one of these steps was implicitly fetching robolectric

• go/luci-users-chat might be a good place to ask if there is a way to see if a build is using a package.
• Our best guess is that that the CIPD update we did last week is not affecting the current builds.

[reidbaker]

Another place dependencies are defined is in the docker image used in ci. Defined here https://github.com/flutter/flutter/blame/master/dev/ci/docker_linux/Dockerfile

[reidbaker]

https://github.com/flutter/flutter/issues/121233#issuecomment-1450644236 is another instance of reaching out to the network.

[reidbaker]

https://github.com/flutter/flutter/issues/121711 another instance

[reidbaker]

A solution for gradle can probably be broken apart from from the rest of the way dependencies are cached.

We can follow the instructions here to host the gradle files in the correct directory. Then we can modify the dockerfile to have the versions of gradle we depend on (it is a lot).

[reidbaker]

related #125232

[reidbaker]

https://maven.apache.org/guides/mini/guide-mirror-settings.html

[reidbaker]

Could potentially run ./gradlew dependencies as an infra step before to download dependencies https://stackoverflow.com/questions/50183848/gradle-task-to-only-download-maven-dependencies/50219857#50219857

[zanderso]

Could potentially run ./gradlew dependencies as an infra step before to download dependencies https://stackoverflow.com/questions/50183848/gradle-task-to-only-download-maven-dependencies/50219857#50219857

That sounds promising. It would also allow scoping retries down to the infra step by itself.

[reidbaker]

Found a team at google that offer a maven proxy that might solve this for us without needed to standup our own server. Will update with the result of that conversation.

[reidbaker]

Pr to try to move dependency download to infra step https://flutter-review.googlesource.com/c/recipes/+/45320

[reidbaker]

https://ci.chromium.org/raw/build/logs.chromium.org/flutter/led/reidbaker_google.com/cd9b45dee2af563478e15dd6779230b28d58d233bc463ddf1386e982411ef96a/+/build.proto?server=chromium-swarm.appspot.com

Build is waiting on submit. Link is a passing led run of this change. led get-build 8777549308103346881 | led edit-system -p 20 | led edit-recipe-bundle | led launch -modernize

[reidbaker]

That pr is an example of what to do. Looking at the original report I also need to modify the engine/engine recipe

[reidbaker]

After Digging into the recipes/engine/engine.py and recipes/devicelab/devicelab_drone.py I could not figure out where android tests were specifically happening. Nor did I see a setup state that could be modified per project.

[zanderso]

It sounds like the question is how to add an infra step like in https://flutter-review.googlesource.com/c/recipes/+/45320 to the engine v2 android build configs like in https://github.com/flutter/engine/blob/main/ci/builders/linux_android_debug_engine.json.

I think @godofredoc would know whether that's possible with engine v2.

[reidbaker]

FWIW I would be happy to pair with someone on infra or engine if we wanted to tackle this together.

[zanderso]

@godofredoc is OOO, but maybe @keyonghan knows if it's possible to add something like https://flutter-review.googlesource.com/c/recipes/+/45320 as an optional step to an engine v2 config?

[keyonghan]

We can put the gradle dependency logics as a recipes module, which can then be used by any other recipes, including existing android_views.py, and to-be-added engine v2 & devicelab_drone.py.

To support as an optional operation (for engine v2), contexts seems a good fit. The above module can be encapsulated as a context, and then from builder config side, add the context if the test needs the gradle dependency.

For android_views and devicelab_drone, we can either use the same contexts property, or directly call the recipes module to add the gradle logic.

[reidbaker]

Packages now has the ability to mark steps as infra steps and a new command to download dependencies is called and marked is infra.

I need to audit framework for more instances where we do not download gradle but packages should be handled. Engine repo however (one of the cause of this bug) is not done.

[flutter-triage-bot[bot]]

This issue is assigned to @reidbaker but has had no recent status updates. Please consider unassigning this issue if it is not going to be addressed in the near future. This allows people to have a clearer picture of what work is actually planned. Thanks!

[loic-sharma]

This issue also affects the Flutter framework's tests. The tree closed 2+ times this week due to Gradle network errors:

1. https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20tool_integration_tests_3_4/16981/overview - Exception in thread "main" java.net.SocketException: Connection reset, caused by: https://github.com/flutter/flutter/blob/6193615f90a800ff5623373137bd88ea67d5a1d2/packages/flutter_tools/test/integration.shard/android_gradle_outputs_app_link_settings_test.dart#L318-L326

2. https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20tool_integration_tests_3_4/16954/overview - Exception in thread "main" java.net.SocketException: Connection reset, caused by https://github.com/flutter/flutter/blob/6193615f90a800ff5623373137bd88ea67d5a1d2/packages/flutter_tools/test/integration.shard/android_plugin_skip_unsupported_test.dart#L105-L109

3. https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20tool_integration_tests_3_4/16822/overview - Could not GET 'https://repo.maven.apache.org/<omitted>.pom'. Received status code 503 from server: Backend unavailable, connection timeout, caused by https://github.com/flutter/flutter/blob/6193615f90a800ff5623373137bd88ea67d5a1d2/packages/flutter_tools/test/integration.shard/android_gradle_outputs_app_link_settings_test.dart#L318-L326