Decision task failure with latest `decision-mobile` image

[ahal]

I made a PR to update Taskgraph and the Decision task image, but when updating to the latest image, the Decision task fails. Here's the relevant log:

[task 2022-12-01T20:32:00.897Z] Exception in thread "main" java.lang.RuntimeException: Could not create parent directory for lock file /builds/worker/.gradle/wrapper/dists/gradle-7.5.1-bin/7jzzequgds1hbszbhq3npc5ng/gradle-7.5.1-bin.zip.lck
[task 2022-12-01T20:32:00.898Z] 	at org.gradle.wrapper.ExclusiveFileAccessManager.access(ExclusiveFileAccessManager.java:43)
[task 2022-12-01T20:32:00.899Z] 	at org.gradle.wrapper.Install.createDist(Install.java:48)
[task 2022-12-01T20:32:00.900Z] 	at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:107)
[task 2022-12-01T20:32:00.900Z] 	at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:63)
[task 2022-12-01T20:32:00.917Z] 2022-12-01 20:32:00,916 - ERROR - Error loading tasks for kind build:
[task 2022-12-01T20:32:00.917Z] Traceback (most recent call last):
[task 2022-12-01T20:32:00.918Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/generator.py", line 309, in _run
[task 2022-12-01T20:32:00.919Z]     new_tasks = kind.load_tasks(
[task 2022-12-01T20:32:00.919Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/generator.py", line 75, in load_tasks
[task 2022-12-01T20:32:00.920Z]     tasks = [
[task 2022-12-01T20:32:00.920Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/generator.py", line 75, in <listcomp>
[task 2022-12-01T20:32:00.920Z]     tasks = [
[task 2022-12-01T20:32:00.921Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/task.py", line 1252, in check_run_task_caches
[task 2022-12-01T20:32:00.921Z]     for task in tasks:
[task 2022-12-01T20:32:00.922Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/task.py", line 1187, in check_task_dependencies
[task 2022-12-01T20:32:00.922Z]     for task in tasks:
[task 2022-12-01T20:32:00.922Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/task.py", line 1173, in check_task_identifiers
[task 2022-12-01T20:32:00.923Z]     for task in tasks:
[task 2022-12-01T20:32:00.923Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/task.py", line 1154, in chain_of_trust
[task 2022-12-01T20:32:00.923Z]     for task in tasks:
[task 2022-12-01T20:32:00.923Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/task.py", line 1147, in add_github_checks
[task 2022-12-01T20:32:00.923Z]     for task in tasks:
[task 2022-12-01T20:32:00.924Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/task.py", line 951, in build_task
[task 2022-12-01T20:32:00.924Z]     for task in tasks:
[task 2022-12-01T20:32:00.924Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/task.py", line 919, in add_index_routes
[task 2022-12-01T20:32:00.924Z]     for task in tasks:
[task 2022-12-01T20:32:00.924Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/task.py", line 881, in validate
[task 2022-12-01T20:32:00.924Z]     for task in tasks:
[task 2022-12-01T20:32:00.924Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/task.py", line 869, in task_name_from_label
[task 2022-12-01T20:32:00.925Z]     for task in tasks:
[task 2022-12-01T20:32:00.925Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/task.py", line 824, in set_defaults
[task 2022-12-01T20:32:00.925Z]     for task in tasks:
[task 2022-12-01T20:32:00.925Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/task.py", line 803, in set_implementation
[task 2022-12-01T20:32:00.925Z]     for task in tasks:
[task 2022-12-01T20:32:00.926Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/job/__init__.py", line 347, in make_task_description
[task 2022-12-01T20:32:00.926Z]     for job in jobs:
[task 2022-12-01T20:32:00.926Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/job/__init__.py", line 229, in use_fetches
[task 2022-12-01T20:32:00.926Z]     for job in order_tasks(config, jobs):
[task 2022-12-01T20:32:00.926Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/cached_tasks.py", line 22, in order_tasks
[task 2022-12-01T20:32:00.926Z]     pending = deque(tasks)
[task 2022-12-01T20:32:00.926Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/job/__init__.py", line 157, in add_resource_monitor
[task 2022-12-01T20:32:00.926Z]     for job in jobs:
[task 2022-12-01T20:32:00.926Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/job/__init__.py", line 145, in set_label
[task 2022-12-01T20:32:00.926Z]     for job in jobs:
[task 2022-12-01T20:32:00.926Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/job/__init__.py", line 129, in set_implementation
[task 2022-12-01T20:32:00.926Z]     for job in jobs:
[task 2022-12-01T20:32:00.926Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/job/__init__.py", line 109, in rewrite_when_to_optimization
[task 2022-12-01T20:32:00.926Z]     for job in jobs:
[task 2022-12-01T20:32:00.927Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/base.py", line 143, in __call__
[task 2022-12-01T20:32:00.927Z]     for task in tasks:
[task 2022-12-01T20:32:00.927Z]   File "/builds/worker/checkouts/vcs/taskcluster/rb_taskgraph/transforms/build_apk.py", line 22, in add_artifacts
[task 2022-12-01T20:32:00.927Z]     variant_config = get_build_variant(build_type)
[task 2022-12-01T20:32:00.927Z]   File "/builds/worker/checkouts/vcs/taskcluster/rb_taskgraph/gradle.py", line 13, in get_build_variant
[task 2022-12-01T20:32:00.927Z]     all_variants = _fetch_all_variants()
[task 2022-12-01T20:32:00.927Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/util/memoize.py", line 25, in __call__
[task 2022-12-01T20:32:00.927Z]     self[args] = self.func(*args)
[task 2022-12-01T20:32:00.927Z]   File "/builds/worker/checkouts/vcs/taskcluster/rb_taskgraph/gradle.py", line 31, in _fetch_all_variants
[task 2022-12-01T20:32:00.927Z]     output = _run_gradle_process('printVariants')
[task 2022-12-01T20:32:00.928Z]   File "/builds/worker/checkouts/vcs/taskcluster/rb_taskgraph/gradle.py", line 47, in _run_gradle_process
[task 2022-12-01T20:32:00.928Z]     raise RuntimeError(f"Gradle command returned error: {exit_code}")
[task 2022-12-01T20:32:00.929Z] RuntimeError: Gradle command returned error: 1
[task 2022-12-01T20:32:00.929Z] Traceback (most recent call last):
[task 2022-12-01T20:32:00.929Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/main.py", line 753, in main
[task 2022-12-01T20:32:00.929Z]     args.command(vars(args))
[task 2022-12-01T20:32:00.929Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/main.py", line 626, in decision
[task 2022-12-01T20:32:00.929Z]     taskgraph_decision(options)
[task 2022-12-01T20:32:00.929Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/decision.py", line 102, in taskgraph_decision
[task 2022-12-01T20:32:00.929Z]     full_task_json = tgg.full_task_graph.to_json()
[task 2022-12-01T20:32:00.929Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/generator.py", line 169, in full_task_graph
[task 2022-12-01T20:32:00.929Z]     return self._run_until("full_task_graph")
[task 2022-12-01T20:32:00.929Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/generator.py", line 423, in _run_until
[task 2022-12-01T20:32:00.929Z]     k, v = next(self._run)
[task 2022-12-01T20:32:00.929Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/generator.py", line 309, in _run
[task 2022-12-01T20:32:00.929Z]     new_tasks = kind.load_tasks(
[task 2022-12-01T20:32:00.929Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/generator.py", line 75, in load_tasks
[task 2022-12-01T20:32:00.929Z]     tasks = [
[task 2022-12-01T20:32:00.929Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/generator.py", line 75, in <listcomp>
[task 2022-12-01T20:32:00.929Z]     tasks = [
[task 2022-12-01T20:32:00.929Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/task.py", line 1252, in check_run_task_caches
[task 2022-12-01T20:32:00.929Z]     for task in tasks:
[task 2022-12-01T20:32:00.929Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/task.py", line 1187, in check_task_dependencies
[task 2022-12-01T20:32:00.929Z]     for task in tasks:
[task 2022-12-01T20:32:00.929Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/task.py", line 1173, in check_task_identifiers
[task 2022-12-01T20:32:00.929Z]     for task in tasks:
[task 2022-12-01T20:32:00.929Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/task.py", line 1154, in chain_of_trust
[task 2022-12-01T20:32:00.929Z]     for task in tasks:
[task 2022-12-01T20:32:00.929Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/task.py", line 1147, in add_github_checks
[task 2022-12-01T20:32:00.930Z]     for task in tasks:
[task 2022-12-01T20:32:00.930Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/task.py", line 951, in build_task
[task 2022-12-01T20:32:00.930Z]     for task in tasks:
[task 2022-12-01T20:32:00.930Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/task.py", line 919, in add_index_routes
[task 2022-12-01T20:32:00.930Z]     for task in tasks:
[task 2022-12-01T20:32:00.930Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/task.py", line 881, in validate
[task 2022-12-01T20:32:00.930Z]     for task in tasks:
[task 2022-12-01T20:32:00.930Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/task.py", line 869, in task_name_from_label
[task 2022-12-01T20:32:00.930Z]     for task in tasks:
[task 2022-12-01T20:32:00.930Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/task.py", line 824, in set_defaults
[task 2022-12-01T20:32:00.930Z]     for task in tasks:
[task 2022-12-01T20:32:00.930Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/task.py", line 803, in set_implementation
[task 2022-12-01T20:32:00.930Z]     for task in tasks:
[task 2022-12-01T20:32:00.930Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/job/__init__.py", line 347, in make_task_description
[task 2022-12-01T20:32:00.930Z]     for job in jobs:
[task 2022-12-01T20:32:00.930Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/job/__init__.py", line 229, in use_fetches
[task 2022-12-01T20:32:00.930Z]     for job in order_tasks(config, jobs):
[task 2022-12-01T20:32:00.930Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/cached_tasks.py", line 22, in order_tasks
[task 2022-12-01T20:32:00.930Z]     pending = deque(tasks)
[task 2022-12-01T20:32:00.930Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/job/__init__.py", line 157, in add_resource_monitor
[task 2022-12-01T20:32:00.930Z]     for job in jobs:
[task 2022-12-01T20:32:00.930Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/job/__init__.py", line 145, in set_label
[task 2022-12-01T20:32:00.930Z]     for job in jobs:
[task 2022-12-01T20:32:00.930Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/job/__init__.py", line 129, in set_implementation
[task 2022-12-01T20:32:00.930Z]     for job in jobs:
[task 2022-12-01T20:32:00.930Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/job/__init__.py", line 109, in rewrite_when_to_optimization
[task 2022-12-01T20:32:00.930Z]     for job in jobs:
[task 2022-12-01T20:32:00.930Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/transforms/base.py", line 143, in __call__
[task 2022-12-01T20:32:00.930Z]     for task in tasks:
[task 2022-12-01T20:32:00.930Z]   File "/builds/worker/checkouts/vcs/taskcluster/rb_taskgraph/transforms/build_apk.py", line 22, in add_artifacts
[task 2022-12-01T20:32:00.930Z]     variant_config = get_build_variant(build_type)
[task 2022-12-01T20:32:00.930Z]   File "/builds/worker/checkouts/vcs/taskcluster/rb_taskgraph/gradle.py", line 13, in get_build_variant
[task 2022-12-01T20:32:00.930Z]     all_variants = _fetch_all_variants()
[task 2022-12-01T20:32:00.930Z]   File "/builds/worker/.local/lib/python3.8/site-packages/taskgraph/util/memoize.py", line 25, in __call__
[task 2022-12-01T20:32:00.930Z]     self[args] = self.func(*args)
[task 2022-12-01T20:32:00.930Z]   File "/builds/worker/checkouts/vcs/taskcluster/rb_taskgraph/gradle.py", line 31, in _fetch_all_variants
[task 2022-12-01T20:32:00.930Z]     output = _run_gradle_process('printVariants')
[task 2022-12-01T20:32:00.930Z]   File "/builds/worker/checkouts/vcs/taskcluster/rb_taskgraph/gradle.py", line 47, in _run_gradle_process
[task 2022-12-01T20:32:00.930Z]     raise RuntimeError(f"Gradle command returned error: {exit_code}")
[task 2022-12-01T20:32:00.930Z] RuntimeError: Gradle command returned error: 1

I confirmed that using the prior image works.

/cc @JohanLorenzo

[JohanLorenzo]

Hey! I'm not sure what happened 🤔 At first, I thought ./gradlew printVariants doesn't exist in reference-browser but it works just fine.

I noticed your PR https://github.com/mozilla-mobile/reference-browser/pull/2126 got fixed without downgrading the docker image. So I don't manage to reproduce the bug. Can you still repro?

[ahal]

The image in the PR is the one before the most recent (from 3 months ago). The error only seems to happen with the most recent image (~8 days ago). I was just using the PR to run some tests, and force pushed the failing image to use the older (but still newer than before) one.

[ahal]

In otherwords, it's not blocking me or anything, but I guess we'll have to figure this out eventually.

[JohanLorenzo]

Thanks for this extra context. I just realized I totally missed this first line in the logs:

Could not create parent directory for lock file /builds/worker/.gradle/wrapper/dists/gradle-7.5.1-bin/7jzzequgds1hbszbhq3npc5ng/gradle-7.5.1-bin.zip.lck

I know what this is, then! We should add these changes to .taskcluster.yml: https://github.com/mozilla-mobile/firefox-android/commit/de10e3faa7689d2092e0c1cb5c5e671d3be2791b

Sorry for the confusion this optimization caused!