code-review icon indicating copy to clipboard operation
code-review copied to clipboard

Add Taskcluster cache for local Mercurial clones

Open marco-c opened this issue 1 year ago • 5 comments

In order to actually use the work done in https://github.com/mozilla/code-review/pull/1517, we need to enable local cloning. In order to enable local cloning, we need to make sure we don't clone from scratch in each task, but we rely on a Taskcluster cache.

marco-c avatar May 29 '24 10:05 marco-c

I'm looking into this and have a few questions:

  • there already exists firefox source code checkouts in Taskcluster cache, but they are restricted to tasks ran by run-task: would you consider moving the code-review bot execution to run-task ?
  • if using custom caches for the code review bot, we would need one cache per repository.
    • when the cache is empty, some code-review bot tasks would need to populate it (initial clone), and will take longer: is that ok ?

La0 avatar Jun 25 '24 06:06 La0

I noticed that we already have cache enabled for the Integration hook.

Unfortunately the hook on Community TC does not successfully run anymore due to a missing secret.

We could restore that functionality, and test the clone & re-use from cache functionalities from that hook before updating runtime tasks

WDYT @marco-c ?

La0 avatar Jun 27 '24 08:06 La0

I spent a bit of time today on this topic and noticed a few things:

  • the integration test is broken because of invalid Taskcluster proxy url and automated conversion from docker-worker to generic-worker. As I do not know what's the long term plan regarding this hook & its viability, I stopped looking into it.
  • I made a first PR to control directly the mercurial cache behaviour from the CLI args, removing Taskcluster "production" check, and simplifying the overall code. In the end, the taskcluster cache is usable because I removed the tempdir path creation. As robust_checkout is already used, the cache will be automatically purged & setup at the correct version on each run
  • The mercurial version tools extension is not at the latest version (but just lack 1 whitespace changeset)
  • The mercurial version is outdated (6.4.5 in code vs 6.7.4 on pypi)
  • I have issues running locally related to sentry-sdk (probably an invalid version locally that has changed some modules)

Next steps:

  • get a review from @Archaeopteryx or @marco-c
  • enable cache on https://hg.mozilla.org/ci/ci-configuration/file/tip/hooks/project-relman/code-review-testing.yml

La0 avatar Jul 01 '24 15:07 La0

I'm looking into this and have a few questions:

* there already exists [firefox source code checkouts](https://firefox-source-docs.mozilla.org/taskcluster/caches.html#version-control-caches) in Taskcluster cache, but they are restricted to tasks ran by run-task: would you consider moving the code-review bot execution to run-task ?

Yes, but I'm not sure what would be needed to do that. Long term, we might even consider moving the review bot into mozilla-central itself, which would reduce the maintenance a lot.

* if using custom caches for the code review bot, we would need one cache per repository.
  
  * when the cache is empty, some code-review bot tasks would need to populate it (initial clone), and will take longer: is that ok ?

Yeah I guess there's not way around that.

marco-c avatar Jul 08 '24 13:07 marco-c

Regarding the integration hook, I think it has been broken for a while. It would be useful to have it working again, though it's not a huge priority.

marco-c avatar Jul 08 '24 13:07 marco-c