Add support for overlays in source.json

[fzakaria]

This is a continuation of #22155 that adds the newly added 'remote_files' attribute for http_archive to the bzlmod functionality.

The end goal is to then update BCR to this new functionality to overlay files rather than use patch files when providing MODULE/WORKSPACE/BUILD files.

https://github.com/bazelbuild/bazel-central-registry/issues/1566 has a good discussion of the rationale.

Co-authored-by: Fabian Meumertzheim [email protected]

[fmeum]

The failing test (testShowModuleAndExtensionReposFromBaseModule) is pretty sensitive to any kind of changes to the spec builders, so you might have to adjust the number of skipped lines. Let me know if you have problems running the test to capture the full output.

[fzakaria]

The failing test (testShowModuleAndExtensionReposFromBaseModule) is pretty sensitive to any kind of changes to the spec builders, so you might have to adjust the number of skipped lines. Let me know if you have problems running the test to capture the full output.

I printed the output but I'm not sure exactly which fields the test wants to validate. Here is the stdout pretty printed:

['## @bar_from_foo2:',
 '# <builtin>',
 'http_archive(',
 '  name = "bar~",',
 '  urls = '
 '["file:/tmp/bazel-working-directory/_main/_tmp/95ebb5496eb54e209206860635685b44/tests_root/tmp9dqgcyja/tmplpgvxv_9/main/archives/bar.2.0.zip"],',
 '  integrity = "sha256-j3wLXYtBrK/WDy97QCU3oI1HB2+0AYPHyjO6mBlbAVw=",',
 '  strip_prefix = "",',
 '  remote_file_urls = {},',
 '  remote_file_integrity = {},',
 '  remote_patches = {},',
 '  remote_patch_strip = 0,',
 ')',
 '# Rule bar~ instantiated at (most recent call last):',
 '#   <builtin> in <toplevel>',
 '# Rule http_archive defined at (most recent call last):',
 '#   '
 '/tmp/bazel-working-directory/_main/_tmp/95ebb5496eb54e209206860635685b44/_bazel_fmzakari/627ca31f67019a002c2b9671f4bd938d/external/bazel_tools/tools/build_defs/repo/http.bzl:397:31 '
 'in <toplevel>',
 '',
 '## [email protected]:',
 '# <builtin>',
 'local_repository(',
 '  name = "ext~",',
 '  path = '
 '"/tmp/bazel-working-directory/_main/_tmp/95ebb5496eb54e209206860635685b44/tests_root/tmp9dqgcyja/tmplpgvxv_9/main/projects/ext",',
 ')',
 '# Rule ext~ instantiated at (most recent call last):',
 '#   <builtin> in <toplevel>',
 '',
 '## @my_repo3:',
 '# <builtin>',
 'data_repo(',
 '  name = "ext~~ext~repo3",',
 '  data = "requested repo",',
 ')',
 '# Rule ext~~ext~repo3 instantiated at (most recent call last):',
 '#   <builtin> in <toplevel>',
 '# Rule data_repo defined at (most recent call last):',
 '#   '
 '/tmp/bazel-working-directory/_main/_tmp/95ebb5496eb54e209206860635685b44/_bazel_fmzakari/627ca31f67019a002c2b9671f4bd938d/external/ext~/ext.bzl:2:28 '
 'in <toplevel>',
 '',
 '## @my_repo4:',
 '# <builtin>',
 'data_repo(',
 '  name = "ext~~ext~repo4",',
 '  data = "requested repo",',
 ')',
 '# Rule ext~~ext~repo4 instantiated at (most recent call last):',
 '#   <builtin> in <toplevel>',
 '# Rule data_repo defined at (most recent call last):',
 '#   '
 '/tmp/bazel-working-directory/_main/_tmp/95ebb5496eb54e209206860635685b44/_bazel_fmzakari/627ca31f67019a002c2b9671f4bd938d/external/ext~/ext.bzl:2:28 '
 'in <toplevel>',
 '',
 '## [email protected]:',
 '# <builtin>',
 'http_archive(',
 '  name = "bar~",',
 '  urls = '
 '["file:/tmp/bazel-working-directory/_main/_tmp/95ebb5496eb54e209206860635685b44/tests_root/tmp9dqgcyja/tmplpgvxv_9/main/archives/bar.2.0.zip"],',
 '  integrity = "sha256-j3wLXYtBrK/WDy97QCU3oI1HB2+0AYPHyjO6mBlbAVw=",',
 '  strip_prefix = "",',
 '  remote_file_urls = {},',
 '  remote_file_integrity = {},',
 '  remote_patches = {},',
 '  remote_patch_strip = 0,',
 ')',
 '# Rule bar~ instantiated at (most recent call last):',
 '#   <builtin> in <toplevel>',
 '# Rule http_archive defined at (most recent call last):',
 '#   '
 '/tmp/bazel-working-directory/_main/_tmp/95ebb5496eb54e209206860635685b44/_bazel_fmzakari/627ca31f67019a002c2b9671f4bd938d/external/bazel_tools/tools/build_defs/repo/http.bzl:397:31 '
 'in <toplevel>',
 '']

[fzakaria]

Okay it should be passing now. That was a miserable test to update.

Target //src/test/py/bazel:mod_command_test up-to-date:
  bazel-bin/src/test/py/bazel/mod_command_test
INFO: Elapsed time: 11.439s, Critical Path: 11.14s
INFO: 2 processes: 1 internal, 1 linux-sandbox.
INFO: Build completed successfully, 2 total actions
//src/test/py/bazel:mod_command_test  

[fzakaria]

I have a few stylistic suggestions as well as a GSON update for record support at fzakaria#1, please take a look.

Otherwise LGTM.

Done -- thank you for the contribution to simplify the process.

[fzakaria]

@Wyverald I believe I've made the changes requested to simplify the archive JSON structure.

[fzakaria]

Ah -- I closed it by accident :(

I also updated the code as required @Wyverald

[fmeum]

@bazel-io fork 7.2.0

[meteorcloudy]

Sorry for missing this in the original review, but we should add documentation for this feature at https://bazel.build/external/registry

[fzakaria]

@meteorcloudy we never landed the registry support https://github.com/bazelbuild/bazel-central-registry/pull/2046

I think last place we landed was how to enable it in the registry and assume version running. Do we want to make modules that are the "new overlay format" only now?

Happy to finish this off given some direction & agreement.

[meteorcloudy]

@fzakaria The bcr_validation.py was fixed in https://github.com/bazelbuild/bazel-central-registry/pull/2249 and the BCR presubmit now supports overlay files after https://github.com/bazelbuild/continuous-integration/pull/2004.

Even without the above changes, since the change in the PR works with any Bazel registry (not just BCR), it should still be documented. ;)