unbundling dependencies (distro packaging)

[selfisekai]

port analogical solutions from chromium:

• tool for replacing normal gn build files with unbundle ones, for use in such environments
• shim headers

I'm working on the same issue in flutter engine as well (flutter/flutter#121106, flutter/buildroot#687) - should at least some of the replacement build files be in a repository shared with flutter? (some are specific to flutter engine, some are shared with dart)

dependency-independent:

• [x] #51612
• [x] #51614

dependency list:

• [ ] icu (~~https://chromium-review.googlesource.com/c/chromium/deps/icu/+/4307370~~ https://dart-review.googlesource.com/c/sdk/+/303241)
• [ ] protobuf
• [x] ~~tcmalloc~~ (no longer valid - https://github.com/dart-lang/sdk/commit/7b2cfdbc8cd74cda8096acdcee4a8e112cf0c3b8)
• [ ] zlib (~~https://chromium-review.googlesource.com/c/chromium/src/+/4414588~~ https://dart-review.googlesource.com/c/sdk/+/303241)

[sortie]

Hi, I build the release engineering infrastructure for Dart and I'd love for Dart to better fit distros, I'd be interested in this work and potentially helping out. I'll look a bit more into your links. What sort of specific changes did you have in mind?

[selfisekai]

using shared C libraries provided by the system, for C dependencies that Dart uses, e.g. zlib, icu, tcmalloc. currently, gclient downloads source code for all of them together with the source code for Dart, and they get compiled in to the binaries that use them. the standard within distributions is that these dependencies are separately packaged in the distribution's repositories, and are made available to other packages that utilize them, e.g. by querying pkgconf/pkg-config.

chromium achieves this with a script that replaces the gn files in the tree (e.g. //third_party/icu/BUILD.gn is getting replaced with https://chromium.googlesource.com/chromium/src/build/+/refs/heads/main/linux/unbundle/icu.gn). distributions run that script in the build process (before gn/ninja) with a list of dependencies to be replaced.

[sortie]

This sounds doable yeah. We have a high level tools/build.py entry point that invokes gn so I'm sure it could have a new option that makes it unbundle the dependencies, perhaps by passing a parameter down through the gn logic. I think that's a cleaner solution than invoking a script to rewrite the tree :)

[sortie]

Do you want to give this work an initial try? I'm happy to do the code reviews to make this happen and help discuss how it can be done :)

Something like this is also on our roadmap to improve the build-from-source journey although it's a little down the road

[selfisekai]

what about flutter engine in this case? also the build workflows for dependencies are coming from other projects, as I see: zlib, icu ~~and tcmalloc~~ (edit: tcmalloc gn files are in the sdk repo) from chromium, protobuf from fuchsia

[sortie]

I'm not sure about Flutter. Their infrastructure is a bit ever-changing and we constantly roll Dart into it via some complicated multi-headed solutions. If we at least solve the problem for Dart, we can see what it would take to do the equivalent work for Flutter. We're happy to try to collaborate with our friends in Chromium and Fuchsia if we need to amend or fork shared code :)

[selfisekai]

@sortie submitted #51612 and #51614 so far.

I've got changes for icu done as well, but:

1. use_system_icu=true without removing some files from //third_party/icu/ ends up with linker errors. chromium has a tool for removing those - should I do the same here?
2. I've been working around the need for depot_tools by teapot_tools, but there's no teapot git-cl, so I must figure this out one way or another

[selfisekai]

@sortie submitted icu changes: https://chromium-review.googlesource.com/c/chromium/deps/icu/+/4307370

[selfisekai]

@sortie I've tried to submit zlib changes, but I'm failing with this:

remote: PERMISSION_DENIED: The caller does not have permission
remote: [type.googleapis.com/google.rpc.RequestInfo]
remote: request_id: "3a3e232721f04ca7a161655235204c1d"
remote: 
remote: [type.googleapis.com/google.rpc.RequestInfo]
remote: request_id: "3b195fb9bc984285a14ef5ae86b4ebe2"
fatal: unable to access 'https://chromium.googlesource.com/chromium/src/third_party/zlib.git/': The requested URL returned error: 403

[adenilsoncavalcanti]

You may want to rethink the idea of dropping Chromium zlib.

It is a highly optimized fork of zlib using SIMD instructions on x86 and Arm.

For further info, check: https://docs.google.com/presentation/d/1_XWLhF4FcTz_HGc4QqvVapkxMj2-QTOZNdyXvpy2oYU/edit?usp=sharing https://docs.google.com/presentation/d/1-sNwEtudmblPKuoIR-Skls3j7NMO24G1BG9Si_1JWrI/edit?usp=sharing

If anything, I'm seriously considering removing support for 'use_system_zlib', as there are no good reasons (performance or security) to use something else.

To quote the discussion (https://chromium-review.googlesource.com/c/chromium/src/+/947464): "I wouldn't recommend running Chromium against a zlib that is not third_party/zlib.".

[nekopsykose]

dropping

i am not sure where it is suggested to "drop" it, this is a proposal for a toggle, not removal.

[adenilsoncavalcanti]

I can't comment on the other dependencies (icu, protobuf, tcmalloc) but what is the gain here for the zlib case?

[adenilsoncavalcanti]

Even if you don't care about performance, from a security perspective would be a bad idea: https://docs.google.com/presentation/d/1GNCnusHEVFY3vuqVMUh1b6fqo212T0HYkYSVe3f3CPw/edit?usp=sharing&resourcekey=0-8_xbWFtMBBLM60NoZwUtnQ

[nekopsykose]

the same as for icu, protobuf, tcmalloc, that it dynamically links a dependency from the system, which allows upgrades (and potential security fixes, etc) independently without requiring any rebuilds for reverse dependencies.

that foundation is why the majority of linux distributions favor dynamic over static linking (though of course there's experiments into various methods wrt all this, etc).

now you could say something like 'well, zlib updates with chromium, which itself updates fairly often, etc', but that still requires a rebuild of the entirety of chromium (on my machine this is about 4 hours without cache) if potentially someone immediately had to release a security fix for the zlib in question.

that's also assuming that 'system zlib' here is actually 'canonical zlib'. i think in the coming years we will see something like zlib-ng take priority for /usr/lib/libz.so.1 over the canonical. but you can't actually know what is there! it is just a preference for dynamic linking, not necessarily "remove the google code".

that said, if you have any zlib 0-days to make my day interesting, drop them right here :)

(this also reminds me to look into the feasibility of making zlib-ng the default in the distro i maintain..)

[adenilsoncavalcanti]

It seems the issue you are trying to solve is related to compilation time, perhaps Goma could help here: https://chromium.googlesource.com/infra/goma/client/+/HEAD/doc/early-access-guide.md

[selfisekai]

it's also a logistics issue. a list of questions comes in with a security upgrade: what packages do have zlib compiled in statically? how to upgrade the version in a codebase, if the source code of a specific version is in the tarball together with other source code? do we have to wait for each project to make a release with upgraded version?

[nekopsykose]

i have no idea how a chromium/google -internal tool open only to chromium contributors and supporting only *checks webpage* ubuntu 16.04, helps anyone build chromium (nor is that even the core issue), but maybe working for google makes you think everyone else has build farms of thousands of machines too?

i'll strike a deal: i'll use chromium zlib if chromium supports JpegXL.

[adenilsoncavalcanti]

i'll strike a deal: i'll use chromium zlib if chromium supports JpegXL.

I'm unsure what is your intention here, but it doesn't seem productive or helpful.

[tiziodcaio]

We should try unbundle binaryen too!