Fast iteration support for huge multipart projects

[jpakkane]

This is a longer and more elaborate version of #13. Suppose that you'd like to develop a Flatpak app, which has the following requirements (fairly typical in game projects, for example):

• consists of many repositories, not one, and the repos are free floating, not submodules of a common base repo
• each one several gigabytes in size
• during development you'd want to change multiple repositories in one go
• the final installed image size is several gigabytes, most of which is resources such as graphics and sounds

This type of setup has some extra limitations and requirements:

• the Git repos may not be copied inside the build container because it would be too slow, instead they must be linked or mounted somehow
• there is no one Git repo, but several, and they must be built as one, not one after the other (usually because they don't install headers or provide .pc files and adding them would be too much work)
• changes not yet committed must be used for the build
• the install command must not copy all resources over every time because it would be too slow, only changed resources would be copied (CMake and Meson already support this)

This might require fairly big changes, but it would make this kind of development cycle a lot smoother.

[TingPing]

there is no one Git repo, but several, and they must be built as one, not one after the other (usually because they don't install headers or provide .pc files and adding them would be too much work)

If I understand what you mean:

- name: foo
  sources:
    - type: git
      url: ...
      dest: sub-part-1
    - type: git
      url: ...
      dest: sub-part-2

Git repos are also shallow clones by default which helps the size at least.

[jpakkane]

Basically what this means is that you have a directory tree like this:

/src_root
  some_dir
    proj1 (contains a .git and the checkout)
    proj2 (same as above)
    proj3 (same as above)
  other_dir
    (bunch of dirs just like above)

And what you'd like to say is "take the entirety of src_root and put it inside the build container". There are even cases where you'd need to run CMake on a subdir inside the tree and not in the root. This is not really a "git repo" as such, just taking an entire subtree of stuff and using it as is. Specifically any changes in the code that are not yet committed have to be used

Yes, all of this is a bit wonky and you might reasonably say "just fix your project layout" for a few of the requirements listed here. Sadly, though, many real world code bases are a bit crap in these ways and fixing them is not really feasible due to various reasons.

[matthiasclasen]

Have you looked at flatbox?

[alexlarsson]

So, i don't think flatpak-builder as it stands is really a tool for day-to-day development. It is made to do controlled, reproducible release builds. This is pretty tied into the model with caching of partial builds and whatnot.

However, I realize that it would be useful if you can use the manifest format as a way to descibe builds during a more free-form development cycle. In fact, in the flatbox project I would like to be able to point to some random set of jsons and build them all (or parts of them) in my pet build container for testing/development.

I haven't really fully researched this, but I think the best way forward is to have f-b as it is, but add additional tools (maybe even a library) so that you can use parts of f-b for other usecases.

[louib]

@matthiasclasen is alex's project the flatbox project you're referring to?

@alexlarsson

However, I realize that it would be useful if you can use the manifest format as a way to descibe builds during a more free-form development cycle. In fact, in the flatbox project I would like to be able to point to some random set of jsons and build them all (or parts of them) in my pet build container for testing/development.

I haven't really fully researched this, but I think the best way forward is to have f-b as it is, but add additional tools (maybe even a library) so that you can use parts of f-b for other usecases.

What I'd love to see is a flatpak-manifest executable that can perform some basic operations on a manifest, like linting, but also compute the build instructions from a manifest, so that one could reuse the flatpak manifest with other sandboxing technologies, or even without any sandboxing to install directly on the host system. Like you mentioned, the new executable could use a library shared between flatpak-builder and flatpak-manifest, from just the part of f-b that parses and validates the manifests and generates the build instructions (haven't looked at the code in detail though so can't tell how hard that would be).