`build.commands` shall support custom package installation (`build.apt_packages`)

[danwos]

Details

This is based on a comment from @humitos : https://github.com/readthedocs/readthedocs.org/issues/3885#issuecomment-1244001249

build.commands does not allow to install extra packages via build.apt_packages.

Even if build.commands gives me more flexibility in the build process, it takes also a lot of "features" away from me, because I'm not able load/install additional, non-python programs/libs anymore.

Expected Result

Take build.apt_packages into account when using build.commands. Or allow using sudo apt-get install XY.

Use case

I want to run PlantUML, which normally works on RTD. But I also need a custom build, as I must build 2 PDFs before building my HTML docs. Used Config: RTD-file of Sphinx-SimplePDF

Without PlantUML, the Sphinx-Needs extension can not be used to build any kind of diagram.

So in the end, for this use case, all I need is Java in machines when using build.commands :)

[humitos]

I think this is a good feature and I'd like to make it possible somehow. We are currently not allowing sudo or using root user because of security. We've discussed to migrate "the build process" completely outside the application itself and communicate each other via a well defined contract (see #9088); in that case, we can give people full-control of the Docker instance without problems. However, this is more a medium/long term goal, I guess.

Or allow using sudo apt-get install XY.

Is it secure to allow people to execute only one command via sudo by using /etc/sudoers with something similar to:

docs NOPASSWD: /usr/bin/apt-get install, /usr/bin/apt-get update

My concern here is people running things like sudo apt-get install `bash -c "malicious-command shutdown"` . In that case, the malicious-command will be executed as root as part of the apt-get command? My example may not be the best one, but I suppose that trying a little harder it may be possible to break the commands that are allowed to be executed by sudo. cc @agjohnson

[agjohnson]

Yeah, sudoers policies are hard to get right and are a common exploit target. There are similar exploits to yours that would allow root access.

build.commands does not allow to install extra packages via build.apt_packages

So, this is where I've always seen build.jobs being a more powerful implementation. We already support the installation of apt packages with our normal build process, so re-implementation of this feature via sudo doesn't seem very valuable.

The linked configuration file above is using build.commands, but seems it could just use build.jobs instead. The config is duplicating what our build process already does, but does a second build pass in addition. To use build.jobs instead, it could still build a secondary Sphinx project and would get all of the benefits of our existing build features (package installation, apt package installation, etc).

[humitos]

@agjohnson

The linked configuration file above is using build.commands, but seems it could just use build.jobs instead. The config is duplicating what our build process already does, but does a second build pass in addition. To use build.jobs instead, it could still build a secondary Sphinx project

This is a good point and I agree that it should work for this particular case 💯 . @danwos would you like to try this approach using build.jobs and reporting back here? 🙏🏼

That said, even if it works for this case, I still think it's a good feature to develop eventually. I don't think it's a priority tho, in particular because we don't have/know many cases where this situation will be a limitation that cannot be achieved by build.jobs. This feature is pretty useful for the use case: "non-standard build process + require installing system packages"

I've already faced that use cases when trying to build https://sphinx-themes.org/ on Read the Docs. It uses nox to build the documentation and requires installing playwright. This is the YAML file I used as a test: https://github.com/humitos/sphinx-themes.org/blob/9fe52d4ddcea0755a9863e93f1af0273a2feb8e4/.readthedocs.yaml -- cc'ing @pradyunsg since he is maintainer of that project and I assume he hit this problem as well and that's probably why this project is not on Read the Docs.

[agjohnson]

That said, even if it works for this case, I still think it's a good feature to develop eventually.

Yeah, this does feel like a disconnect, but I think we're probably in agreement on priority too.

I would probably still want to discuss a way to reference existing features from build.commands before reimplementing this with another build process pattern. I imagine there is a configuration file pattern that we might like around referencing build job steps from build.commands, like:

build:
  commands:
    - readthedocs: install-apt-packages
    - readthedocs: install-python-packages
    # Or    
    - /usr/bin/readthedocs-install-apt-packages
    # Etc

Or perhaps the long term plan of providing this customization using a buildpack like structure would be another conversation.

[humitos]

I would probably still want to discuss a way to reference existing features from build.commands before reimplementing this with another build process pattern

I'd prefer if we don't build anything inside the core application for this and we push forward the build contract instead (https://github.com/readthedocs/readthedocs.org/issues/9088). Once we have that implemented, we can execute the build process in a completely isolated instance outside our application and we can give people full control of it (even with root permissions). We only need to know where the artifacts generated by that process are and copy them back to our application instance before post-processing and uploading them to S3.

[henryiii]

FYI, at scientific-python dev summit, we've been trying to implement a Jekyll build in readthedocs, and apt commands are needed to install ruby (Ruby as a tool would be exactly what we want, actually!). Just wanted to show another use case that doesn't work with build.jobs.

[agjohnson]

@henryiii just to clarify, are you using build.commands or build.jobs configuration? We discussed a path forward to making build.jobs easier to use for arbitrary documentation generation, and this would make APT package installation possible at the same time.

[henryiii]

Our first attempt was with build.commands. I would highly recommend a note in the documentation that apt does not work with build.commands - that would have saved us a lot of debugging. Now we are trying build.jobs; I'm assuming, since we can't control the actual build step, that we'll have to put in a dummy sphinx or mkdocs config to make it pass the build step (though not sure what it does for other languages like nodejs).

And setting up the Python virtual env takes a little useless time. :)

[agjohnson]

Okay yeah, this is exactly the case we discussed then. You have pinpointed the current issue, and you will need to get creative with build.jobs to replace the build command step for building. With the ability to override this step too, build.jobs would be usable by projects looking to replace this step entirely.

[humitos]

We should probably add Ruby to "build.tools" options. There is no reason to not have it there

[humitos]

@henryiii BTW, you could solve your immediate issue by doing something like:

build:
  commands:
    # Install Ruby using "asdf"
    - asdf plugin add ruby https://github.com/asdf-vm/asdf-ruby.git
    - asdf install ruby 2.6.4
    - asdf global ruby 2.6.4
    # ... your other commands here

[henryiii]

This is what we ended up with:

https://github.com/scientific-python/cookie/blob/f84144ac83f080412b3516e2cdb9cd1d8806f19c/.readthedocs.yml

(Also 1-3 of the Zarr webpages are setup like this now too)

It takes about 7 mins, 5 of which are building Ruby, and 1.5 or so are building four of the plugins that Jekyll needs (Ruby doesn't have Python's redistributable "wheels", just an "SDists" equivalent). If we could avoid building Ruby, that would be a nice savings, and if we could cache the binaries, we could save most of the rest of the time. Ruby was already installed on the runners, we were just missing the headers package, otherwise we probably could have used the built-in Ruby (though I'd still have to come up with the right environment variable incantation to use it without sudo, but should be possible).

[humitos]

Glad the yaml snippet helped here. #10346 will improve the download/compiling time spent on Ruby. We cannot cache package/gem installation, tho, but at least the compiling time will be saved. We can continue the conversation on that issue, since we are deviating the topic a little from this original issue 👍

[agjohnson]

I'd like to prioritize adding support for build.jobs.build step override so that we can point users towards build.jobs when they want to use custom build pipelines. I still do think build.jobs is a best fit for advanced usage patterns, and build.commands is more for expert users. But without a build.jobs.build override, we'll have to force these users into expert usage right away.

I think it will be hard for us to feel good about implementing a sudo layer for build.commands usage, so there is a more obvious path in making build.jobs a nicer experience.

Do we want to treat this as a separate issue (perhaps there is one already, but I wasn't finding it)?

[humitos]

I think it will be hard for us to feel good about implementing a sudo layer for build.commands usage, so there is a more obvious path in making build.jobs a nicer experience. Do we want to treat this as a separate issue (perhaps there is one already, but I wasn't finding it)?

Yes. Overriding pre-defined jobs (e.g. build.jobs.build) is something different from supporting the installation of OS packages. I'd treat them differently 👍🏼 . Also, their requirements differ too much from each other.

[agjohnson]

Yeah, the two are different, technically speaking, but what I'm saying is that if we add support for overriding build.jobs.build, I think we would point most users towards customization through build.jobs instead of build.commands. This issue might then not be a strong concern if users have a way to both use most of our configuration file and replace the build job commands.

I'm also not suggesting to do anything like remove build.commands, but I think build.jobs would be a better implementation and much easier for us to communicate to users.

[humitos]

Gotcha! I think for a lot of cases build.jobs will be enough and I will be happy recommending that to most of our users. However, installing a OS package with build.command, and giving people sudo superpower will still be a good and valid feature to support eventually. I'm not worried about giving people sudo superpowers once we have our build system more isolated. However, I don't think we will be able to implement it before that.

[ssbarnea]

I ended up here while trying to enable git-lfs,... just to discover that it does not work if we use build commands...

[humitos]

@ssbarnea we have a git-lfs example in our documentation that does not require installing packages with APT: https://docs.readthedocs.io/en/latest/build-customization.html#support-git-lfs-large-file-storage. Can you try that and let me know if it works for you?

[ktbarrett]

Why is there this discussion and decision to wait for future infrastructure updates when simply supporting build.apt_packages and build.commands together would solve the original problem? Not supporting it doesn't seem "correct" anyways.

I'm running into the same problem and such a fix would do everything I need it to do.

[ktbarrett]

It seems that build.commands was introduced to allow user full customization of the build process and override the standard process. That's cool and all, but it was introduced the infrastructure to support it was available.

Instead what should have been introduced is a way to override the build step of the standard process. That's something that could be immediately supported and would offer a lot of flexibility for the user. I'm just trying to change what build command to run, which is the extent of what the example in the docs shows, and I think is sufficient for 95% of people who ever look into this feature.

[humitos]

@ktbarrett the thoughts/patterns behind this decision are:

• build.jobs allows users to quickly setup their project and assist them with the default commands that have to be executed for Sphinx and MkDocs projects [^1], creating an abstraction layer
• build.commands allows users to override the default build process completely, giving full control to the user. Read the Docs won't run any command on behalf of the user in this scenario

That said, we weren't able to add support for sudo on build.commands (and build.jobs) yet due to security reasons. On the other hand, we don't want to add support for build.apt_packages when using build.commands because it re-introduces "running command on behalf the users" in the build process we have created specifically to avoid that particular scenario.

Supporting sudo, which I consider is the correct solution here, is in our roadmap but we haven't been able to prioritize this work yet.

I hope that helps to clarify the situation.

[^1]: we plan to expand these doctools in the future.

[ktbarrett]

Yes, I understand what's intended here. What I'm saying is that a solution achievable now would be a separate feature that would allow the user to override the build stage in build.jobs (currently you can only override pre and post actions), so we can customize the build call without overriding the whole build flow. I have 0 interest in overriding the whole build flow and I'm gonna guess that 99+% of other users also have 0 interest in that. IMO It's overkill to make users redo the whole build flow just to override the build call.

[humitos]

What I'm saying is that a solution achievable now would be a separate feature that would allow the user to override the build stage in build.jobs (currently you can only override pre and post actions), so we can customize the build call without overriding the whole build flow.

Yes. We've discussing this internally and we already have a plan to move forward. However, we weren't able to prioritize this work yet.

I'm gonna guess that 99+% of other users also have 0 interest in that. IMO It's overkill to make users redo the whole build flow just to override the build call.

Right, if you want to build Sphinx or MkDocs projects you probably don't need to override it. However, if you want to build a project that uses Pelican, Docusaurus or any other documentation tool, you definitely will need build.commands to override the whole process.

[agjohnson]

I have felt that even Pelican/Docusaurus/etc should all be supported with build.jobs too. Would there be anything stopping the user from doing something like this if we supported it?

build:
  jobs:
    build:
      html:
        - pelican --settings docs/pelicanconf.py --output $READTHEDOCS_OUTPUT/html/ docs/

To me, the build.commands split feels closer to what @ktbarrett is describing -- something that only the expert, or 1-5%, users should use. The majority of our users should use build.jobs so that all of our features/fixes are usable by the project, and so most users don't have to deeply understand RTD to configure their project.

[humitos]

I think this issue is deviating a little from its original purpose at this point.

@agjohnson that workflow probably works for most of the Python doctools. However for any non-Python doctool it doesn't make too much sense to create a virtualenv, install pre defined dependencies, etc.

We will need to un-tie our current build process from Sphinx/MkDocs and Python more to support other doctools with build.jobs.

We've touched on "other pre-defined build packages" that's related to this, but that's for a different conversation that is outside the scope of adding sudo support for build.commads

[agjohnson]

Yeah, I'm not talking that deep on package reuse yet.

This conversation is centered around finishing the overrides to build.jobs.build because we get more value from less effort with that feature. I agree these two features are not necessarily in conflict, but we also haven't been able to prioritize finishing the build.job.build overrides. They accomplish the same goal of getting APT to users with custom build commands though.

There is work in build job overrides, but duplicating apt_packages is going to require the processing the minefield of sudo hardening, which is full of edge cases.

It's good to be mindful of what we're designing for, but I also feel like designing primarily for non-Python tools is a preoptimization. Python projects are the vast majority of projects and gain more from an intermediate configuration of build.jobs.build.html etc.