Using the new (non-legacy) `build` specification results in a significant performance hit

[Jackenmen]

Details

• Read the Docs project URL: https://readthedocs.org/projects/red-discordbot/
• Build URL (if applicable):
  • New syntax with ubuntu-22.04: https://readthedocs.org/projects/red-discordbot/builds/17157068/
  • New syntax with ubuntu-20.04: https://readthedocs.org/projects/red-discordbot/builds/17157068/
  • Old syntax with latest image: https://readthedocs.org/projects/red-discordbot/builds/17157107/
• Read the Docs username (if applicable): https://readthedocs.org/profiles/jack1142/

Here's the configuration diff between the new ubuntu-22.04 build and the legacy latest build:

diff --git a/.readthedocs.yml b/.readthedocs.yml
index d180ab09a80..e73461d335d 100644
--- a/.readthedocs.yml
+++ b/.readthedocs.yml
@@ -1,10 +1,11 @@
 version: 2
 
 build:
-  image: latest
+  os: "ubuntu-22.04"
+  tools:
+    python: "3.8"
 
 python:
-  version: 3.8
   install:
     - method: pip
       path: .

ubuntu-20.04 configuration was the exact same, just with a different build.os value.

Expected Result

I expected there to be no significant performance penalty in changing to the new build specification syntax.

Actual Result

Changing to a new build specification resulted in over a 20-second slowdown.

[humitos]

Hi @jack1142! Thanks for opening this issue.

There are different things happening here that could explain the increase of 20 seconds on your build:

1. there is a difference of 9s when upgrading pip and setuptools
2. there is 10s difference when installing Read the Docs common dependencies
3. extra 5s when installing project's dependencies
4. 1 extra second when running Sphinx

That explains this increase in building time to me. The first 3 points are probably network related, where we can't do too much from our side.

Originally, when I read this issue I thought it could be done because on the new build specification we install the Python version on the fly by using a cached .tar.gz. That could have been introducing some extra time. However, that does not seem to be the issue.

I think we could reduce 1) by upgrading the pip and setuptools versions installed on the Python versions by default](https://github.com/readthedocs/readthedocs.org/blob/3e70b81902e499589494a59f3ac4057b702fa99c/scripts/compile_version_upload_s3.sh#L96-L97) to avoid re-installing them --but when a new pip version was released, we will have the same situation.

That said, I'm not seeing too much room for other improvements here.

[Jackenmen]

I made this issue because this difference seemed to be consistent, which is most surprising for installing RTD common dependencies since seemingly they should be doing the same work. I was kinda hoping that maybe there's going to be some obvious answer like there being different servers handling running one and the other types of builds or something like that which would explain one being slower than the other but I guess it rarely is that easy :)

About the difference in time in the pip/setuptools upgrade step, it would certainly be nice to cut off those seconds on something that always has to be installed and isn't changed that often (pip has a release once per quarter) but I don't know how much maintenance such regular image rebuild requires. It seems that this was already being done for the old latest image since pip upgrade was skipped there?

The differences in time for other two steps aren't that significant so not much to worry about there.

Anyway, I made this issue in hope that there's something that can be done about this, it not then I'll just stay with old setup until I can use build.commands and cut off the time in a different way.

[Jackenmen]

The difference seems to not be as big anymore and mostly comes down to a few seconds difference in pip/setuptools step (counting virtualenv creation + pip/setuptools upgrade). Other than that, there are random few-second differences in Sphinx step in favor of either build specification depending on the run so there's nothing to see there.

I decided to switch to the new build specification. Eventually, I'll get these seconds back by skipping unnecessary double install of Sphinx and some other dependencies by using build.commands or some other solution that becomes available but the difference is not all too significant so I no longer see the need to use the legacy build specification.

[humitos]

Thanks for your commenting back here with your findings and your decisions, @Jackenmen. I will close the issue since there isn't too much we can do about this right now. However, feel free to re-open if you consider.