Issues after upgrading to buildbot 4.0

[vstinner]

Hi,

While trying to update requirements to fix a security issue, I upgrade buildbot from 3.x to 4.0: https://github.com/python/buildmaster-config/pull/497

The ./venv/bin/buildbot upgrade-master /data/buildbot/master command was run by Salt and the buildbot server is running fine.

... but, the "Release status" page is gone! I think that we have to follow https://docs.buildbot.net/4.0.0/manual/upgrading/4.0-upgrade.html#custom-plugins guide to upgrade our plugins.

[vstinner]

I tried to downgrade to buildbot 3.x. The database "migration" went badly :-(

Warning: Stopping this process might cause data loss
Got fatal Exception on DB
Traceback (most recent call last):
  File "/srv/buildbot/venv/lib/python3.9/site-packages/twisted/python/threadpool.py", line 285, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/srv/buildbot/venv/lib/python3.9/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/srv/buildbot/venv/lib/python3.9/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/srv/buildbot/venv/lib/python3.9/site-packages/buildbot/db/pool.py", line 235, in __thd
    log.err(e, 'Got fatal Exception on DB')
--- <exception caught here> ---
  File "/srv/buildbot/venv/lib/python3.9/site-packages/buildbot/db/pool.py", line 208, in __thd
    rv = callable(arg, *args, **kwargs)
  File "/srv/buildbot/venv/lib/python3.9/site-packages/buildbot/db/model.py", line 1136, in thd
    context.run_migrations()
  File "/srv/buildbot/venv/lib/python3.9/site-packages/alembic/runtime/migration.py", line 615, in run_migrations
    for step in self._migrations_fn(heads, self):
  File "/srv/buildbot/venv/lib/python3.9/site-packages/buildbot/db/model.py", line 1122, in upgrade
    return alembic_scripts._upgrade_revs(current_script_rev_head, rev)
  File "/srv/buildbot/venv/lib/python3.9/site-packages/alembic/script/base.py", line 455, in _upgrade_revs
    return [
  File "/usr/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/srv/buildbot/venv/lib/python3.9/site-packages/alembic/script/base.py", line 283, in _catch_revision_errors
    raise util.CommandError(resolution) from re
alembic.util.exc.CommandError: Can't locate revision identified by '066'

[vstinner]

I replaced buildbot-wsgi-dashboards dependency with buildbot-react-wsgi-dashboards: it was worse, buildbot failed to even start:

    (...)
    File "/srv/buildbot/venv/lib/python3.9/site-packages/buildbot/www/service.py", line 225, in reconfigServiceWithBuildbotConfig
      self.setupSite(new_config)
    File "/srv/buildbot/venv/lib/python3.9/site-packages/buildbot/www/service.py", line 325, in setupSite
      self.configPlugins(root, new_config)
    File "/srv/buildbot/venv/lib/python3.9/site-packages/buildbot/www/service.py", line 302, in configPlugins
      raise RuntimeError(f"could not find plugin {key}; is it installed?")
  builtins.RuntimeError: could not find plugin wsgi_dashboards; is it installed?

[vstinner]

Currently, the dependencies are:

buildbot@buildbot:/srv/buildbot$ venv/bin/python -m pip list|grep buildbot
buildbot                 4.0.0
buildbot-console-view    4.0.0
buildbot-grid-view       4.0.0
buildbot-waterfall-view  4.0.0
buildbot-worker          4.0.0
buildbot-wsgi-dashboards 4.0.0
buildbot-www             4.0.0

I also asked for help on #buildbot IRC channel.

[vstinner]

@pablogsal @ambv @encukou: Do you have any idea on how to debug such buildbot issue? Sorry for breaking the buildbot "Release Status" page by mistake :-(

[vstinner]

I reported the issue to buildbot bug tracker: https://github.com/buildbot/buildbot/issues/7775

[vstinner]

The "Log in with GitHub" URL is broken: /all is missing at the start of the path.

[vstinner]

https://buildbot.python.org/all/#/builders page loads https://buildbot.python.org/all/api/v2/builders which fails with HTTP 404 error, whereas it should load https://buildbot.python.org/all/api/v2/builders (all/ is missing in the API URL).

[vstinner]

cc @zware

[zware]

https://buildbot.python.org/all/#/builders page loads https://buildbot.python.org/all/api/v2/builders which fails with HTTP 404 error, whereas it should load https://buildbot.python.org/all/api/v2/builders (all/ is missing in the API URL).

I got this partially working with a hack to the Nginx config; I'll try to get it fully working and submit a PR to the Salt config.

[vstinner]

Maybe we should simply drop all/ from the URL and use a "regular" https://buildbot.python.org/ home URL?

[vstinner]

wsgi_dashboards in https://buildbot.python.org/all/?#/about says:

[{"name":"release_status","caption":"Release Status","app":"'flask.app.Flask' not yet IConfigured","order":2,"icon":"rocket"}]

Oh, there is a Flask error.

[vstinner]

https://buildbot.python.org/all/#/builders page loads https://buildbot.python.org/all/api/v2/builders which fails with HTTP 404 error, whereas it should load https://buildbot.python.org/all/api/v2/builders (all/ is missing in the API URL).

I reported a separated issue: https://github.com/buildbot/buildbot/issues/7776

[zware]

With python/psf-salt#372 merged and deployed, the web UI is at least mostly back (without the /all prefix). Release status still needs a fix, but that's beyond me for right now.

[vstinner]

It seems like buildbot 4.0 is most fully functional. The Release Status page is available, but it's not fully rendered.

"Releasable" versions only contain the following HTML in their "panel-body":

<div style="text-align: center;">
    <i class="fa fa-check" style="font-size:48px;color:green;"></i>
</div>

"Not Releaseable" versions only contain the following HTML in their "panel-body":

<div> <buildsummary buildid="1539736" condensed="1"> </buildsummary></div>

AngularJS is supposed to handle <buildsummary />, but I didn't understand if buildbot 4.0 still uses AngularJS or switched to React.

[encukou]

I'm back from EuroPython now.

For me it's not only the custom page that's misbehaving:

• Console & Grid views are down as well (stuck on a Loading spinner)
• Waterfall loads, but only shows the header
• In Builds/Builders, cells in the "Builds" column are stuck at "Loading..."

All show 504 (Gateway Timeout) HTTP responses to various API requests. I guess the pages will load if the server was lass busy.

---

According the the 4.0 upgrade instructions, all plugins do need to be rewritten to use React. (They recommend doing it before the big update, but I guess that's not an option any more.)

Does anyone here already know React, or should I learn it?

[pablogsal]

All the pages you mention are supposed to be in the included buildbot package so there is nothing we need to rewrite in our side. Can you confirm this with the buildbot team perhaps?

[vstinner]

Be careful of old URLs including /all/.

[encukou]

Right, the pages I mentioned are part of Buildbot; I opened https://github.com/buildbot/buildbot/issues/7820.

We do need a React rewrite for the release dashboard, which is a custom plugin. According to dashboard docs, there still is an AngularJS application, but that's probably stale docs.

[encukou]

I also get 404 on JS and CSS files: https://github.com/buildbot/buildbot/issues/7439

[vstinner]

I created https://github.com/buildbot/buildbot/issues/7826 "buildbot 4.0: Builders page doesn't limit to 25 builders per page".

[vstinner]

In Builds/Builders, cells in the "Builds" column are stuck at "Loading..."

The problem may be related to https://github.com/buildbot/buildbot/issues/7826 : lack of pagination. Loading all data for 100 or 200 builders take too long, and a timeout somewhere kill the request, so the UI is stuck at "Loading...".

If you limit the Builders view by clicking on a tag, the Builds column can be properly loaded.

[vstinner]

On Firefox, the Waterfall View is truncated, the view seems to be limited to like 100 pixels!?

Same on Chromium.

Something limits the height to ... 5 pixels!?

[vstinner]

I changed the Release Status page to an ugly but working page: https://github.com/python/buildmaster-config/commit/e5d6b0672b9c189a64c2c09eb91d77e20c4c7cdd

[encukou]

Thank you!

[encukou]

I found another issue: https://github.com/python/buildmaster-config/issues/509

[pablogsal]

I think it's time to rollback the upgrade, the buildbot page it's certainly less usable and the release dashboard it's broken and everything feels either slower or slightly broken.

[edelsohn]

Each release of Buildbot has been more bloated and slower. That apparently is progress. It's more features! It's not going to get better. Buildbot developers behave as if they are funded by cpu and memory manufactures to encourage system upgrades.

[pablogsal]

@encukou could you please revert https://github.com/python/buildmaster-config/commit/e5d6b0672b9c189a64c2c09eb91d77e20c4c7cdd and the buildbot 4.0 upgrade PRs? If you don't have free cycles I can give it a go if you preefer

[pablogsal]

Also, I think we should bring this up with the buildbot project. We are probably among their heavier users and I am not sure if they are informed of all the problems we are having

[zware]

I think it's time to rollback the upgrade, the buildbot page it's certainly less usable and the release dashboard it's broken and everything feels either slower or slightly broken.

What's broken other than our custom release dashboard? I'm not seeing the same "everything is slower"; on the contrary, everything actually seems rather snappier since a selective pruning of the database.

We did rush into 4.0, but I'm not convinced that a downgrade is worth the hassle.