Add an example of using codebases

[tardyp]

This ticket is a migrated Trac ticket 2945

People contributed to the original ticket: @sa2ajj, benoit.allard@... Ticket created on: Oct 15 2014 Ticket last modified on: Jan 15 2015

---

While the notion is introduced and all relevant options are documented, it seems to be a good idea to add a separate example where codebases are used.

---

Comment from: @Ben Date: Oct 20 2014

I could help there. Just need an example where it makes sense to use codebases: i.e. It is necessary to checkout multiple repository in order to run a command that need the outcome of each of them.

---

Comment from: @Ben Date: Oct 20 2014

It's been suggested on IRC to add a picture about the conversion from changes to sourcestamps to buildset in the introduction chapter. I'll give it a shot.

---

Comment from: @sa2ajj Date: Nov 07 2014

Based on the [[IRC discussion|http://irclogs.jackgrigg.com/irc.freenode.net/buildbot/2014-11-07#i_3490587]], a couple of different codebase mappings need to be shown and how they are handled by different schedulers should be explained.

---

Comment from: @sa2ajj Date: Jan 07 2015

Things to cover:

• codebase generator
• scheduler configuration
• change filter
• use of codebase in build steps

Ben, an example where it makes sense is a two repo setup (e.g. lib + app) where we want to build both regardless of whether the change is coming to the lib repo or to the app repo. And in this scenario we'd like to build the revision for the change + master/stable of the other repo.

---

Comment from: @sa2ajj Date: Jan 15 2015

One seemingly common use case: checkout two branches (of the same or different repositories) -- one according to the change, one according to the configuration. With the "default" configuration, both are checked out with the revision in the change, which causes errors.

[Frodox]

ping @ben ?:) Do you have enough motivation to still help here? This part is highly important, because I still not very clear with correct use-case of codebase (well, yes, I need to build thousands repos, but not clear when codebases can be useful)

+1 for picture

[WayneKeenan]

I'd certainly be interested in seeing an example of using codebases, is there one or more available yet please?

I've found a few pieces of info but haven't manged to get it to work: https://github.com/buildbot/buildbot/issues/3351 http://buildbot.readthedocs.io/en/latest/manual/concepts.html#attr-codebase

But for now am going with the pre- 9.0 advice I found here which works for me:

https://stackoverflow.com/questions/25560328/buildbot-fails-on-reset-hard

Thanks Wayne

[jwitheford]

I would also be keen to see how codebases should be used. Is there any further progress on this?

[p12tic]

I could look into this ticket in near future.

[0x8000-0000]

Ping?

[kfriedberger]

Also Ping?

[WayneKeenan]

Ping too.

[kfriedberger]

Ok, figured it out myself :-)

Short summary: Our Buildbot instance is running since years with a single codebase, i.e. the repository of our software. As we moved some input data for our software to a different repository, we wanted to run a new build of our software with new input data, if any of the two repositories notifies the BuildBot via a commit hook.

Enabling multiple codebases is actually straight-forward. I had to do the following steps:

• the starting point was this documentation (the graphic is really nice to get an overview).
• set up different codebases with a simple codebase generator.
• enable codebases for the schedule. In our case it was sufficient to use the parameter codebases=[list of all codebases] for the AnyBranchScheduler.
• enable codebases for every source step (SVN/Git update step) that accesses codebase or revision information. This is also required for source steps that only access HEAD revisions (alwaysUseLatest=True), because all source steps need an non-empty codebase, if any source step has a codebase. (If this is forgotten, the exception is clearly visible in the log :-) ).
• enable codebases for all property and interpolation usages, i.e. replace prop:revision with src:<codebase>:revision.
• we do not use filters or other advanced stuff.

Take care of the following hints:

• by default, the information after a commit hook will only contain the information about the commit and its repository, other codebases do not have any revision information available (looks like empty string). Thus, updates for those repositories will result in HEAD revision. That was ok for us. Using util.Interpolate("%(src:codebaseX:revision:~HEAD)s") for build steps that require a valid non-empty revision information provides either the correct revision of repository codebaseX (if it triggered the build) or HEAD. (Maybe there is an easier way for such default values. As it works for us, I did not try further).
• somehow the names of build steps are limited to 50 chars (database restriction?). When using codebases, their names will automatically be appended to the names. We had to shorten some names to avoid crashes.

After those small changes, everything works nice as expected.

[tardyp]

hi @kfriedberger thanks for your detailed summary!

We would love you go a bit further and contribute the tutorial/example that everybody is asking for in this issue. Most of our tutorial documentation is written by newcomers like you. It makes more sense, because you have a new view, so you know what are the key points to understand.

Maintainers can help for the details if there is still some blurry detail.

[JanezCim]

+1 Thank you for explaining. Is a clear code example available somewhere yet?

[Try2Code]

yes, a working example would be great. My project uses multiple forks for multiple developer groups. Changes can come from everywhere (which works fine via merge requests). But nightly scheduler for a single repo fail with codebase errors

esp. the part with codebase generators is hard to understand for me atm