Specify Bobfile path or a [[container]] name to build in REST Interface

[danielschonfeld]

In my setting, I have many different services under the same git repo. I need to build like 5-6 different images and i'd like to not have to build all of them everytime I push to github or perhaps even when I manually queue a build via the REST interface.

It would be very helpful adding a [[container]] name or a bobfile path when doing something like the following:

curl -s -XPOST http://localhost:5000/jobs -d '{"account":"modcloth","repo":"docker-builder","ref":"master","bobfile":"services/backend-service-1/Bobfile"}'

or maybe:

curl -s -XPOST http://localhost:5000/jobs -d '{"account":"modcloth","repo":"docker-builder","ref":"master","container_name":"backend-service-1"}'

cc @rafecolton

[rafecolton]

Hi @danielschonfeld, thank you for contributing!

I like both of your suggestions, so I am going to implement both of them.

Following are my initial thoughts. tl;dr

"bobfile":

• If value is empty string or key is not provided, default to Bobfile
• For safety, reject values that contain ".."
  • Should this fail the build?
  • Another option would be to replace any occurrences of ".." with "/", which would then get cleaned up with path.Clean(). This seems less than ideal though - we should probably just fail the build.
• For safety, reject values that contain symlinks
  • This should probably fail the build too
• All paths will forced to be relative to project root (i.e. path = projectRoot + "/" + path)

"container_name":

I actually want to approach this one a little differently. Instead of specifying a container name, users should be able to specify a container group that will contain a subset (but not necessarily proper subset) of all [[container]] sections in a Bobfile.

My initial thought on implementation would be to include a stanza called groups in each [[container]] section that would contain an array of group names. This brings up a couple of questions off the top of my head:

1. What happens if the groups stanza is specified in the [container_globals] section?
   • Does it get overwritten by the groups stanza in an individual container section? This would be consistent with the behavior of other stanzas in [container_globals]
   • Does it get merged into the individual [[container]] groups? This would be convenient for specifying an all-inclusive group, but it would be inconsistent with the default [container_globals] section behavior, which is probably not a good thing.
2. Can the user specify multiple groups to build? If so, what is the syntax? It should probably be a JSON array, but if instead it's something like a comma-separated value, does that preclude specific values (i.e. commas in this case) from being in the group names?
3. Can this be easily implemented without lots of nested for{} loops or otherwise difficult-to-read logic?
   • The answer may very well be yes, I haven't actually thought about it in detail yet
   • Does it matter?
4. Is there an overall better way to implement this feature?
   • I'm open to suggestions...

Additionally, it would probably be helpful to specify these two options statically at a per-repo level so that they can be used with GitHub/Travis webhooks instead of the need for manual build invocation. It seems like the easiest way to do this would be with a .bobconfig file or similar. I will open another issue for that.

Finally, I will update this thread with the branch name once I start the implementation.

[rafecolton]

@danielschonfeld this work is taking place on the bobconfig-bobfile-and-container-groups branch

[danielschonfeld]

@rafecolton thank you for taking this request and running with it. I know it's going to make my life a hell lot easier to implement a CI pipeline that relies on docker-builder.

I agree with most everything you wrote above, but I do have one question about groups stanza; if you wanted to only build one container form a list of 5 different containers, would you have to put each container in it's own group? ie:

/backend-services/Bobfile:

[[container]]
name=redis
group=redis

[[container]]
name=elasticsearch
group=elasticsearch

Assume that this was all in the same repo, and you wanted to set up a CI pipeline where every time there was a push to a git repo, you would check if files changed in any of the subdirectories inside /backend-services and build the appropriate container.

Perhaps there's room to implement both group and container-by-name. Or maybe not :).

[rafecolton]

@danielschonfeld if you think this will make your life easier now, wait until it gets integrated with quay.io :wink:

So the way it's designed above, yes, you would have to make a different group for each container. When stated that way, it doesn't seem that much more convenient, but I think that's because it's actually a different use case.

I think maybe the best way to break this down is to try to describe the use cases in plain English.

The first use case is, "My application requires a bunch of relayed or layered containers, but I don't need to build them all every time." For this case, container groups make a lot of sense. Maybe you have a base layer and two app layers (like a front end and back end), so you have a "base" group and an "app" group and only build the "app" group by default.

The second use case is, "I have a bunch of related services in a shared repo, so I only need to rebuild the one I'm working on." This one I think is your use case, and it could be served equally well with either container groups or just a separate Bobfile for each service. Both solutions require a greater quantity of a thing than might be a desirable (either number of bobfiles or number of unique groups), but such is the nature of the use case.

The final case isn't really addressed by either of these features: "I'd like to build all containers specified in each Bobfile that lives in a directory in which at least one file was changed in the most recent git commit." I think this would actually be the most useful for your particular use case and also potentially the most interesting :smile: It is a bit out of scope for this feature request, but I think it would be really interesting to implement, so I will open another issue for that feature.

[rafecolton]

The Bobfile path part of this request is addressed here

Still working on the rest - would love for you to build from the above branch and give some feedback