u-root: Go modules

[hugelgupf]

Hi all,

I've been doing some thinking on Go modules, and I'd like to hear what people think. As a side note, build system support for Go modules (and bazel!) is developing in https://github.com/u-root/gobusybox. Give it a try if you have some time, but it may still be a little rough around the edges (error messages need work).

In general, the Go ecosystem expects that you increase your major version number when your external API changes. Given the size of u-root (100+ commands, 100+ packages) that's likely to happen at pretty much every single release. (I'd wager that almost every single commit changes some API that would be affected.)

I think that's excessive, and will make u-root a little harder to interact with. Also, with every release we'd have to rewrite imports in every single Go file in the repo by doing a big sed with s#github.com/u-root/u-root/v2#github.com/u-root/u-root/v3 or something like that. (That's probably hard to get around.)

Option 1

1. The main u-root repo should not have any external-facing Go packages. All packages move to internal/, which the Go compiler restricts to be used inside the repo.
2. All packages which we want to be used externally (or which currently are) move out of this repo into their own u-root repository. They each get their own Go module, and are appropriately versioned & released. Some candidates are just like in #1753, where I previously talked about this.
3. u-root remains at the same major version it is now at - v7. We only do minor version releases, because the external Go API never changes.

Pros: we clearly delineate what's internal & external. Smaller projects are easier to reuse by others, and we may get some contributions back (same as in #1753).

Cons: more repos means more Go modules means more releases. They all need to be appropriately tagged & released every once in a while, or when a user requests so. Development gets a little slower: have to first modify & land changes to other repos; then do the same in u-root.

Option 2

We suck it up and do a major version release every couple of weeks. Rewrite all import paths, it's just a bit of sed.

Pros: no need to decide what's internal and external. Development remains as 'quick' as it is today, with no complex web of dependencies. We can still split out some packages as proposed in #1753.

Cons: lot of large changes for every release, though mechanical.

[GanShun]

I vote 2, Chrome pretty much does this today I think. The additional con of doing 2 though is since pretty much every release is a major release, none of the automatic minimum version selection will happen when u-root is used in other repos. They'll pretty much have to manually rev it all the time.

[JulienVdG]

In my new job, we have a process more like 1, and I often miss the simplicity of the single repository in u-root.

my 2 cents

[fotonick]

It feels like a big switch to go fully with option 1 to the max. An in-between is to pursue option 1 only for components that are 1) really stable, 2) have a simple dependency tree within u-root, and 3) the community outside of u-root would care. You'd gain the benefits without much downside. To go down this route, we might write down the criteria somewhere and let folks propose on a package by package basis.

On Mon, Sep 14, 2020 at 12:31 PM Julien Viard de Galbert < [email protected]> wrote:

In my new job, we have a process more like 1, and I often miss the simplicity of the single repository in u-root.

my 2 cents

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/u-root/u-root/issues/1829#issuecomment-692266871, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAPJ3VAGFUZIKGBLM2SLJ3DSFZVS7ANCNFSM4RLHI4KA .

[rminnich]

I'm inclined to prefer 2. The original goal of u-root was to provide a freestanding system that people could both use and learn from. The scope of the split will, I think, make that original goal a lot harder.

Ryan had set up a release process which we were supposed to do every 6 weeks. This was based on the chrome cycle, b/c we learned in chrome that 6 week cycles had the nice property that if you missed a release, it was not so long until the next one. But we've not done a good job of keeping the release up. If this could be automated, maybe that problem would go away.

But we've found with projects like nichrome and webboot that every additional dependency is a headache, and I am worried that splitting u-root as in option 1 will increase the headaches.

[hugelgupf]

Alright, I hear ya. The simplicity of the "monorepo" seems preferred.

Re criteria, I haven't thought about it much, but I'll say why I broke out https://github.com/u-root/gobusybox -- it needed bazel support, and I didn't want to add bazel BUILD files to all of u-root (and expect all contributors to know how to use bazel). It also needed a more intense test env - test under Go modules, Go vendoring, and bazel - and everything else in u-root doesn't need it. We're ok with testing u-root at Go 1.13, but I thought it useful to test gobusybox at 1.13, 1.14, and 1.15 because it interacts closely with the build system.

Maybe more generalized criteria from that example:

• we want to maintain support for something that not all of u-root cares about (Windows support for a package, maybe, or bazel support)
• needs longer or more complicated CI integrations to test well - stuff that we also don't want to do with all of u-root.

Ultimately, both would be possible within u-root as well though. But depends on whether we want to be relatively uniform on what we support with u-root.

[orangecms]

I'd very much prefer 2. From the npm ecosystem I know the pain of hundreds and thousands of dependencies piling up and occasional clashes.

I do see a point in splitting up u-root into a few things, very much as described in https://github.com/u-root/u-root#description. Pulling out the busybox tool and the cpio tool makes sense as standalones imho. To get the current behavior, a very simple script or program would still do. That could still live in one repo with the cmds to maintain the presets that are currently defined.

[hugelgupf]

I do see a point in splitting up u-root into a few things, very much as described in https://github.com/u-root/u-root#description. Pulling out the busybox tool and the cpio tool makes sense as standalones imho. To get the current behavior, a very simple script or program would still do. That could still live in one repo with the cmds to maintain the presets that are currently defined.

Yeah, this weekend I also started https://github.com/u-root/mkinitramfs for the remaining build tools, because we also need bazel support for that. Same reason -- I don't want normal u-root contributors to have to learn how to use bazel to pass the tests etc (despite how awesome I think Starlark is).