proposal: break out matured packages into repos

[hugelgupf]

This might be a bit ironic at first glance, given how much I wanted systemboot and u-root to merge back together. But hear me out.

u-root has become a pretty big grab-bag of tools. We've written a pretty extensive set of packages for all kinds of system software. Some of it is starting to mature and not be touched that often anymore.

I realized the other day that I don't think hugelgupf/p9 would have gotten Windows or BSD support from someone if it had been part of u-root. It'd have been hard to discover, and hard to contribute to.

A lot of it, I think, could also be reusable by other Go system software projects. But I think that is hindered by two factors:

• discoverability: it's hard to find that e.g. we have an implementation of cpio in u-root, because the repo is huge, and it's not indexed by GitHub that way.

• contributability: if I'm someone who just wants to use the cpio package, and/or fix one small mistake, it's hard not to be intimidated by the size of the repo and the testing; as well as the potential dependencies cpio might have. I might not use pkg/cpio from u-root because it's hard to tell how many other packages I'll be pulling with it, despite the fact that cpio is relatively self-contained here.

It's self-contained packages (not commands) like cpio that I think are good candidates to split out into their own GitHub repos and to maintain separately.

pros:

• small projects are easier to use for other projects
• small projects are easier to contribute back to
• ensures that our packages remain self-contained, and don't grow to a huge web of dependencies
• if we want to bring in new dependencies that use alternative packages to ours, we could conceivably convince them to use our version (if it's better in some way)
  • for example, I broke out https://github.com/hugelgupf/go-shlex so that I can later send a PR to replace anmitsu/go-shlex in https://github.com/gliderlabs/ssh, which we use in cmds/exp/newsshd.
• if we get other users, maybe they'll add other OS/Arch support as needed for them

cons:

• maintenance becomes more disaggregated -- more repos to look after
• harder for us to make sweeping changes across u-root, as now you have to make changes in stages across different repos
  • mitigation: only break out relatively small or relatively matured self-contained packages
• harder to import u-root into various companies internal repositories -- the more repos, the more to maintain
• until we have Go module support, we can only split out packages, not commands, or we'll end up with multiple vendored copies of dependencies in the final binary
  • mitigation: get Go module support!

some possible candidates

• pkg/cpio
• pkg/shlex
• pkg/qemu, maybe even pkg/vmtest
• pkg/uio should be split up anyway, into something like pkg/uio/lazyfile, pkg/uio/readers (for reader.go, progress.go), pkg/uio/cachedreader (cached.go CachingReader), pkg/uio/lexer (for buffer.go Lexer and Buffer types)
• pkg/lockfile
• pkg/bb - all the Go busybox tooling

less sure about these, but I think could be good:

• pkg/gpio
• pkg/ipmi (eventually, when matured?)
• pkg/msr
• pkg/pci
• pkg/smbios (maybe dedup with https://github.com/digitalocean/go-smbios?)

What do y'all think?

[sevki]

Hey @hugelgupf now that he have a an other OS in here :) could we revisit this?

[hugelgupf]

@sevki I'm happy to revisit it. I was hoping to kind of split out things as needed over time -- this is not one big projects that can be done at once. I also talked about it a bit in #1829.

[hugelgupf]

Are there particular packages you're interested in?

[sevki]

Harvey packages I thought would have been a good candidate. Mount could be a good one since it has two implementations as well :)