skx
[question] How do you intend to handle module `init` latency as dependencies pile up?
First of all let me say that I don't criticize your busybox-like single binary that contains multiple tools. (I myself have a similar project that bundles inside the same binary multiple smaller tools.)
However, as per Go's runtime, at the moment every dependency's func init () is run at (each) program startup regardless if one actually will need that dependency in that particular run. (I.e. there is no lazy-initialization, and if you bundle both some terminal library and some RPC library, they'll run their initialization routines regardless if in that particular run you actually use their code.)
For example at the moment the startup latency seems to be around 4-5 ms just for running sysbox version. For comparison the execution of /usr/bin/true takes around 0.5 ms and bash --version takes around 1 ms.
You can profile this by running:
GODEBUG=inittrace=1 ./sysbox version
In your case it seems that the following are the top latency causes:
golang.org/x/net/htmlwith 0.4 msnet/rpcwith 0.4 ms (the one that also plagued my own tool)github.com/gdamore/encodingwith 0.3 msgithub.com/rivo/tviewwith 0.2 msgolang.org/x/crypto/sshwith 0.2 msgoogle.golang.org/protobuf/types/descriptorpbwith 0.1 msgithub.com/hashicorp/go-sockaddrwith 0.1 ms
As you can see, just initializing net/rpc and the HTML library eats almost as much time as running bash --version...
If you want to dig deeper into what I'm describing, I've wrote last year a short article about the subject here https://notes.volution.ro/v1/2021/02/notes/378ae6f6/ where I specifically take into account the use-case of Go for "single binary do all" tools like yours. And for the record, I still don't have a solution... Thus asking you for thoughts. :)
Honestly? It's something I'm aware of, and I've seen people discuss on blog-posts like your own. But it is something that I've mostly ignored.
It is true that over time adding more and more libraries will slow things down, but I think that cost is essentially overridden by the utility of the various tools themselves.
Ideally "everything" would be "fast", and I lament the way that we support, encourage, and are resigned to slow, buggy and bloated applications and tools - but at the same time each of the things I added here is probably faster than writing a pipeline with tools such as awk, sed, etc.
Ultimately I think my answer is "convenience of the tools outweighs slowness". But I guess that's an easy argument since I personally use almost all of them, I suspect other people might think "Oooh that's neat" about one tool in particular, and not care about the rest.
