elvish
elvish copied to clipboard
elves
Investigate binary size from dependencies
The current HEAD version builds into a 14MB binary on macOS. When Elvish starts the entire binary is loaded into RAM, occupying 14MB of RAM.
In comparison, on my machine, zsh occupies roughly 3MB of RAM when started, bash slightly less than 3MB.
Consider using the plugin package introduced in Elvish 1.8 to dynamically link some components like the line editor and the web interface to reduce the binary size.
Hmm, in fact the whole binary is not loaded into RAM at once. However, the latest HEAD version of Elvish does occupy roughly 14MB of RAM when started up.
Older versions do occupy much less RAM. For instance, the 0.8 version occupies slightly less than 7MB, which seems quite acceptable. It should be investigated why the RAM usage has doubled.
Some more accurate information:
-
On my machine, Elvish occupies slightly less than 12MB of RAM when started, not 14MB.
-
The culprit for the increase in RAM usage since 0.8 seems to be the addition of the web interface. While that addition did not increase the binary size significantly, it did blow up the RAM usage.
On my macOS server starting an interactive elvish shell built from Git head as I type this, with a modest number of modules (mostly from those documented at https://github.com/zzamboni/dot-elvish), results in a RSS (resident set size) of 16 MB.
My main complaint is that the web UI is not documented. It is also probably not used by 99.9999% of elvish users. It should probably be a distinct binary rather than a feature of the usual elvish CLI program that is enabled via a flag. The only good explanation of the Elvish web UI I found was this closed issue: https://github.com/elves/elvish/issues/387.
Also, metrics such as RSS are inherently problematic since they don't reflect the sharing of memory by other instances of elvish, or any other program, of static pages. And are therefore often more pessimistic than reality unless we are talking about a single elvish process running on the system.
RSS of a just-started Elvish in my system right now is 20MB 😮
I agree the web UI is probably something that could be separated (does anyone use it? Maybe it could be retired?).
@xiaq, How do you feel about making a web accessible Elvish shell a separate program rather than a feature enable via elvish -web? Doing that should reduce the overhead of starting a non-web enabled elvish process. Which is what most elvish users probably want. I love mechanisms such as https://play.golang.org/. But that doesn't mean the common case of a program written in Go has to support a REPL Web UI. In other words, why should the normal elvish binary support a web UI given that the vast majority of the time anyone using it does not need the Web UI functionality?
The main program no longer includes the web subprogram. See https://github.com/elves/elvish/commit/4ac464a1682b9548a4309a14a89dd88e3dac5136 for the change in RAM consumption and binary size (it's not very big).
With your recent change I see a 4% decrease in file size and a 3% reduction in initial RSS on macOS. Which is a surprisingly small decrease. The core Go runtime itself is on the order of 2 MiB. Running nm on the new elvish binary shows that it still includes net/http. Sadly, importing net/rpc transitively imports net/http so there isn't any way to avoid that overhead short of a major architectural change to eliminate the daemon process.
P.S., Running elvish -help still shows the -port and -web options and they are silently ignored.
Hmm, net/rpc's dependency on net/http surely explains the small magnitude of reduction.
According to goweight, here are the two 10 largest packages that contribute to Elvish's binary size:
~/go/src/github.com/elves/elvish> goweight | head -n10
3.9 MB net/http
3.8 MB runtime
1.9 MB github.com/elves/elvish/pkg/eval
1.8 MB net
1.8 MB crypto/tls
1.5 MB golang.org/x/sys/unix
1.4 MB reflect
998 kB math/big
977 kB github.com/elves/elvish/pkg/edit
881 kB encoding/gob
The net/rpc package now used to implement communication with the daemon has many features that Elvish does not use. Maybe switch to a more lightweight implementation.
https://github.com/urld/fatdeps provides a graph of package sizes and dependencies in the elvish binary. It confirms that net/http (13% of the binary) is included only as a consequence of using net/rpc (1% of the binary).
Note also that crypto/tls (and related packages) is quite large and included only due to net/rpc. Whether potentially reducing the size of the elvish binary by ~20% justifies changing the implementation is debatable. Especially since a dependency on a third-party package such as gRPC is unlikely to be substantially smaller compared to just using net/rpc and in fact is likely to be bigger. The alternative is a custom RPC implementation that does only what elvish needs. But now you're reinventing the wheel which is hard to justify.
Looking at the source code of net/rpc, Elvish does not actually use any code from net/http, even transitively.
A smart-enough linker should be able to remove the dead code, but Go's linker currently disables function reachability analysis whenever reflect.Value.Call is used. There seems to be some work in that area to lift this unnecessary restriction, but I am not sure whether it will land in Go 1.15. (There is an ongoing effort to virtually rewrite the linker.)
Another possibility is to vendor a stripped down version of net/rpc, removing the parts that depends on net/http. It's a tiny library and hasn't been touched for years anyway.
Promising results from gotip:
~/go/src/github.com/elves/elvish> go version
go version go1.14.2 linux/amd64
~/go/src/github.com/elves/elvish> gotip version
go version devel +53f2747 Sun May 3 07:23:32 2020 +0000 linux/amd64
~/go/src/github.com/elves/elvish> go build -o elvish
~/go/src/github.com/elves/elvish> gotip build -o elvish-tip
~/go/src/github.com/elves/elvish> stat -c %s elvish
13814455
~/go/src/github.com/elves/elvish> stat -c %s elvish-tip
11020553
A reduction of 20% in binary size!
And a very rough estimation of how much of net/http is built into the binaries:
~/go/src/github.com/elves/elvish> strings elvish | grep net/http | wc -l
2163
~/go/src/github.com/elves/elvish> strings elvish-tip | grep net/http | wc -l
831
This issue should probably be closed. It's 6.5 years old and the current Elvish binary size is very close to that from the previous comment made 4 years ago. Which is slightly surprising given the number of features added in the past four years. The only practical way to dramatically reduce the size of the Elvish binary is to replace the go.etcd.io/bbolt dependency with a more traditional command history as proposed in issue #126. See also issue #1222.