lf-edge
Reduce EVE rootfs size by removing unused files and stripping binaries
EVE rootfs is close t 250Mb in size. The obvious option is to increase a partition size but first let's try to get rid of unused files/tools/packages and strip debug information from binaries. We cannot increase rootfs size > 300 MB
- [x] Strip VTPM libraries https://github.com/lf-edge/eve/pull/4324
- [ ] Remove VIM from debug container (~8Mb) https://github.com/lf-edge/eve/pull/4328
- [x] Increase rootfs size to 270Mb https://github.com/lf-edge/eve/pull/4329
Are we comfortable with shared libraries vs static binaries? I guess within a single container it's fine, although we would like to break pillar apart.
Then again, does it matter, if most of it is a single pillar binary?
Then again, does it matter, if most of it is a single pillar binary?
There is f.e. also the vtpm binary
single container
containerd is 36MB, I guess it would also profit a bit from shared libraries
I think if we have the same (i.e. same hash sum) shared library in different containers, then squashfs can compress it very well.
It would be an interesting experiment to see what actually happens if we use shared libraries. We could be spending all of this time to discover that it makes little difference.
I know that if you launch two copies of the same executable, the read only memory consumption is once. Same for shared objects. If two distinct executables, each way 5MB, use a shared object of 3MB, then total memory is 5*2+3, and not (5+3)*2, ie one copy of the shared object.
What about statically compiled? Is it intelligent enough to recognize the common bits? I would think no, but @rucoder would know much better.
@deitch this is exactly why shared libraries were born. Static images cannot share their code. I would be nice if we can have all libs in /lib in 'on_init' section in linux kit. LK can do some magic to move all libs from all containers to /lib or /usr/lib and mount them into containers behind the scene
this is exactly why shared libraries were born
I know. I didn't know if in the years since, loaders had gotten smarter about recognizing identical binary chunks in different files.
Well, this thread isn't really about memory usage, is it? I was just checking that as as side question.
I would be nice if we can have all libs in /lib in 'on_init' section in linux kit. LK can do some magic to move all libs from all containers to /lib or /usr/lib and mount them into containers behind the scene
You mean to share libraries across containers? I don't see it. Nothing in containerd infrastructure supports it now, I haven't seen anyone else doing it.
I have seen people mounting some shared libs from a host filesystem into multiple containers, but it isn't all that common. Mainly because size of root filesystem doesn't matter all that much to most?
@deitch this is exactly what we need to solve: the image size. I think it is not that hard to bind mount /hosfs/lib into /lib inside the container. We can even do it manually in build.yml but LK must move all libs from individual containers to host fs and de-duplicate them
Nothing in containerd infrastructure supports it now
If two containers share the same base layer, then in the overlayfs they have the same lower mount, don't they? Then they share the files.
yes, if SHA is the same but all our containers are FROM scratch with different SHAs
this is exactly what we need to solve: the image size
Right, not memory usage. Agreed.
I think it is not that hard to bind mount /hosfs/lib into /lib inside the container. We can even do it manually in build.yml but LK must move all libs from individual containers to host fs and de-duplicate them
Doable? Yes. I am really concerned about the management and synchronization overhead. We would need to have something that makes it easier to use. Like we have our FROM eve-alpine system that ensures it all stays in sync, we would need something like that here. And then how do you have each container run on its own, when not mounted?
As a general rule, containers that cannot run on their own, that depend on some volume mount or bind mount just to get executables to run, goes against the grain.
In any case, are we really sure that shared libraries across containers is what will make the difference? As it is, pillar - the biggest one - is mostly a single binary anyways.
We should take a good hard look at a built eve and see where the sizes are big.
FWIW, I just built eve from latest master commit:
$ du -s * | sort -n
4 config
4 home
4 mnt
4 persist
4 proc
4 root
4 run
4 srv
4 sys
4 tmp
8 opt
12 dev
16 media
84 var
1292 EFI
1372 init
3844 etc
4180 sbin
4236 bin
22580 boot
127140 usr
234784 lib
585720 containers
So the only things that really matter:
- containers
- lib
- usr
- boot
Starting with the smallest:
$ du -s -h boot/*
4.0K boot/cmdline
14M boot/kernel
7.8M boot/ucode.img
1.2M boot/xen.gz
Kernel and code, not much else there.
$ du -s lib/* | sort -n
0 lib/libc.musl-x86_64.so.1
0 lib/libcom_err.so.2
0 lib/libfdisk.so.1
0 lib/libmount.so.1
0 lib/libpam.so.0
0 lib/libpam_misc.so.0
0 lib/libpamc.so.0
0 lib/libsmartcols.so.1
0 lib/libuuid.so.1
0 lib/libz.so
4 lib/mdev
4 lib/modules-load.d
8 lib/sysctl.d
16 lib/libcom_err.so.2.1
16 lib/libpam_misc.so.0.82.1
16 lib/pkgconfig
20 lib/libpamc.so.0.82.1
32 lib/libuuid.so.1.3.0
52 lib/resolvconf
60 lib/libpam.so.0.85.1
88 lib/libkmod.so.2
100 lib/libz.so.1
100 lib/libz.so.1.2.12
100 lib/libz.so.1.2.13
128 lib/libudev.so.1
180 lib/libapk.so.3.12.0
200 lib/libsmartcols.so.1.1.0
308 lib/libblkid.so.1
308 lib/libblkid.so.1.1.0
332 lib/libmount.so.1.1.0
400 lib/libfdisk.so.1.1.0
512 lib/libssl.so.1.1
556 lib/udev
592 lib/ld-musl-x86_64.so.1
592 lib/libssl.so.3
740 lib/security
996 lib/apk
2552 lib/libcrypto.so.1.1
3792 lib/libcrypto.so.3
65556 lib/modules
156420 lib/firmware
lib/firmware is 153MB. 202 lines in there, not going to post it all here. But most are .fw or .ucode files of 1.5-2MB, with a few really big ones:
$ du -s lib/firmware/* | sort -n
...
4236 lib/firmware/intel
6232 lib/firmware/cypress
8876 lib/firmware/display-t234-dce.bin
8996 lib/firmware/mrvl
9452 lib/firmware/ti-connectivity
13732 lib/firmware/brcm
19960 lib/firmware/ath10k
Do we need all of those on every build?
The only other really big one in lib was:
$ du -s lib/modules/6.1.106-linuxkit-06a737b0f212/* | sort -n
0 lib/modules/6.1.106-linuxkit-06a737b0f212/build
12 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.order
28 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.builtin
28 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.dep
36 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.builtin.bin
40 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.dep.bin
164 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.symbols
192 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.builtin.modinfo
196 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.symbols.bin
316 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.alias.bin
320 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.alias
8288 lib/modules/6.1.106-linuxkit-06a737b0f212/extra
55928 lib/modules/6.1.106-linuxkit-06a737b0f212/kernel
Moving on to usr:
$ du -s -h usr/*
104M usr/bin
3.8M usr/include
14M usr/lib
388K usr/libexec
20K usr/local
2.0M usr/sbin
720K usr/share
OK, mostly usr/bin:
$ du -s usr/bin/* | sort -n
...
11648 usr/bin/containerd-shim-runc-v2
13168 usr/bin/runc
14252 usr/bin/service
20324 usr/bin/ctr
37128 usr/bin/containerd
Only really big things there are containerd, runc and service.
Last /containers:
$ du -s containers/*/* | sort -n
28 containers/services/pillar
1444 containers/onboot/000-rngd
1884 containers/onboot/001-sysctl
6764 containers/services/watchdog
8272 containers/onboot/006-measure-config
8612 containers/onboot/003-kdump
11460 containers/services/wlan
13140 containers/services/memory-monitor
18592 containers/services/newlogd
18648 containers/onboot/005-apparmor
21460 containers/services/guacd
24936 containers/services/edgeview
28708 containers/onboot/002-storage-init
45268 containers/services/wwan
53160 containers/services/vtpm
88840 containers/services/debug
97104 containers/services/xen-tools
137388 containers/onboot/004-pillar-onboot
Most of them are pretty small. The last bunch start getting really big.
$ du -s containers/onboot/004-pillar-onboot/lower/* | sort -n
0 containers/onboot/004-pillar-onboot/lower/containers
4 containers/onboot/004-pillar-onboot/lower/dhcpcd.conf
4 containers/onboot/004-pillar-onboot/lower/fscrypt.conf
4 containers/onboot/004-pillar-onboot/lower/home
4 containers/onboot/004-pillar-onboot/lower/init.sh
4 containers/onboot/004-pillar-onboot/lower/mnt
4 containers/onboot/004-pillar-onboot/lower/proc
4 containers/onboot/004-pillar-onboot/lower/root
4 containers/onboot/004-pillar-onboot/lower/run
4 containers/onboot/004-pillar-onboot/lower/srv
4 containers/onboot/004-pillar-onboot/lower/sys
4 containers/onboot/004-pillar-onboot/lower/tmp
12 containers/onboot/004-pillar-onboot/lower/dev
16 containers/onboot/004-pillar-onboot/lower/media
104 containers/onboot/004-pillar-onboot/lower/var
1604 containers/onboot/004-pillar-onboot/lower/etc
2320 containers/onboot/004-pillar-onboot/lower/bin
5068 containers/onboot/004-pillar-onboot/lower/sbin
7076 containers/onboot/004-pillar-onboot/lower/lib
52460 containers/onboot/004-pillar-onboot/lower/usr
68652 containers/onboot/004-pillar-onboot/lower/opt
So usr and opt.
$ du -s containers/onboot/004-pillar-onboot/lower/usr/bin/* | sort -n
...
1032 containers/onboot/004-pillar-onboot/lower/usr/bin/sgdisk
1100 containers/onboot/004-pillar-onboot/lower/usr/bin/coreutils
2044 containers/onboot/004-pillar-onboot/lower/usr/bin/qemu-io
2100 containers/onboot/004-pillar-onboot/lower/usr/bin/qemu-img
2220 containers/onboot/004-pillar-onboot/lower/usr/bin/qemu-nbd
3064 containers/onboot/004-pillar-onboot/lower/usr/bin/qemu-storage-daemon
usr/lib is 174 files, almost all in the hundreds of KB range. Just the last few slightly larger ones:
$ du -s containers/onboot/004-pillar-onboot/lower/usr/lib/* | sort -n
...
852 containers/onboot/004-pillar-onboot/lower/usr/lib/libisoburn.so.1.111.0
1064 containers/onboot/004-pillar-onboot/lower/usr/lib/libglib-2.0.so.0.7200.4
1084 containers/onboot/004-pillar-onboot/lower/usr/lib/libp11-kit.so.0.3.0
1660 containers/onboot/004-pillar-onboot/lower/usr/lib/libunistring.so.2.2.0
1732 containers/onboot/004-pillar-onboot/lower/usr/lib/libgio-2.0.so.0.7200.4
1892 containers/onboot/004-pillar-onboot/lower/usr/lib/libgnutls.so.30.34.1
2008 containers/onboot/004-pillar-onboot/lower/usr/lib/xtables
4164 containers/onboot/004-pillar-onboot/lower/usr/lib/libzpool.so.5.0.0
and:
$ du -s containers/onboot/004-pillar-onboot/lower/opt/zededa/bin/* | sort -n
...
8 containers/onboot/004-pillar-onboot/lower/opt/zededa/bin/onboot.sh
20 containers/onboot/004-pillar-onboot/lower/opt/zededa/bin/device-steps.sh
392 containers/onboot/004-pillar-onboot/lower/opt/zededa/bin/dnsmasq
5288 containers/onboot/004-pillar-onboot/lower/opt/zededa/bin/fscrypt
62916 containers/onboot/004-pillar-onboot/lower/opt/zededa/bin/zedbox
zedbox is big.
You can keep going for the other big ones, xen-tools and debug.
FWIW, CDI can also be used to provide a common software stack base to all containers, we could have a directory with the whole common Alpine base files and mount it inside all containers through a CDI spec.... but considering our ecosystem, I still don't see any "easy/medium implementation efforts" for this kind of approach....
Yeah, it gets pretty complicated. We don't want solutions that make it even harder to build and run EVE and its components.
In pillar, zedbox is big, but so are usr and lib. Do we need everything there?
Why is onboot container so large? For instance, do we need qemu* in onboot?
And why does onboot need zedbox??
Breakdown of zedbox done with https://github.com/Zxilly/go-size-analyzer
root@ae3e50cac946:/src/zedbox# gsa -f text zedbox | tee gsa.txt
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ zedbox │
├─────────┬─────────────────────────────────────────────────────────────────────────────┬────────┬───────────┤
│ PERCENT │ NAME │ SIZE │ TYPE │
├─────────┼─────────────────────────────────────────────────────────────────────────────┼────────┼───────────┤
│ 14.87% │ .rodata │ 14 MB │ section │
│ 9.40% │ github.com/google/gopacket │ 8.7 MB │ vendor │
│ 7.35% │ .strtab │ 6.8 MB │ section │
│ 7.33% │ github.com/lf-edge/eve/pkg/pillar │ 6.8 MB │ vendor │
│ 6.66% │ .dynstr │ 6.2 MB │ section │
│ 6.21% │ .symtab │ 5.7 MB │ section │
│ 5.77% │ .debug_info │ 5.3 MB │ section │
│ 4.71% │ github.com/aws/aws-sdk-go │ 4.4 MB │ vendor │
│ 4.36% │ .dynsym │ 4.0 MB │ section │
│ 3.77% │ .debug_loc │ 3.5 MB │ section │
│ 3.15% │ .debug_line │ 2.9 MB │ section │
│ 2.21% │ github.com/Azure/azure-storage-blob-go │ 2.0 MB │ vendor │
│ 2.11% │ net │ 2.0 MB │ std │
│ 2.00% │ google.golang.org/protobuf │ 1.8 MB │ vendor │
│ 1.91% │ github.com/containerd/containerd │ 1.8 MB │ vendor │
│ 1.66% │ runtime │ 1.5 MB │ std │
│ 1.59% │ crypto │ 1.5 MB │ std │
│ 1.48% │ github.com/lf-edge/eve-api/go │ 1.4 MB │ vendor │
│ 1.30% │ github.com/gogo/protobuf │ 1.2 MB │ vendor │
│ 1.23% │ google.golang.org/grpc │ 1.1 MB │ vendor │
│ 1.16% │ github.com/robertkrimen/otto │ 1.1 MB │ vendor │
│ 1.13% │ golang.org/x/net │ 1.0 MB │ vendor │
│ 1.11% │ │ 1.0 MB │ generated │
│ 1.01% │ .gnu.hash │ 935 kB │ section │
│ 0.94% │ .debug_ranges │ 872 kB │ section │
│ 0.87% │ google.golang.org/api │ 802 kB │ vendor │
│ 0.76% │ github.com/miekg/dns │ 704 kB │ vendor │
│ 0.75% │ .debug_frame │ 693 kB │ section │
│ 0.73% │ cloud.google.com/go │ 677 kB │ vendor │
│ 0.72% │ github.com/lf-edge/eve-libs │ 671 kB │ vendor │
│ 0.53% │ golang.org/x/crypto │ 494 kB │ vendor │
│ 0.53% │ github.com/docker/docker │ 491 kB │ vendor │
│ 0.51% │ github.com/klauspost/compress │ 470 kB │ vendor │
│ 0.49% │ github.com/xeipuuv/gojsonschema │ 451 kB │ vendor │
│ 0.45% │ github.com/google/go-containerregistry │ 414 kB │ vendor │
│ 0.44% │ github.com/godbus/dbus/v5 │ 409 kB │ vendor │
│ 0.42% │ reflect │ 388 kB │ std │
│ 0.41% │ github.com/vishvananda/netlink │ 383 kB │ vendor │
│ 0.40% │ github.com/prometheus/client_golang │ 374 kB │ vendor │
│ 0.40% │ github.com/prometheus/procfs │ 368 kB │ vendor │
│ 0.38% │ text/template │ 354 kB │ std │
│ 0.38% │ golang.org/x/text │ 353 kB │ vendor │
│ 0.36% │ .gnu.version │ 336 kB │ section │
│ 0.36% │ math │ 330 kB │ std │
│ 0.35% │ github.com/containerd/cgroups │ 321 kB │ vendor │
│ 0.34% │ github.com/google/go-cmp │ 319 kB │ vendor │
│ 0.34% │ go.opentelemetry.io/otel │ 312 kB │ vendor │
│ 0.33% │ gopkg.in/yaml.v3 │ 308 kB │ vendor │
│ 0.29% │ gopkg.in/yaml.v2 │ 271 kB │ vendor │
│ 0.27% │ github.com/pkg/sftp │ 251 kB │ vendor │
│ 0.26% │ github.com/go-playground/validator/v10 │ 244 kB │ vendor │
│ 0.26% │ os │ 242 kB │ std │
│ 0.26% │ github.com/google/s2a-go │ 240 kB │ vendor │
│ 0.25% │ encoding/gob │ 233 kB │ std │
│ 0.25% │ .data │ 229 kB │ section │
│ 0.24% │ github.com/gabriel-vasile/mimetype │ 221 kB │ vendor │
│ 0.22% │ regexp │ 205 kB │ std │
│ 0.20% │ go.opencensus.io │ 188 kB │ vendor │
│ 0.20% │ github.com/shirou/gopsutil │ 184 kB │ vendor │
│ 0.19% │ google.golang.org/genproto │ 178 kB │ vendor │
│ 0.19% │ encoding/json │ 177 kB │ std │
│ 0.18% │ vendor/golang.org/x/text/unicode/norm │ 170 kB │ std │
│ 0.18% │ golang.org/x/oauth2 │ 166 kB │ vendor │
│ 0.18% │ time │ 164 kB │ std │
│ 0.17% │ github.com/andrewd-zededa/go-libzfs │ 159 kB │ vendor │
│ 0.16% │ github.com/ti-mo/conntrack │ 149 kB │ vendor │
│ 0.16% │ internal/profile │ 144 kB │ std │
│ 0.15% │ log │ 143 kB │ std │
│ 0.15% │ github.com/google/go-tpm │ 139 kB │ vendor │
│ 0.15% │ html │ 137 kB │ std │
│ 0.15% │ github.com/packetcap/go-pcap │ 136 kB │ vendor │
│ 0.15% │ tags.cncf.io/container-device-interface │ 135 kB │ vendor │
│ 0.14% │ encoding/xml │ 130 kB │ std │
│ 0.13% │ oras.land/oras-go │ 118 kB │ vendor │
│ 0.13% │ github.com/jmespath/go-jmespath │ 116 kB │ vendor │
│ 0.12% │ .noptrdata │ 112 kB │ section │
│ 0.11% │ fmt │ 105 kB │ std │
│ 0.11% │ syscall │ 99 kB │ std │
│ 0.11% │ archive/tar │ 98 kB │ std │
│ 0.11% │ internal/poll │ 98 kB │ std │
│ 0.10% │ vendor/golang.org/x/net/dns/dnsmessage │ 96 kB │ std │
│ 0.10% │ internal/abi │ 91 kB │ std │
│ 0.10% │ github.com/sirupsen/logrus │ 91 kB │ vendor │
│ 0.10% │ github.com/gorilla/websocket │ 88 kB │ vendor │
│ 0.09% │ github.com/lf-edge/edge-containers │ 85 kB │ vendor │
│ 0.09% │ .typelink │ 84 kB │ section │
│ 0.09% │ github.com/Azure/azure-pipeline-go │ 82 kB │ vendor │
│ 0.09% │ mime │ 82 kB │ std │
│ 0.08% │ compress/flate │ 77 kB │ std │
│ 0.08% │ encoding/asn1 │ 72 kB │ std │
│ 0.08% │ github.com/golang/protobuf │ 72 kB │ vendor │
│ 0.08% │ strconv │ 71 kB │ std │
│ 0.07% │ golang.org/x/sys │ 67 kB │ vendor │
│ 0.07% │ vendor/golang.org/x/net/idna │ 66 kB │ std │
│ 0.07% │ github.com/mdlayher/netlink │ 63 kB │ vendor │
│ 0.07% │ sync │ 61 kB │ std │
│ 0.06% │ github.com/go-chi/chi/v5 │ 57 kB │ vendor │
│ 0.06% │ github.com/go-logr/logr │ 57 kB │ vendor │
│ 0.06% │ internal/reflectlite │ 57 kB │ std │
│ 0.06% │ strings │ 56 kB │ std │
│ 0.06% │ io │ 55 kB │ std │
│ 0.06% │ vendor/golang.org/x/crypto/chacha20poly1305 │ 55 kB │ std │
│ 0.06% │ archive/zip │ 54 kB │ std │
│ 0.06% │ bufio │ 52 kB │ std │
│ 0.06% │ vendor/golang.org/x/crypto/cryptobyte │ 51 kB │ std │
│ 0.06% │ github.com/jaypipes/ghw │ 51 kB │ vendor │
│ 0.05% │ slices │ 46 kB │ std │
│ 0.05% │ github.com/docker/distribution │ 46 kB │ vendor │
│ 0.05% │ encoding/binary │ 46 kB │ std │
│ 0.05% │ github.com/linuxkit/linuxkit/src/cmd/linuxkit │ 45 kB │ vendor │
│ 0.05% │ github.com/mdlayher/socket │ 45 kB │ vendor │
│ 0.05% │ context │ 45 kB │ std │
│ 0.05% │ github.com/googleapis/gax-go/v2 │ 44 kB │ vendor │
│ 0.05% │ flag │ 43 kB │ std │
│ 0.05% │ github.com/distribution/reference │ 42 kB │ vendor │
│ 0.04% │ github.com/prometheus/client_model │ 39 kB │ vendor │
│ 0.04% │ vendor/golang.org/x/net/http2/hpack │ 38 kB │ std │
│ 0.04% │ bytes │ 38 kB │ std │
│ 0.04% │ unicode │ 34 kB │ std │
│ 0.04% │ github.com/facebook/time │ 34 kB │ vendor │
│ 0.04% │ sort │ 33 kB │ std │
│ 0.03% │ github.com/ti-mo/netfilter │ 32 kB │ vendor │
│ 0.03% │ github.com/go-logr/stdr │ 32 kB │ vendor │
│ 0.03% │ go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp │ 31 kB │ vendor │
│ 0.03% │ github.com/tatsushid/go-fastping │ 30 kB │ vendor │
│ 0.03% │ github.com/docker/go-metrics │ 30 kB │ vendor │
│ 0.03% │ path │ 29 kB │ std │
│ 0.03% │ sigs.k8s.io/yaml │ 28 kB │ vendor │
│ 0.03% │ .itablink │ 28 kB │ section │
│ 0.03% │ github.com/grandcat/zeroconf │ 27 kB │ vendor │
│ 0.03% │ github.com/opencontainers/go-digest │ 24 kB │ vendor │
│ 0.03% │ github.com/google/uuid │ 24 kB │ vendor │
│ 0.02% │ vendor/golang.org/x/text/unicode/bidi │ 23 kB │ std │
│ 0.02% │ github.com/pkg/errors │ 23 kB │ vendor │
│ 0.02% │ github.com/jackwakefield/gopac │ 21 kB │ vendor │
│ 0.02% │ github.com/golang-jwt/jwt │ 21 kB │ vendor │
│ 0.02% │ github.com/satori/go.uuid │ 19 kB │ vendor │
│ 0.02% │ github.com/cshari-zededa/eve-tpm2-tools │ 19 kB │ vendor │
│ 0.02% │ github.com/containerd/ttrpc │ 19 kB │ vendor │
│ 0.02% │ github.com/docker/go-events │ 18 kB │ vendor │
│ 0.02% │ github.com/anatol/smart.go │ 18 kB │ vendor │
│ 0.02% │ encoding/base64 │ 18 kB │ std │
│ 0.02% │ .text │ 16 kB │ section │
│ 0.02% │ github.com/leodido/go-urn │ 16 kB │ vendor │
│ 0.02% │ github.com/containerd/fifo │ 16 kB │ vendor │
│ 0.02% │ github.com/fsnotify/fsnotify │ 16 kB │ vendor │
│ 0.02% │ internal/bisect │ 16 kB │ std │
│ 0.02% │ debug/dwarf │ 15 kB │ std │
│ 0.02% │ vendor/golang.org/x/net/http/httpproxy │ 15 kB │ std │
│ 0.02% │ text/tabwriter │ 15 kB │ std │
│ 0.01% │ compress/gzip │ 13 kB │ std │
│ 0.01% │ github.com/googleapis/enterprise-certificate-proxy │ 12 kB │ vendor │
│ 0.01% │ github.com/lf-edge/go-qemu │ 12 kB │ vendor │
│ 0.01% │ hash/crc32 │ 12 kB │ std │
│ 0.01% │ github.com/containerd/typeurl │ 12 kB │ vendor │
│ 0.01% │ github.com/moby/sys/mountinfo │ 11 kB │ vendor │
│ 0.01% │ encoding/csv │ 10 kB │ std │
│ 0.01% │ gopkg.in/sourcemap.v1 │ 10 kB │ vendor │
│ 0.01% │ encoding/base32 │ 10 kB │ std │
│ 0.01% │ main │ 10 kB │ main │
│ 0.01% │ internal/godebug │ 9.9 kB │ std │
│ 0.01% │ container/list │ 9.6 kB │ std │
│ 0.01% │ expvar │ 9.4 kB │ std │
│ 0.01% │ .debug_loclists │ 9.3 kB │ section │
│ 0.01% │ .debug_str │ 9.3 kB │ section │
│ 0.01% │ internal/fmtsort │ 9.1 kB │ std │
│ 0.01% │ golang.org/x/sync │ 9.0 kB │ vendor │
│ 0.01% │ .eh_frame │ 9.0 kB │ section │
│ 0.01% │ encoding/pem │ 8.8 kB │ std │
│ 0.01% │ github.com/tklauser/go-sysconf │ 8.6 kB │ vendor │
│ 0.01% │ internal/bytealg │ 8.4 kB │ std │
│ 0.01% │ github.com/docker/go-connections │ 8.0 kB │ vendor │
│ 0.01% │ github.com/moby/sys/user │ 8.0 kB │ vendor │
│ 0.01% │ vendor/golang.org/x/crypto/chacha20 │ 7.8 kB │ std │
│ 0.01% │ github.com/hashicorp/go-envparse │ 7.7 kB │ vendor │
│ 0.01% │ .go.buildinfo │ 7.3 kB │ section │
│ 0.01% │ github.com/vishvananda/netns │ 6.6 kB │ vendor │
│ 0.01% │ github.com/xeipuuv/gojsonpointer │ 6.4 kB │ vendor │
│ 0.01% │ go/token │ 6.0 kB │ std │
│ 0.01% │ vendor/golang.org/x/sys/cpu │ 5.9 kB │ std │
│ 0.01% │ internal/cpu │ 5.8 kB │ std │
│ 0.01% │ github.com/vbatts/tar-split │ 5.7 kB │ vendor │
│ 0.01% │ golang.org/x/mod │ 5.7 kB │ vendor │
│ 0.01% │ internal/lazyregexp │ 5.6 kB │ std │
│ 0.01% │ errors │ 5.2 kB │ std │
│ 0.01% │ internal/intern │ 5.0 kB │ std │
│ 0.01% │ github.com/containerd/continuity │ 4.9 kB │ vendor │
│ 0.01% │ internal/singleflight │ 4.7 kB │ std │
│ 0.00% │ vendor/golang.org/x/net/http/httpguts │ 4.5 kB │ std │
│ 0.00% │ github.com/golang/groupcache │ 4.4 kB │ vendor │
│ 0.00% │ github.com/docker/go-units │ 4.3 kB │ vendor │
│ 0.00% │ encoding/hex │ 4.2 kB │ std │
│ 0.00% │ github.com/syndtr/gocapability │ 4.0 kB │ vendor │
│ 0.00% │ internal/chacha8rand │ 3.6 kB │ std │
│ 0.00% │ .rela.plt │ 3.6 kB │ section │
│ 0.00% │ .debug_abbrev │ 3.5 kB │ section │
│ 0.00% │ github.com/eriknordmark/ipinfo │ 3.4 kB │ vendor │
│ 0.00% │ github.com/tklauser/numcpus │ 3.4 kB │ vendor │
│ 0.00% │ container/ring │ 3.3 kB │ std │
│ 0.00% │ go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc │ 3.2 kB │ vendor │
│ 0.00% │ github.com/cenkalti/backoff │ 3.2 kB │ vendor │
│ 0.00% │ vendor/golang.org/x/crypto/internal/poly1305 │ 2.8 kB │ std │
│ 0.00% │ internal/syscall/unix │ 2.7 kB │ std │
│ 0.00% │ github.com/xeipuuv/gojsonreference │ 2.7 kB │ vendor │
│ 0.00% │ github.com/kr/fs │ 2.7 kB │ vendor │
│ 0.00% │ github.com/moby/locker │ 2.6 kB │ vendor │
│ 0.00% │ github.com/eshard/uevent │ 2.5 kB │ vendor │
│ 0.00% │ testing │ 2.5 kB │ std │
│ 0.00% │ github.com/cespare/xxhash/v2 │ 2.4 kB │ vendor │
│ 0.00% │ .plt │ 2.4 kB │ section │
│ 0.00% │ vendor/golang.org/x/text/secure/bidirule │ 2.4 kB │ std │
│ 0.00% │ github.com/digitalocean/go-libvirt │ 2.3 kB │ vendor │
│ 0.00% │ internal/testlog │ 2.3 kB │ std │
│ 0.00% │ vendor/golang.org/x/crypto/hkdf │ 2.2 kB │ std │
│ 0.00% │ github.com/prometheus/common │ 2.1 kB │ vendor │
│ 0.00% │ github.com/containerd/log │ 2.0 kB │ vendor │
│ 0.00% │ github.com/opencontainers/runtime-tools │ 2.0 kB │ vendor │
│ 0.00% │ .eh_frame_hdr │ 1.9 kB │ section │
│ 0.00% │ github.com/opencontainers/selinux │ 1.7 kB │ vendor │
│ 0.00% │ github.com/lithammer/shortuuid/v4 │ 1.6 kB │ vendor │
│ 0.00% │ github.com/opencontainers/image-spec │ 1.4 kB │ vendor │
│ 0.00% │ github.com/moby/sys/signal │ 1.3 kB │ vendor │
│ 0.00% │ .got │ 1.3 kB │ section │
│ 0.00% │ github.com/go-playground/locales │ 1.3 kB │ vendor │
│ 0.00% │ github.com/golang-design/lockfree │ 1.2 kB │ vendor │
│ 0.00% │ CGO C11 │ 1.2 kB │ cgo │
│ 0.00% │ internal/saferio │ 1.1 kB │ std │
│ 0.00% │ internal/itoa │ 777 B │ std │
│ 0.00% │ .debug_line_str │ 659 B │ section │
│ 0.00% │ .dynamic │ 528 B │ section │
│ 0.00% │ .shstrtab │ 527 B │ section │
│ 0.00% │ github.com/opencontainers/runtime-spec │ 450 B │ vendor │
│ 0.00% │ .debug_aranges │ 426 B │ section │
│ 0.00% │ regexp.(*onePassInst).regexp/syntax │ 411 B │ vendor │
│ 0.00% │ github.com/ghodss/yaml │ 408 B │ vendor │
│ 0.00% │ .debug_rnglists │ 379 B │ section │
│ 0.00% │ github.com/containerd/stargz-snapshotter/estargz │ 370 B │ vendor │
│ 0.00% │ github.com/jaypipes/pcidb │ 306 B │ vendor │
│ 0.00% │ database/sql/driver │ 278 B │ std │
│ 0.00% │ github.com/docker/cli │ 273 B │ vendor │
│ 0.00% │ github.com/josharian/native │ 147 B │ vendor │
│ 0.00% │ .gnu.version_r │ 128 B │ section │
│ 0.00% │ .rela.dyn │ 120 B │ section │
│ 0.00% │ .note.go.buildid │ 100 B │ section │
│ 0.00% │ .debug_gdb_scripts │ 46 B │ section │
│ 0.00% │ .comment │ 38 B │ section │
│ 0.00% │ .note.gnu.build-id │ 36 B │ section │
│ 0.00% │ .note.ABI-tag │ 32 B │ section │
│ 0.00% │ .note.gnu.property │ 32 B │ section │
│ 0.00% │ .interp │ 28 B │ section │
│ 0.00% │ .init │ 27 B │ section │
│ 0.00% │ .fini │ 13 B │ section │
│ 0.00% │ .fini_array │ 8 B │ section │
│ 0.00% │ .init_array │ 8 B │ section │
│ 0.00% │ github.com/docker/docker-credential-helpers │ 0 B │ vendor │
│ 0.00% │ github.com/go-playground/universal-translator │ 0 B │ vendor │
│ 0.00% │ github.com/mattn/go-ieproxy │ 0 B │ vendor │
│ 0.00% │ github.com/gorilla/mux │ 0 B │ vendor │
│ 0.00% │ github.com/beorn7/perks │ 0 B │ vendor │
│ 0.00% │ github.com/coreos/go-systemd/v22 │ 0 B │ vendor │
│ 0.00% │ CGO Language(32769) │ 0 B │ cgo │
│ 0.00% │ github.com/felixge/httpsnoop │ 0 B │ vendor │
│ 0.00% │ golang.org/x/time │ 0 B │ vendor │
│ 0.00% │ github.com/mitchellh/go-homedir │ 0 B │ vendor │
│ 0.00% │ github.com/lf-edge/eve/pkg/kube/cnirpc │ 0 B │ vendor │
├─────────┼─────────────────────────────────────────────────────────────────────────────┼────────┼───────────┤
│ 100.00% │ Known │ 92 MB │ │
│ 100% │ Total │ 92 MB │ │
└─────────┴─────────────────────────────────────────────────────────────────────────────┴────────┴───────────┘
Have we tried building pillar while removing the reflect package? It is included in 27 files (will paste here below). One of the interesting pieces of golang optimization is that in general, it strips out everything that is unused, e.g. structs that are not referenced, etc. If you use reflect package, because it does not know and cannot know in advance what you will referenced, it keeps everything. We have seen cases where it made a compiled binary 30-50% bigger in size.
$ grep -r -w '"reflect"' * | grep -v vendor
cipher/cipher.go: "reflect"
cmd/usbmanager/subscriptions.go: "reflect"
cmd/tpmmgr/tpmmgr.go: "reflect"
cmd/zedagent/attesttask.go: "reflect"
cmd/domainmgr/domainmgr_test.go: "reflect"
cmd/zedmanager/handlezedrouter.go: "reflect"
containerd/oci_test.go: "reflect"
dpcmanager/dpc_test.go: "reflect"
dpcreconciler/genericitems/resolvconf.go: "reflect"
dpcreconciler/linuxitems/wlan.go: "reflect"
dpcreconciler/linuxitems/bond.go: "reflect"
evetpm/tpm_test.go: "reflect"
hypervisor/hypervisor_test.go: "reflect"
netmonitor/mock.go: "reflect"
objtonum/map.go: "reflect"
pubsub/large_test.go: "reflect"
pubsub/subscribe.go: "reflect"
pubsub/util.go: "reflect"
pubsub/pubsub.go: "reflect"
pubsub/publish.go: "reflect"
types/dns.go: "reflect"
types/assignableadapters.go: "reflect"
types/errortime_test.go: "reflect"
types/dpc.go: "reflect"
types/errortime.go: "reflect"
utils/deserialize_test.go: "reflect"
utils/deserialize.go: "reflect"
Actually, that would be an interesting strategy. We should check for reflect in all of our packages. I would start with pillar and see what a difference it makes, but if we can get rid of it and makes a big difference, we can add "reflect alerts" to CI.
@mikem-zed @rene @milan-zededa any thoughts on this comment, especially the part about halfway through it:
lib/firmware is 153MB. 202 lines in there, not going to post it all here. But most are .fw or .ucode files of 1.5-2MB, with a few really big ones:
$ du -s lib/firmware/* | sort -n ... 4236 lib/firmware/intel 6232 lib/firmware/cypress 8876 lib/firmware/display-t234-dce.bin 8996 lib/firmware/mrvl 9452 lib/firmware/ti-connectivity 13732 lib/firmware/brcm 19960 lib/firmware/ath10k Do we need all of those on every build?
Actually, that would be an interesting strategy. We should check for
reflectin all of our packages. I would start with pillar and see what a difference it makes, but if we can get rid of it and makes a big difference, we can add "reflect alerts" to CI.
Doesn't json marshalling depend on reflect?
Good, that I just added more use of it ... (https://github.com/lf-edge/eve/pull/4295 )
Maybe I'll try next week to build pillar without any reflect and see what happens.
Not sure about json marshalling. I don't think it depends on it, but not sure.
Maybe I'll try next week to build pillar without any reflect and see what happens.
Not sure about json marshalling. I don't think it depends on it, but not sure.
287 func (e *encodeState) marshal(v any, opts encOpts) (err error) {
288 defer func() {
289 if r := recover(); r != nil {
290 if je, ok := r.(jsonError); ok {
291 err = je.error
292 } else {
293 panic(r)
294 }
295 }
296 }()
297 e.reflectValue(reflect.ValueOf(v), opts)
298 return nil
299 }
json/encode.go
For TinyGo they do it f.e. like this: https://github.com/json-iterator/tinygo
I just checked the source code for golang/go, same thing. I wish I could remember where it was I had the conversation about reflect and removing unused, and if it is all uses of reflect, or only certain ones.
I spent quite some time digging more deeply. There is an excellent presentation - and pretty good tool - given by the author of whydeadcode (links to slides and talk at the repo).
Short form, it is not all usage of reflect, but of:
- reflect.Value.MethodByName()
- reflect.Value.Method()
- reflect.Type.MethodByName()
- reflect.Type.Method()
whydeadcode does a pretty good job reading the output of go build -ldflags="-dumpdep" and showing the dependencies. The problem is they are everywhere.
Here is one example from our own code, pkg/pillar/zedpac/zedpac.go:
package zedpac
import (
"fmt"
"github.com/jackwakefield/gopac"
)
func Find_proxy_sync(pac, url, host string) (string, error) {
parser := new(gopac.Parser)
if err := parser.ParseBytes([]byte(pac)); err != nil {
return "", fmt.Errorf("invalid proxy auto-configuration file: %v", err)
}
return parser.FindProxy(url, host)
}
And the dependency chain:
reflect.Value.MethodByName reachable from:
github.com/robertkrimen/otto.goArrayGetOwnProperty
github.com/robertkrimen/otto.(*_runtime).convertCallParameter
github.com/robertkrimen/otto.(*_runtime).toValue.func1
github.com/robertkrimen/otto.(*_runtime).toValue
github.com/robertkrimen/otto.(*_runtime).toValueArray
github.com/robertkrimen/otto.(*_runtime).cmpl_evaluate_nodeCallExpression
github.com/robertkrimen/otto.Otto.Call.func2
github.com/robertkrimen/otto.Otto.Call
github.com/jackwakefield/gopac.(*runtime).findProxyForURL
github.com/jackwakefield/gopac.(*Parser).FindProxy
github.com/lf-edge/eve/pkg/pillar/zedpac.Find_proxy_sync
github.com/lf-edge/eve/pkg/pillar/zedcloud.LookupProxy
github.com/lf-edge/eve/pkg/pillar/zedcloud.SendOnIntf
github.com/lf-edge/eve/pkg/pillar/zedcloud.SendOnAllIntf
github.com/lf-edge/eve/pkg/pillar/cmd/client.myPost
github.com/lf-edge/eve/pkg/pillar/cmd/client.selfRegister
github.com/lf-edge/eve/pkg/pillar/cmd/client.Run.func1
github.com/lf-edge/eve/pkg/pillar/cmd/client.Run
github.com/lf-edge/eve/pkg/pillar/cmd/client.Run·f
main.init
main..inittask
runtime.main
runtime.mainPC
runtime.rt0_go
main
_
That should be replaceable. I tried building without that dependency - not replacing the functionality, just to test. The problem is that it is not the only place we use those methods, and here it gets harder:
reflect.Value.MethodByName reachable from:
text/template.(*state).evalField
text/template.(*state).evalFieldChain
text/template.(*state).evalFieldNode
text/template.(*state).evalCommand
text/template.(*state).evalPipeline
text/template.(*state).walk
text/template.(*Template).execute
html/template.(*Template).Execute
golang.org/x/net/trace.RenderEvents
golang.org/x/net/trace.Events
golang.org/x/net/trace.Events·f
golang.org/x/net/trace.init.0
golang.org/x/net/trace..inittask
google.golang.org/grpc..inittask
github.com/containerd/containerd..inittask
github.com/lf-edge/eve/pkg/pillar/agentlog..inittask
main..inittask
runtime.main
runtime.mainPC
runtime.rt0_go
main
_
That is a long chain, but not only containerd, everything that uses text/template or google.golang.org/grpc or golang.org/x/net/trace.
There is a huge amount of installed code which depends on these reflect methods, directly or indirectly.
I suspect if we could, the size of the binary would drop a lot, although I couldn't begin to guess how much. But it is far bigger than us. It even goes into core classes. And I checked latest mainline branch for text/template, it still is there.
@deitch
If I comment out the content of Find_proxy_sync and return "", nil then the binary size drops from 88MB to 85MB (on Ubuntu). So I guess at max we can save 3MB with solely this change.
New output starts with:
github.com/google/go-cmp/cmp/internal/value.appendTypeName reachable from:
github.com/google/go-cmp/cmp.formatOptions.FormatType
github.com/google/go-cmp/cmp.formatOptions.FormatDiff
github.com/google/go-cmp/cmp.(*defaultReporter).String
github.com/google/go-cmp/cmp.Diff
github.com/lf-edge/eve/pkg/pillar/cmd/zedagent.parseConfigItems
github.com/lf-edge/eve/pkg/pillar/cmd/zedagent.parseConfig
github.com/lf-edge/eve/pkg/pillar/cmd/zedagent.inhaleDeviceConfig
github.com/lf-edge/eve/pkg/pillar/cmd/zedagent.maybeLoadBootstrapConfig
github.com/lf-edge/eve/pkg/pillar/cmd/zedagent.Run
github.com/lf-edge/eve/pkg/pillar/cmd/zedagent.Run·f
main.map.init.0
main.entrypoints
main.main
runtime.main_main·f
runtime.main
runtime.mainPC
runtime.rt0_go
main
_
So I guess at max we can save 3MB with solely this change
Correct, but that isn't because of any general deadcode detection. It is because of anything directly referenced by what we filtered out, and its dependencies. There might be some deadcode in there as well, that only was included because of the reflect issues, but those are minor.
To make a real difference, we would need to get rid of the deadcode inclusion-causing calls (reflect.Method and MethoadByName) from everywhere.