fs-repo-migrations

fs-repo-migrations copied to clipboard

IPFS nodes may automatically download these binaries off the internet during migration. They should not have dependencies on any dynamic libraries, not even glibc, not even ld-linux.so. Some particular distros are not able to execute the binaries as-is.

Still not taken care of for amd64. arm64 (https://ipfs.io/ipfs/QmdaCHYBDHEhXCMoynH5UcohEay6m1XayZCcxWZzKAHNVN/fs-repo-11-to-12/v1.0.2/fs-repo-11-to-12_v1.0.2_linux-arm64.tar.gz) is a static executable already for some reason.

$ ldd */fs-repo-11-to-12/fs-repo-11-to-12
amd64/fs-repo-11-to-12/fs-repo-11-to-12:
	linux-vdso.so.1 (0x00007ffc0af00000)
	libdl.so.2 => /nix/store/lyl6nysc3i3aqhj6shizjgj0ibnf1pvg-glibc-2.34-210/lib/libdl.so.2 (0x00007fbaa6661000)
	libpthread.so.0 => /nix/store/lyl6nysc3i3aqhj6shizjgj0ibnf1pvg-glibc-2.34-210/lib/libpthread.so.0 (0x00007fbaa665c000)
	libc.so.6 => /nix/store/lyl6nysc3i3aqhj6shizjgj0ibnf1pvg-glibc-2.34-210/lib/libc.so.6 (0x00007fbaa645e000)
	/lib64/ld-linux-x86-64.so.2 => /nix/store/lyl6nysc3i3aqhj6shizjgj0ibnf1pvg-glibc-2.34-210/lib64/ld-linux-x86-64.so.2 (0x00007fbaa6668000)
arm64/fs-repo-11-to-12/fs-repo-11-to-12:
	not a dynamic executable

It sounds like you're looking for a static build here rather than not having CGO enabled. If you could investigate the tradeoffs in things like binary size that would be great and we could consider building the migrations statically. However, there are reasons we might find ourselves needing CGO for the migrations (beyond just for plugins) so committing to keeping migrations CGO free does not seem to be something the kubo maintainers can currently commit to.

Note: You can load migrations custom built migrations locally using env vars (e.g. https://github.com/ipfs/kubo/blob/master/docs/environment-variables.md#ipfs_dist_path), or add the binary to your PATH even when running kubo and having it autorun the migration.

Some interesting results

x86_64

go version go1.18.4 linux/amd64
gcc (GCC) 11.3.0

make -j12

   2.11 MB fs-repo-8-to-9/fs-repo-8-to-9
   2.56 MB fs-repo-0-to-1/fs-repo-0-to-1
   2.60 MB fs-repo-7-to-8/fs-repo-7-to-8
   2.70 MB fs-repo-5-to-6/fs-repo-5-to-6
   2.72 MB fs-repo-4-to-5/fs-repo-4-to-5
   2.87 MB fs-repo-9-to-10/fs-repo-9-to-10
   3.67 MB fs-repo-1-to-2/fs-repo-1-to-2
   4.38 MB fs-repo-3-to-4/fs-repo-3-to-4
   4.98 MB fs-repo-2-to-3/fs-repo-2-to-3
  11.72 MB fs-repo-migrations/fs-repo-migrations
  15.20 MB fs-repo-6-to-7/fs-repo-6-to-7
  51.84 MB fs-repo-11-to-12/fs-repo-11-to-12
  51.86 MB fs-repo-10-to-11/fs-repo-10-to-11
 159.24 MB total

make -j12 CGO_ENABLED=0

   2.11 MB fs-repo-8-to-9/fs-repo-8-to-9
   2.56 MB fs-repo-0-to-1/fs-repo-0-to-1
   2.60 MB fs-repo-7-to-8/fs-repo-7-to-8
   2.70 MB fs-repo-5-to-6/fs-repo-5-to-6
   2.72 MB fs-repo-4-to-5/fs-repo-4-to-5
   2.87 MB fs-repo-9-to-10/fs-repo-9-to-10
   3.67 MB fs-repo-1-to-2/fs-repo-1-to-2
   4.37 MB fs-repo-3-to-4/fs-repo-3-to-4
   4.96 MB fs-repo-2-to-3/fs-repo-2-to-3
  11.67 MB fs-repo-migrations/fs-repo-migrations
  15.15 MB fs-repo-6-to-7/fs-repo-6-to-7
  34.74 MB fs-repo-10-to-11/fs-repo-10-to-11
  35.65 MB fs-repo-11-to-12/fs-repo-11-to-12
 125.78 MB total

aarch64

go version go1.18.4 linux/arm64
gcc (GCC) 11.3.0

make -j4

   2.17 MB fs-repo-8-to-9/fs-repo-8-to-9
   2.56 MB fs-repo-0-to-1/fs-repo-0-to-1
   2.64 MB fs-repo-7-to-8/fs-repo-7-to-8
   2.67 MB fs-repo-5-to-6/fs-repo-5-to-6
   2.74 MB fs-repo-4-to-5/fs-repo-4-to-5
   2.85 MB fs-repo-9-to-10/fs-repo-9-to-10
   3.62 MB fs-repo-1-to-2/fs-repo-1-to-2
   4.24 MB fs-repo-3-to-4/fs-repo-3-to-4
   4.90 MB fs-repo-2-to-3/fs-repo-2-to-3
  11.29 MB fs-repo-migrations/fs-repo-migrations
  14.76 MB fs-repo-6-to-7/fs-repo-6-to-7
  53.68 MB fs-repo-11-to-12/fs-repo-11-to-12
  53.74 MB fs-repo-10-to-11/fs-repo-10-to-11
 161.86 MB total

make -j4 CGO_ENABLED=0

   2.17 MB fs-repo-8-to-9/fs-repo-8-to-9
   2.56 MB fs-repo-0-to-1/fs-repo-0-to-1
   2.64 MB fs-repo-7-to-8/fs-repo-7-to-8
   2.67 MB fs-repo-5-to-6/fs-repo-5-to-6
   2.74 MB fs-repo-4-to-5/fs-repo-4-to-5
   2.85 MB fs-repo-9-to-10/fs-repo-9-to-10
   3.62 MB fs-repo-1-to-2/fs-repo-1-to-2
   4.24 MB fs-repo-3-to-4/fs-repo-3-to-4
   4.83 MB fs-repo-2-to-3/fs-repo-2-to-3
  11.27 MB fs-repo-migrations/fs-repo-migrations
  14.74 MB fs-repo-6-to-7/fs-repo-6-to-7
  33.80 MB fs-repo-10-to-11/fs-repo-10-to-11
  34.66 MB fs-repo-11-to-12/fs-repo-11-to-12
 122.80 MB total