Build S3 and other stores as plugins

[roberth]

Is your feature request related to a problem? Please describe.

"Nix itself depends on aws-sdk-cpp"

It kind of doesn't. It's an optional dependency, and many users don't need it. Some might even feel it's tainted because they don't understand S3 is just an HTTP interface with multiple good open source server implementations, and this is just an HTTP client library. Nonetheless, we can reap benefits from modularizing internally, and other stores could follow the same pattern:

• Validate the plugin system and lower the barrier for using it for this purpose. It probably has a few kinks we'll iron out in the process.
• Finer grained build leads to shorter incremental build times. E.g. the functional tests VM test doesn't need it.
• Let users enable/disable without having to recompile (only rebuilding the final wrapper package)

Describe the solution you'd like

• Build this store implementation in a separate derivation and meson subproject. Combine it into the Nix CLI at a late stage.
• Keep distributing the S3 store in the default package and NixOS images. Its removal would be unnecessarily disruptive. It could perhaps be removed from the default NixOS configuration, but it's probably best to enable it in the installer images.

Describe alternatives you've considered

Additional context

If we extend the C API to handle this, we could turn it into a separately distributed package, but figuring out a stable interface for this implies a level of maturity that I don't think the store layer has reached yet. This public/stable API would follow all quirks we may currently have, and fixing those would then involve writing and maintaining compatibility logic, so I think we shouldn't do this yet.

Priorities

Add :+1: to issues you find important.

[edolstra]

The S3 store was originally part of Hydra before we moved it into Nix, so it should be straightforward to move it into its own derivation. No need to involve the C API in this.

One issue is that the plugin mechanism is incompatible with nix-static, so we'd also need a way to link it statically.

However, it's unclear if it's really worth it, since we have much bigger dependencies than aws-sdk-cpp.

[arianvp]

Do we? aws-sdk-cpp is the second largest dependency of nix in terms of closure size. Only boost is bigger.

arian@Arians-MacBook-Pro ~ % nix path-info ./result/  -rSh | sort -hk2 
/nix/store/q2rir7qz4cav2czkyrn76pcl1lsha6wf-apple-lib-libDER                       	   5.5K
/nix/store/1ls4bj6vhww2ma5pvazwxfd5g3y2g150-editline-1.17.1                        	  98.2K
/nix/store/7gmnfg07ysqb4gx0c074ypm8hwvj8lqi-bzip2-1.0.8                            	 105.6K
/nix/store/0ib2jb462dh3mg5yrv77g4zv7x5m2gy3-zlib-1.3.1                             	 123.5K
/nix/store/99bkrjywb7pwphnvcigh8rbvxx892q41-nghttp2-1.62.1-lib                     	 244.5K
/nix/store/q68qmn066kv2jwnik5hjap76pcmibagv-xz-5.6.2                               	 248.6K
/nix/store/1qxv5lvyha8ygdhr5bn0hxmm68bnnp8r-libobjc-11.0.0                         	 262.4K
/nix/store/1vz22mb6bz64qlbwa3b2lhl1ijnyjg96-lowdown-1.1.0-lib                      	 281.7K
/nix/store/prisp02igbgddb4f45g46mv8bxqp9fjq-publicsuffix-list-0-unstable-2024-01-07	 302.5K
/nix/store/dwh3bhajbzppdj29sd7y32mkizysm247-libsodium-1.0.20                       	 362.5K
/nix/store/yjkmk4kz8r38lrxjhd519nii83lb5a1x-nix-2.18.5-man                         	 557.2K
/nix/store/0vpnvg84xi8382213lf71m4rgky3ihk1-nlohmann_json-3.11.3                   	 916.2K
/nix/store/8dphxn1ayqxmd4kdrzsk3i1z18q8j41k-apple-framework-CoreFoundation-11.0.0  	 942.5K
/nix/store/2sml21d45bama47dw9gw3v9ccn9150nz-brotli-1.1.0-lib                       	 968.5K
/nix/store/j987d55qyicdw484scba3zqjhv3qar8i-aws-c-common-0.9.23                    	   1.0M
/nix/store/hw2rk784b20j9239ddi9jjg81ixncaf3-aws-c-compression-0.2.18               	   1.1M
/nix/store/wlds54ajqyyfi31mafpfflw8bqg7yka6-aws-checksums-0.1.18                   	   1.1M
/nix/store/xn952idabrilsvjqm6c8y3p224vdh4nx-aws-c-sdkutils-0.1.16                  	   1.2M
/nix/store/kpq03ylpiya2vbzja2313f1nnvg55sy9-sqlite-3.46.0                          	   1.8M
/nix/store/fyjay93q3dq2hx3dhx7zhr8kyjnkh9m8-bash-5.2p26                            	   2.7M
/nix/store/rhbd53iilv2kjfkxihrqh1yf1r4q5m7y-apple-framework-IOKit-11.0.0           	   3.5M
/nix/store/xfva28w11lyjdl9az6cn9gma550vb8yf-libcxx-16.0.6                          	   3.6M
/nix/store/sn7ximy8sin3nah9958xs3dyq7czg4cj-boehm-gc-8.2.6                         	   4.0M
/nix/store/zyrq8llafvxs3nlwpf9fmk4qqm9gw06s-openssl-3.0.14                         	   4.4M
/nix/store/vn0k5m0k3yi6aji8pbppqxxc7slr348y-zstd-1.5.6                             	   4.5M
/nix/store/8aihf085i13q6akrjwy126c44dcdrwjp-libssh2-1.11.0                         	   4.9M
/nix/store/69gs5xqvq1jyr5871mq0wi5i7fish4fr-apple-framework-Security-11.0.0        	   5.0M
/nix/store/x81ij8c0c5rz35j3hijscnqzxlvjikgr-libkrb5-1.21.3                         	   5.0M
/nix/store/pjlaiddxgpx9nqm7h0vhqjs8hhzwfjjr-aws-c-cal-0.6.15                       	   6.1M
/nix/store/kazpdi4cwwdgxvp1dh7fal4g90fpkqy5-aws-c-io-0.14.9                        	   6.7M
/nix/store/y7jvbmrza024azcc0mffdkqnpj5c51ci-aws-c-event-stream-0.4.2               	   7.0M
/nix/store/7yai4mc74vqq9db0g7vkaz1yqqxlvisp-aws-c-http-0.8.2                       	   7.3M
/nix/store/6npnx03wq6yfgb8rg5aq6rj8mg6qlijg-aws-c-mqtt-0.10.4                      	   7.7M
/nix/store/4ghmfr1nwdfp75w2kk95ka38vqxqf7xz-aws-c-auth-0.7.22                      	   7.8M
/nix/store/xnlhg6zr9z72wm45fzzp1zc99lps2fp9-aws-c-s3-0.6.0                         	   8.2M
/nix/store/kbjw75mf4dmlk6nbfvc9s2n9jmvp3fbc-aws-crt-cpp-0.26.12                    	  13.0M
/nix/store/7br16abanr99mgg7dahfmvby28ad5qj3-icu4c-74.2                             	  38.7M
/nix/store/zrp98d2dm0zix7ghj1v0bz7hp7a0h134-libiconv-99                            	  43.7M
/nix/store/5bim9dc64yzjb6b26mzajpbn79vbj9kw-libxml2-2.13.2                         	  45.1M
/nix/store/jhdqkbp2bmrv0mjxfhlmbq0adqz5kql9-libunistring-1.2                       	  45.7M
/nix/store/5r6r7648xvc1yddksq8627rz7zpan5pw-libidn2-2.3.7                          	  45.9M
/nix/store/76g6gla8rnj9nw8jd7iz42i5qwdzk35r-libarchive-3.7.4-lib                   	  51.0M
/nix/store/4k323haw2b7vj59j6rbray5qqish1bd7-gettext-0.21.1                         	  59.8M
/nix/store/qfqbfx6lkii9l6i49m6y3q9s3mmdnglj-libpsl-0.21.5                          	  62.4M
/nix/store/9v7hc5hm591539hlka47dj8ibjnbv0r2-curl-8.8.0                             	  72.4M
/nix/store/8c6gj1jvsfjy3q1aybbz766l2h2danzp-aws-sdk-cpp-1.11.336                   	  87.3M
/nix/store/8j82bflcgf4kdr2f1b0dsn30w3zh0w23-boost-1.81.0                           	  95.6M
/nix/store/ma0p24nzjcylflyn4bz4wj8kf28a7ida-nix-2.18.5                             	 155.3M

[roberth]

Closure sizes don't add and subtract normally due to shared dependencies, so it's often not as bad as it looks. Rebuilding Nix 2.24 without the aws-sdk-cpp dependency gives me a 13% decrease in closure size. (98M to 86M) That's not insignificant though.

Boost otoh is only 160k by itself and brings in no dependencies we didn't already have.

This is on x86_64-linux, and 2.18 is getting a bit old by now, but the overall picture is similar.

[Ericson2314]

I like separate library, non-plugin, as a way to still support statically-linked Nix, but moving us in the right direction.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/why-does-the-nix-package-manager-depend-on-aws-libraries/54064/2

[abitrolly]

Is it possible to load libaws-cpp-sdk* libs dynamically?

Arch Linux folks say nix can't do this https://bugs.archlinux.org/task/78424

Is it still the case?

[lovesegfault]

I think #13752 improved this:

❯ nix path-info ./result/ -rSh | grep aws
/nix/store/bq29zvl6qfr9jk51cfp0idnbfbfc5hg0-aws-c-common-0.10.3                           32.4 MiB
/nix/store/6ksqfff4i5qy6szyfq9frsxrbfnpp347-aws-c-compression-0.3.0                       32.5 MiB
/nix/store/wvf82idmmcxhyw7cs3a8s0qnc8bpfsa5-aws-c-cal-0.8.0                               40.4 MiB
/nix/store/k6xkcvw4vxdcp8zvn7kzyy520lybi1ml-aws-c-io-0.15.3                               42.4 MiB
/nix/store/hk6h8n1s4qpi3zd0z1q1hk0pajmqgb79-aws-c-http-0.9.2                              43.0 MiB
/nix/store/jsk4xkcmwdicf759xsvxvfn87p27dq7h-aws-checksums-0.2.2                           32.5 MiB
/nix/store/s06qynsgmhfn7jk8b2756cvy29qflya5-aws-c-sdkutils-0.2.1                          32.6 MiB
/nix/store/w7r70amih1yk6myqs69v0p0sigr4dgkv-aws-c-auth-0.8.1                              43.5 MiB
/nix/store/d116kc4jas5y0cbys0m4lvbj8093fiqc-aws-c-s3-0.7.1                                44.0 MiB
/nix/store/vyi7w7n6q739j59fgplpb9amxy7xgrb7-aws-c-mqtt-0.11.0                             43.5 MiB
/nix/store/ylyy48mib1bkh4qy71f1xpvmnj8xl5nl-aws-c-event-stream-0.5.0                      42.6 MiB
/nix/store/a769vpvqkg22dzww2kwxcl5601q8d095-aws-crt-cpp-0.29.4                            55.3 MiB

❯ nix-closure-size /nix/store/a769vpvqkg22dzww2kwxcl5601q8d095-aws-crt-cpp-0.29.4
56Mi