Split into multiple Go modules?

[glasser]

Depending on the lego Go module ends up causing your project to depend on dozens of other modules which implement the APIs for different DNS providers (primarily). While their packages won't bloat your binary size if unused, it does cause your dependency graph to be larger, and if you do use one of those libraries it can affect what version of the library gets used in your project even if you don't use the relevant piece. (eg, when we added lego to our project it affected our AWS library version even though we aren't using either AWS provider.)

Would it be possible to split off providers into its own Go module? cmd would also need to be its own module because it depends on providers. That way, projects consuming lego as a library that implement their own provider or use one of the dependency-free built in providers won't end up with the 25+ provider libraries in their project.

[ldez]

Hello,

for now, we don't plan to do that, but we will keep that in mind.

[mholt]

I would be willing to help move the DNS providers to a separate repo where each one can be their own module.

EDIT: I created github.com/libdns where each repository is a DNS provider. These packages are lightweight, yet capable of more than just setting DNS records for ACME challenges; they can Append, Set, Delete, and Get records of any kind, generally!

---

Multi-module repositories are kind of tricky but once it's automated it shouldn't be too bad.

Caddy binaries are currently about 20 MB but without the DNS providers it would go down to around 12 MB, so they currently make up almost 50% of the size of the binary and the compile time...

[256dpi]

Another option would be to extract the "core" libraries into a separate core repository/module. This would be backwards compatible for users that already import this module. And then users that only require the library parts without commands and DNS providers can use the core module with a minimal set of dependencies.

[kroppt]

I considered using this as an alternative to using certbot. I found it annoying that installing certbot on Gentoo would also install 29 Python libraries. I went to install the lego client from source, and there were 2GB of dependencies. I would use the source packages and Go to cut that down if possible, but it's not.

[ldez]

2 GB of source's dependencies? It's 25,1 MB.

The dependencies in Go are managed really differently than Python but go modules system have some side effects.

[ldez]

To explain the lego context of the library if it is divided:

• CLI depends on core and providers
• providers depends on core

[kroppt]

~ > rm -rf go
~ > GO111MODULE=on go get -u github.com/go-acme/lego/v3/cmd/lego
~ > du -sh go/pkg/mod
2.0G    go/pkg/mod

This actually failed to compile. Nevertheless...

[ldez]

The go modules have some side effects because the real size of the source dependencies is lower:

$ go mod vendor
$ du -sh vendor/
31M     vendor/

[anderspitman]

@mholt it looks like Caddy/certmagic have settled on libnds for this, correct?

[mholt]

@anderspitman That's correct. They are general-purpose DNS provider packages that can also be used to set and delete TXT records for ACME challenges.

[ozkar99]

+1 on this, any interest in revisiting this?

This is a real problem.

I have a different reason for this: I am actually having a dependency mismatch with https://github.com/StackExchange/dnscontrol

Moving the providers (or having each provider) into their own module would solve this.

[EX0l0N]

I want to express my support for this request.

I've been using lego for a long time now and I've always liked it for being a small drop in solution to get LetsEncrypt certificates for my systems without pulling any other dependencies (like dozens of otherwise unused python packages) to small installations.

Unfortunately this use-case is eroding with the addition of every single new provider. For sure being a single statically linked go binary lego is still selfcontained, but due to its mere size it became an unbearable bloat for small systems itself.

For me having the credentials to modify my DNS at a public facing system is something I would never do. I will always rely on http and tls providers.

If adding more and more providers and therefore adding more and more SDKs of various platforms is the main interest of this community, that's fine with me, but at some point it should be optional! Even for users that use the DNS providers having support for hundreds of DNS providers in a one size fits all fashion is probably not what they want. Most of them will be fine with just a few providers (if they were selectable) and size really became an issue here.

I was running a go get at a new arm64 system lately and it started to pull literally gigabytes of SDKs ending up with an 80MB binary. If I had tried this on my Raspi B the build time would be probably 1 hour by now. (Yes I'm aware there are pre built binaries. That's not the point.)

When lego added support for AWS Route53 I was like did they really just pull the whole AWS SDK just to support a DNS challenge? Back then the binary was still below 20MB, something which I'd consider acceptable even for small armv7 systems.

A binary twice the size of the systems kernel should be something to worry about, even if you're a friend of adding more and more DNS providers. I don't think lego should carry on like this.

[ldez]

As you can see, the size of the binary is stable since v2.7.0 (Jul 21, 2019) Since v2.7.0, we added 25 DNS providers, as you can also see, it doesn't impact the binary size.

[EX0l0N]

Then I must be doing something totally wrong. I can't find any build approach in the docs, which would give me anything close to that size.

Either one of the go get and the make build approaches do create a binary of ~50MB. (for linux/amd64) And while doing so they pull >1GB of dependencies depending on the method.

I don't see that size reflected in your graph. What is it referring to? The compressed size of the pre-built archive I could download here?

Btw. in the meantime I was able to build myself a binary of ~12MB by just commenting out every DNS provider and its dependencies in dns_providers.go.

Since it is so easy to disable one ore more providers I think there should be a Makefile target to do so. Don't you think there is an interest in a minimal version of lego if the difference is so enormous in both build time and file size?

[ldez]

Don't you think there is an interest in a minimal version of lego if the difference is so enormous in both build time and file size?

Currently, I don't think so.

The current size of our prebuild binary is ~35MB. The target build, not use to create the prebuild binaries, can be optimized.

[EX0l0N]

The target build, not use to create the prebuild binaries, can be optimized.

I don't understand this sentence. How is that ~35MB binary built? Are you saying build is not the correct Makefile target?

Is my result bigger because I'm building from the top of the master branch?

[EX0l0N]

Is my result bigger because I'm building from the top of the master branch?

To answer myself here: It's the -sand -w ldflags. They are present in the .goreleaser.yml but not in the build target of the Makefile.

Using those flags with disabled DNS providers results in a ~8MB binary. I'm happy with it.