COPY: Recursive globbing

[mxey]

It would be useful if Earthly supported recursive globbing.

I would like to turn this

    COPY ./headless-exporter/*.go ./headless-exporter/
    COPY ./cleaner/*.go ./cleaner/
    COPY ./discovery/*.go ./discovery/
    RUN CGO_ENABLED=1 go test --race ./...

into this

    COPY **/*.go .
    RUN CGO_ENABLED=1 go test --race ./...

[taylor-sutton]

Upvoted, I can add a little about our case here, which is also about Go projects. It's not common to have a src/ directory in Go projects, and many projects have a sprawling set of source directories at the top level (e.g. https://github.com/moby/moby/). src/ is actually explicitly dis-recommended.

The problem this solves for us would be to have a COPY command which:

1. Easily adds all the files needed for testing and/or building, but excludes all other files to get better caching.
2. Doesn't require remembering to add new COPY lines when we add new top level directories.

[anjmao]

Also best practices page https://docs.earthly.dev/best-practices recommends to avoid copying all files.

# Avoid
COPY . .

# Best
COPY ./*.go ./

Without recursive globbing only top level go files will be copied. In real world there are more packages which can be left untested.

Explicitly adding all packages for copying is error prone too. It's better to use COPY . . and ignore unwanted files via .earthlyignore In worst case you get cache miss instead of untested or not compiling code.

[mxey]

I realized that my example above is not how recursive globbing usually works, because it would not keep the directory structure. I guess it would be better as:

COPY --only="*.go" . .

Think of it as earthlyignore configurable per COPY instead of globally.

[gjermundgaraba]

I have the same go use case, but also linters and other things that require a lot of specific files to be copied in, and we have had multiple scenarios now where CI has been green, but only because something was not remembered to be copied. So we have to go back to COPY . . even if it is inefficient.