Feature Request: support negated globs in `outputs`

[felixmosh]

What version of Turborepo are you using?

1.4.0

What package manager are you using / does the bug impact?

Yarn v2/v3 (node_modules linker only)

What operating system are you using?

Linux

Describe the Bug

With the following turbo.json file,

{
  "globalDependencies": [
    "./env"
  ],
  "pipeline": {
    "build": {
      "dependsOn": [
        "^build"
      ],
      "outputs": [
        ".next/**",
        "!.next/cache/**"
      ]
    },
  }
}

The !.next/cache/** glob doesn't ignores the .next/cache folder, which is redundant in artifact cache & bloats the artifact zip size.

Expected Behavior

It should ignore the specified folder.

To Reproduce

Use the basic Next.js example with Next.js cache enabled with the given turbo.json file

[nathanhammond]

Heya @felixmosh! This is actually something that we've never implemented.

https://github.com/vercel/turborepo/blob/a7de4010a1691a4c9ec0f11fd4adb85669eac3f5/cli/internal/runcache/runcache.go#L272

This is a feature we should definitely add, though.

[felixmosh]

Funny because @jaredpalmer gave me this answer in Discord 😅

[nathanhammond]

@jaredpalmer thinks it probably existed in the JS version and was never ported over. It is definitely on our list now.

[felixmosh]

There is any progress with this one?

[felixmosh]

Is it solved? I've updated to the latest Turbo, and seems that my cache artifacts are ignoring next's cache folder.

[nathanhammond]

@felixmosh I'm terribly sorry we forgot to get back here and update you! @NicholasLYang got to this in #2031 which showed up as part of v1.5.6.

I think you've already discovered that though. Let me know if you discover anything weird with it!

[felixmosh]

@nathanhammond thank you! This feature helps me a lot 🎊

[filipw01]

@nathanhammond @NicholasLYang What do you think about adding this to inputs too? I'm trying to get cache based on all files except package.json because we do version bump after merge which invalidates cache that was created just before merging a feature

[nathanhammond]

Unfortunately, we cannot generalize a solution for that. It might work, it might not work. Certain build scripts could pull in the version number and include that in the actual execution of the command.

Presumably you do not version after every single merge, so this is likely not a significant reduction in your cache hit ratio?

[filipw01]

Presumably you do not version after every single merge, so this is likely not a significant reduction in your cache hit ratio?

Actually we do. Only changed packages, but on every merge to master. I know package.json could affect some outputs, but it's not the case for us

[nathanhammond]

So, unfortunately, I'm not aware of a generalizable solution here. We explicitly include package.json ourselves and even if you were to specify !package.json we'd add it back in for you. Excluding it is not viable for the general case.

So, your workflow is not compatible with the Turborepo workflow.