Add exclude option to pipelines

[wdfinch]

Describe the feature you'd like to request

Within a pipeline section, we can specify an inputs option to tell turbo whether the task needs to be run again based on the files that changed. It would be nice if we could specify an exclude option to indicate which files should be ignored when determining if the task should be run again.

For example, when building a package, I do not want to rebuild the package if only test files are changed as these are excluded from the output. However, I do want to rerun my tests because a test file has changed.

I cannot simply use the include statement, because my test files end in .ts and there's no easy way to say include all the .ts files but not the .test.ts files.

Describe the solution you'd like

I would like to be able to specify an array of files that when changed are ignored and do not cause the task to be rerun.

{
  "$schema": "https://turborepo.org/schema.json",
  "pipeline": {
    "build": {
      "dependsOn": ["^build"],
      "exclude": ["src/**/*.test.ts"]
    },
    "test": {
      "outputs": [],
      "dependsOn": ["build"],
      "include": ["src/**/*.test.ts"]
    },
  }
}

So in this example, if test files change, then the build script is not rerun. However, the test script is run because the test files have changed.

Describe alternatives you've considered

Rerunning build whenever a test file or non-build file is changed.

[zaripych]

Hey, looks like this is related https://github.com/vercel/turborepo/discussions/1372. I have a patched version of Turbo that supports negative globs but wasn't sure if this feature would be accepted.

[zaripych]

Something along these lines works in my repo https://github.com/zaripych/turborepo/commit/4b3d7bbb11f628c7ae68cba39dcef7932a025538 just need more input to figure out if this is how this feature should go.

The negative patterns would be similar to how --ignore parameter of the CLI works.

[wdfinch]

Ok, thanks! Something like this would be nice. Seems like a pretty standard feature when you are choosing file paths.

[gsoltis]

Just want to leave a friendly note that glob handling is currently slated for reworking, and I don't think it's unreasonable to want to exclude files as well. I don't have a timeline for it right now, but it is definitely something the team is interested in.

[wdfinch]

Gotcha thanks!

[jineshshah36]

@gsoltis on a separate, but related note. We're having a lot of trouble tracing cache misses. It would be great to see a list of files or paths to that are considered for generating a hash so that we may determine what could be causing the cache miss.

[sppatel]

I am able to do this today with path spec syntax..

    "//#format": {
      "inputs": [
        ":!packages/",
        ":!tools/",
        "*.js",
        "*.jsx",
        "*.ts",
        "*.tsx",
        "*.json",
        "*.md"
      ],
      "outputs": []
    },

[zaripych]

@sppatel I think it should be :!: as per https://stackoverflow.com/questions/36753573/how-do-i-exclude-files-from-git-ls-files - but maybe it works either way - it works for me too, but that code has two paths in turbo ... if git doesn't work (or doesn't exist maybe) it tries to do manual hashing and that's where it would fail to interpret :!: 👍 - this is not officially supported feature yet - more like side effect of implementation.

[sppatel]

I followed https://css-tricks.com/git-pathspecs-and-how-to-use-them/#aa-exclude. I have def noticed inconsistency on what works and what doesn't especially with comparison to ls-files.

[HQ92]

Was there any update on being able to exclude files from the include blob, my specific use case was to include the public folder in a nextjs app to rebuild the app based on pdf or image changes but without a few auto-generated files such as those produced by next-sitemap. I've built next-sitemap to run after the build step, but it inserts a sitemap.xml into the public folder which when it comes to a rebuild, causes turbo to come up with a different hash that invalidates the cache and causes it to rebuild everytime.

[gajus]

Need this to exclude Dockerfile.

[b12k]

Need this feature to exclude generated CHANGELOG.md which invalidates turbo cache of a respective app or lib

[pburgmer]

I'm also interested in this feature. Would be nice to be able to exclude test files in build.

[gsoltis]

Apologies for leaving this open so long. inputs now supports a globbing syntax consistent with the rest of Turborepo, so you can do things like:

"inputs": ["**/*.js", "!**/*.test.js"]

Same as you would with outputs

[haquezameer]

@gsoltis It still does not seem to work. I have tried !src/mocks/** or !src/mocks/**/*.ts or :!:src/mocks/**/*.ts nothing worked.

{
  "$schema": "https://turbo.build/schema.json",
  "pipeline": {
    "build": {
      "dependsOn": ["^build"],
      "outputs": ["build/**"],
      "inputs": ["src/**/*.tsx", "src/**/*.ts", "src/**/*.ts", "!src/mocks/**"]
    },
    "test": {
      "inputs": ["src/**/*.tsx", "src/**/*.ts", "test/**/*.ts", "test/**/*.tsx"]
    },
    "lint": {}
  }
}

turbo version: 1.8.3 Mac OS Ventura yarn version: 3.0.0 node: 16.14.2

[gsoltis]

@haquezameer ah! my mistake, you'e right. We've unified the syntax with outputs, but not implemented the exclusions.

I'll reopen and see what we can do.

[vjcspy]

Why NOT have ignore configuration? Currently turbo watch all folder such as (docker, docker data, config.... It MUST be in same folder with mono project) that not related to the project.

[pegiadise]

Would be nice to be able to define only excludes. Right now even if my inputs are only negative globs, it never misses the cache even if src/** changes and you have to define both inclusions and exclusions.

[b12k]

Similar Proposal also available in this issue https://github.com/vercel/turbo/issues/3970