No support for Gradle Build Cache with NodeJS tasks

[sergeykad]

Using the new Gradle build caching can really speed up build execution.

[srs]

Will investigate how to implement this. Thanks for the tips.

[eriwen]

+1 from Gradle Team ❤️

[eriwen]

Here's an example of a similar task to NodeTask that leverages the build cache: https://gist.github.com/eriwen/0219d9cb815909a8409eb08651459f3a

[zfy0701]

+1 it seems really useful since we do global build a lot, and some node tasks is just taking too long to execute.

I assume even without the new gradle cacheable task

adding a field like

    @InputFiles
    def source

would be able to let user control if a task could be cached based on certain file.

[tizki]

@eriwen can you please give an example how to use the NodeTask?

[samsieber]

Support for this is in 1.2 (I think) - see this comment on the pull request https://github.com/srs/gradle-node-plugin/pull/205#issuecomment-321287894

[NikolayMetchev]

What would be even better is if NpmTask were cached as well. Should I open a different issue?

[Flarna]

At least for npm_install caching may result in issues on linux. There are symlinks created in node_modules/.bin for some modules but gradle is not (yet) able to handle them correct. As a result the unpacked cache holds files instead links and that doesn't work.

[NikolayMetchev]

is there an issue for symlinks support in the gradle cache that we can vote on?

[Flarna]

it's documented: https://guides.gradle.org/using-build-cache/#symbolic_links and I found following: https://github.com/gradle/gradle/issues/3525

[NikolayMetchev]

Thanks @Flarna but it does feel like we need a new issue to make NpmTask cacheable.

[Flarna]

Agreed as besides the gradle issues with symlinks also changes in this plugin are needed to set PathSensitivity.RELATIVE. It should work then on windows or in case the .bin links are not needed.

[nhoughto]

Its worth noting that npm_install might have problems because of node_modules symlinks, but other npm tasks like run xyz will work ok (as long as no symlinks). This means you can use build cache for your build (and potentially test) processes, just not npm install.

Easy to enable via outputs.cacheIf { true }

https://docs.gradle.org/current/javadoc/org/gradle/api/tasks/TaskOutputs.html#cacheIf-org.gradle.api.specs.Spec-

[MartinHaeusler]

I'm facing this issue too. I'm using this plugin to trigger a yarn run xyz command (which internally triggers webpack), and gradle does not recognize when the source files are unchanged, running the yarn command way more often than needed. Considering that yarn runs for a good 5 minutes, multiple needless executions really stack up fast.

Any ideas how to fix this?

[MartinHaeusler]

Here's a workaround:

task build(type: YarnTask){
    // the usual yarn config...
   
    // tell gradle what the input and output files are
    inputs.file("package.json").withPathSensitivity(PathSensitivity.RELATIVE)
    inputs.dir("src").withPathSensitivity(PathSensitivity.RELATIVE)
    inputs.file("yarn.lock").withPathSensitivity(PathSensitivity.RELATIVE)
    outputs.dir("dist")
    // tell gradle to apply the build cache
    outputs.cacheIf { true } 

}

For more info: https://guides.gradle.org/running-webpack-with-gradle/#step_4_leverage_gradle_build_cache

[JamesXNelson]

Just an fyi for others: you can use --no-bin-links to avoid those symlinks. In your package.json, you'd need to replace "raw executable references" like react-scripts with node node_modules/react-scripts/bin/react-scripts.js... so it's uglier, but at least you don't get broken symlinks!

[NikolayMetchev]

@JamesXNelson I attempted to use your workaround but I can't seem to get yarn_install to be cacheable Here is what I put in my build.gradle. If I use --scan it tells me that the yarn install task is not configured to be cacheable. Any help would be appreciated.

tasks.yarn_install {
  inputs.file('package.json').withPathSensitivity(PathSensitivity.RELATIVE)
  outputs.dir('node_modules')
  if (System.getenv("CI") != null) {
    args = ['--frozen-lockfile','--no-bin-links']
  } else {
    args = ['--no-bin-links']
  }
  outputs.upToDateWhen { true }
  dependsOn clean
}

[NikolayMetchev]

Nevermind. I have a solution now:

yarn_install {
  inputs.file('package.json').withPathSensitivity(PathSensitivity.RELATIVE)
  outputs.dir('node_modules')
  if (System.getenv("CI") != null) {
    args = ['--frozen-lockfile','--no-bin-links']
  } else {
    args = ['--no-bin-links']
  }
  outputs.upToDateWhen { true }
  outputs.cacheIf { true }
  dependsOn clean
}

[deepy]

If you use the preconfigured yarn install task from the fork I think it should work out-of-the-box

[NikolayMetchev]

If you use the preconfigured yarn install task from the fork I think it should work out-of-the-box

Which fork?

[nhoughto]

fork of this project, https://github.com/node-gradle/gradle-node-plugin

worth noting for google visitors that --no-bin-links works, but its kind of fighting the framework, if any packages use post-install actions they will likely fail, and you can't really update these scripts like you can with your own package.json commands. There may be more things that this breaks too i'd guess.

[JamesXNelson]

Using --no-bin-links really sucks, as you have to use absolute paths for everything, and the clutter makes things very difficult to maintain. A proper fix would be ideal.

We manage caching by handling the node_modules directories directly (using a maze of overlayfs and tmpfs) and then teaching both gradle and npm how to dance lightly around who-gets-to-delete-what. It's not pretty, but it works.

Having gradle able to cleanly delete symlinks when it has already deleted the referent (delete links first) would be helpful...