Play Assets Pipeline runs multiple times

[ssachtleben]

Hey,

I saw on my projects that sbt-plugins from the play assets pipeline runs multiple times. I created a new small sbt-plugin https://github.com/ssachtleben/sbt-test which is really basic and make a log message. I have created a new play project based on play-java-intro https://github.com/ssachtleben/play-pipeline-test and added only the small sbt-plugin in the assets pipeline:

pipelineStages in Assets := Seq(sbttest)

Now when I use activator clean and run it logs following:

[info] Compiling 6 Scala sources and 3 Java sources to /Users/anonymous/Development/Projects/playtest/target/scala-2.10/classes... [info] Running sbt-test [info] CoffeeScript compiling on 1 source(s) [info] Running sbt-test [info] - application - Creating Pool for datasource 'default'

So it looks for me that the pipeline runs once before the coffeescript compiling and afterwards. If I have heavy plugins in my pipeline it will take ages (for example digest). I know I can remove some for debug but still it would be nice when the pipeline only executed once after coffeescript compiling since my plugins are working on the js files. Or is it somehow a bug in the sbt-plugin or in the configuration?

Best, Sebastian

[ssachtleben]

Even with the easiest pipeline plugin possible:

val transform = taskKey[Pipeline.Stage]("transformer")

transform := { (mappings: Seq[PathMapping]) =>
  streams.value.log.info("Running transform")
  mappings
}

pipelineStages in Assets := Seq(transform)

it runs multiple times:

[info] CoffeeScript compiling on 1 source(s) [info] Running transform [info] Running transform

[anox1337]

+1

[richdougherty]

@anox1337, are you seeing this behaviour too?

[awilhelmer]

i have got the same behaviour in play 2.4 with the project example of ssachtleben

[richdougherty]

OK thanks. It's possible the CoffeeScript plugin is running twice too, but it only shows the message once because it uses incremental compilation.

[awilhelmer]

Possible, but if the Asset Plugin Pipeline runs twice and has to check 1000 files for changes twice, the incremental JIT compilation needs too much time. Even if it is only checking the file modification timestamps. Thus, development in huge projects needs more time than necessary. Short running incremental JIT compilations are one of the main features in play...

[ssachtleben]

Ok I made some more tests. I have added new log message in the SbtJsTask before this line: https://github.com/sbt/sbt-js-engine/blob/master/src/main/scala/com/typesafe/sbt/jse/SbtJsTask.scala#L285

streams.value.log.info(s"Running ${(taskMessage in task in config).value}")

Now the log looks like this:

[info] Compiling 6 Scala sources and 3 Java sources to /Users/anonymous/Development/Projects/playtest/target/scala-2.10/classes...
[info] Running JavaScript linting
[info] Running CoffeeScript compiling
[info] Running LESS compiling
[info] CoffeeScript compiling on 1 source(s)
[info] Running transform
[info] Running CoffeeScript compiling
[info] Running JavaScript linting
[info] Running LESS compiling
[info] Running transform
[info] - application - Creating Pool for datasource 'default'

So I'm sure that all sbt plugins runs twice on start. Is there any case where it make sense to run the plugins multiple times? Since all plugins handle their stuff and when the dependencies are properly set there is no need for a re-run.

As @awilhelmer mentioned even if the plugins check the timestamps from the files if there are 5k coffee files it needs to check 10k instead of 5k files. This will slow down big projects alot.

Anyone got an idea where the code is for the plugins execution? I'm not that deep in sbt build process to know where to start looking...

[mkurz]

Could https://github.com/sbt/sbt-js-engine/pull/43 https://github.com/sbt/sbt-web/pull/131 https://github.com/sbt/sbt-web/pull/130 be related?

[mkurz]

I can confirm this bug still appears in Play 2.5 using sbt 0.13.11. Does anyone have an idea what's the issue here?

[shkhln]

After staring at sbt-optimizer output a bit and inspecting task dependencies, that double compilation looks pretty straightforward:

playReload -> playCompileEverything -> playAssetsWithCompilation -> web-assets:assets

playReloaderClasspath -> Runtime / exportedProducts -> web-assets:exportedProducts # seems to be a webjar generating task -> web-assets:webExportedDirectory

Related to https://github.com/playframework/playframework/issues/5765

[dispalt]

I'd love to help with this since I am using play w/ scalajs and scalajs-bundler it really slows things down, but I am not entirely sure where to begin.

Any good ideas?

[dispalt]

Actually it looks like the whole compile pipeline is called twice on every change. Thanks for the hint on sbt-optimizer it's awesome thanks @jrudolph!

[dispalt]

It looks like @benmccann has been in the class loader code looking around, he might have an idea...

[benmccann]

Not off the top of my head unfortunately. I'd have to dig in just as much as anyone else

[shkhln]

@dispalt Try setting WebKeys.exportedMappings in Assets := Seq() in build.sbt.

[dispalt]

@shkhln it works indeed, wow, that is awesome.

[schmitch]

actually I think we should make WebKeys.exportedMappings in Assets := Seq() the default in play.

[shkhln]

@schmitch Sure, go ahead. I initially thought that setting might break webjar-based Play projects (I've never used webjars myself), but I haven't been able to think up any failure cases.

[mkurz]

So I was having a look at this and can confirm

Assets / WebKeys.exportedMappings in := Seq()
TestAssets / WebKeys.exportedMappings in := Seq()

fixes the problem without breaking any other webjar behaviour. So we need to make this default.

[mkurz]

So I did some testing with the fix (exportedMappings = Nil) locally and the nice side effect of it is that if you have many files and/or big files inside assets/ and/or public/ a reload in dev mode will now be 50% faster :+1: See my pseudo benchmark here: https://github.com/playframework/playframework/issues/7799#issuecomment-1999742140 (Like mentioned in the referenced ticket, the 50% is/was kind of predictable because with a fix for this PR and #5765 the assets pipeline does not run twice anymore and also the assets will not be copied twice anymore in the target folder...)

[mkurz]

This is not fixed yet, #12483 "only" fixed #5765.

[mkurz]

Fasten your seatbelts: This is finally fixed after 8,5 years :tada:

Turns out that setting exportedMappings to Nil is not always desired and (according to our scripted tests) can lead to problems in certain situation. Anyway, good to see this fixed finally.