Watchable fileset reified in the REPL session

[stephenbrady]

Filing this as an issue per discussions in Slack.

Once boot enters the repl task, no other task can run within that pipeline. If a user has a task or pipeline of tasks to run that should be triggered based on changes to the fileset, these can only be run within the same JVM by spawning a new boot pipeline from within the repl. In other words, the repl task blocks the pipeline, rendering a pipeline such as boot watch a-reactive-task repl inert because it only runs once through (only calling the pre of each task and never returning to the watch).

While the user can spawn a new boot pipeline from within the repl, if that pipeline involves a watch, a whole new set of file system watcher threads will be spawned. This scales very poorly as each directory watched, including every subdirectory within, requires a new thread (at least in OSX). At several hundred threads, most JVMs come to a crawl.

Boot itself syncs user directories using a set of watcher threads while the repl task is running. One possible solution to this problem overall, as discussed in Slack, is exposing the fileset within the repl and enabling the user to add watchers to that fileset that then can trigger and feed into tasks/pipelines. This would enable reuse of these filesystem watcher threads and decouple the watching of the filesystem from the triggering of pipeline(s).

[azizzaeny]

yap... why calling boot watch cljs from repl suddenly loosing its repl ability...? then we must connect from another repl -c from another terminal, and create another JVM process... to continue our immediete feedback mode... it seem watch task never return back to repl...

Starting file watcher (CTRL-C to quit)...

Writing main.cljs.edn...
Compiling ClojureScript...
• main.js
Elapsed time: 20.643 sec

[micha]

@azizzaeny I think what you want to do is run the build in the background in your REPL, like this:

boot.user=> (def f (future (boot (watch) (cljs))))

Then you can continue to work in the REPL with the build still running in a background thread. When you want to stop the build you can do this:

boot.user=> @f

which will bring the build to the foreground. You can then press ctrl+c to kill it.

There is also the excellent gfredricks/repl-utils library that can make this kind of work more pleasant.

[pesterhazy]

Also see https://github.com/boot-clj/boot/issues/473

[azizzaeny]

thanks.. that improve workflow... rather than connect from antoher repl.... when we recompile manually it takes longer than using watch... for example:

; manually
boot.user=>  (boot (cljs))
; takes about 12s

; one pipeline task within watch
boot.user=> (boot (watch) (cljs))
;after make changes and save only 1s

i think like @stephenbrady issue is "Once boot enters the repl task, no other task can run within that pipeline. If a user has a task or pipeline of tasks to run that should be triggered based on changes to the fileset..."

Even we are under same boot-env, invoking another (boot) from same repl.. skiping pre task and next handler fileset. is this create another pods rather than use the same pods...? or watch and cljs works in the same callback pipe..?

[Deraen]

Re: slow cljs, each (cljs) creates new compiler env and separate output directories etc. You should be able to create one compiler env and then manually trigger build, if you want:

(def foo (cljs))

(boot foo) ;; first compile, slow
(boot foo) ;; fast

This approach should work with most compilation tasks.