Interactive interface (avoiding yaml)

[richfitz]

It should be possible to run maker directly from an .R file, rather than going through yaml: all the yaml is doing is holding lists of things. Something like

m$add_target(foo <- command(bar))

or

maker::add_target(m, foo <- command(bar))

should work fairly well and don't look too horrible (both include the assignment symbol which I think is probably important). Doing this could also allow for loops over files, etc, but that will require some extra glue to do variable substitution.

We could also arrange to do this in "global mode" where creating a target will also create either a delayed assignment or active binding function.

[richfitz]

The idea for syntax above does not work so well in the case of plots:

m$add_target("foo.plot" <- myplot(), plot=TRUE)

just looks weird. With a custom replacement/complex assignment function it might be possible to do something like

m$targets["plot.pdf"] <- target(myplot(), plot=TRUE)
m$targets["plot.pdf"] <- plot_target(myplot())

but there's a lot of syntax there. The simplest case is still:

maker::add_target(m, "plot.pdf", myplot(), plot=TRUE)

which could map to

m$add_target("plot.pdf", myplot(), plot=TRUE)

[richfitz]

Mildly nicer:

  m <- maker(NULL)
  m$add <- "package:testthat"
  m$add <- "code.R"
  m$add <- target("data.csv", download_data(target_name),
                  cleanup_level="purge")
  m$add <- target("processed", process_data("data.csv"))
  m$add <- target("plot.pdf", myplot(processed),
                  plot=list(width=8, height=4))

  m$make("plot.pdf")

At the same time, I'm working on "links" that bind maker-controlled objects into an environment as read-only variables that fetch the appropriate thing (all offline at the moment).

[richfitz]

Something like this would be doable (but with a bit more effort) and might be really nice:

m <- maker({
  library(testthat)
  source("code.R")

  file("data.csv", cleanup_level="purge") <- download_data(target_name)
  processsed <- process_data("data.csv")
  plot("plot.pdf", width=8, height=4) <- myplot(processed)
})

[srenatus]

Hi there,

I'm amazed to find that your last comment describes exactly what I had on my "one could propose that improval"-list. (So, :+1:)

One last mutation of the idea: maybe it'd be nice to just introduce a remake.Rm file (the "m" hinting at remake) that is (superficially, with <- redefined) plain R but actually encodes targets and their dependencies, just like in your maker(...) call:

  library(testthat)
  source("code.R")

  file("data.csv", cleanup_level="purge") <- download_data(target_name)
  processed <- process_data("data.csv")
  plot("plot.pdf", width=8, height=4) <- myplot(processed)

It'd give us better IDE support than wrangling strings in YAML, too.

Just a thought. Thanks for your work on this library, it greatly affected my workflow ;)

[wlandau]

I think a nice addition to interactivity would be a way to take an inventory of the storr cache and load cached objects in an R session. This is what recall() and recallable() are for in parallelRemake, and I am wondering if you might like remake to have this functionality natively. Of course, these functions should not be used directly in remake.yml or its commands, but it is nice to casually inspect output between long jobs.

[fmichonneau]

@wlandau are you aware of create_bindings() and make_environment()?

[richfitz]

Probably also worth noting that I walked back on the implementation of interactivity I first thought about and removed it (might be in the refactor branch?)

[wlandau]

@fmichonneau I was not aware of create_bindings() or make_envorinment() because I did not see them in the README or docs. I just tried them out, though, and they seem useful, though I had one project from the summer where create_bindings() would try to load 30 GB into memory.

recall() and recallable() interface a bit differently, so I think these parallelRemake functions can coexist.

[krlmlr]

Why not just:

library(testthat)
source("code.R")

download_data("data.csv") # cleanup_level: purge
processed <- process_data("data.csv")
myplot(processed) # plot: "plot.pdf", width=8, height=4

This should give roughly the same results both inside and outside of remake.