DOCUMENTATION: A vignette / example with a basic Shiny app using futures

[PeterVermont]

Original subject: Please use my sample code in a vignette if helpful...

If you think it helpful please incorporate any and all ideas from some sample code I posted here.

[HenrikBengtsson]

Thanks. It's probably useful to have a basic vignette or similar illustrating how futures is / can be used with Shiny. Could probably be a Shiny app where one can choose different plan():s and the output illustrates the difference etc. I'll add it to the to-do list.

[PeterVermont]

I know that future is designed to work across clusters and to isolate processes but I would really like to be able to have some shared memory when running on a local machine. Basically I want to keep track of a long running background process with shiny as the Ui. Your restrictions on globals make this difficult so I am using writing and reading of files to communicate which is very inefficient.

Thanks

Peter

…………………………………………….. PETER ANDREWS Senior Software Engineer

RSG 55 Railroad Row | White River Junction, VT 05001 o 802.359.6411 | m 802.356.3141 www.rsginc.com ……………………………………………..

---

From: Henrik Bengtsson [email protected] Sent: Thursday, April 6, 2017 5:27:44 PM To: HenrikBengtsson/future Cc: Peter Andrews; Author Subject: Re: [HenrikBengtsson/future] Please use my sample code in a vignette if helpful... (#140)

Thanks. It's probably useful to have a basic vignette or similar illustrating how futures is / can be used with Shiny. Could probably be a Shiny app where one can choose different plan():s and the output illustrates the difference etc. I'll add it to the to-do list.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/HenrikBengtsson/future/issues/140#issuecomment-292328512, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AA2RkeBmKJ7SlGnCpsGNeCta6IMh68MXks5rtVjQgaJpZM4M0WBF.

[HenrikBengtsson]

I know that future is designed to work across clusters and to isolate processes but I would really like to be able to have some shared memory when running on a local machine. Basically I want to keep track of a long running background process with shiny as the Ui. Your restrictions on globals make this difficult so I am using writing and reading of files to communicate which is very inefficient.

I'm not sure if you're asking for a new feature (a feature request) or reporting on something that you think is broken.

Also, not sure what "restrictions on globals" refers to, but please note that you have full control over globals and how and if they are handled automatically, e.g. globals = TRUE (default), globals = <vector of names>, and globals = <named list of global values>. See argument globals in help("future") for details.

[PeterVermont]

Sorry to be unclear. I was frustrated but I think my main point is same as Issue #141 about communicating with a running process. As I said, Future may not be the right choice given that it is much more powerful since it designed to run across many different architectures. My case is super simple in that I only care about threads on the same machine but I don't know how to do it. I don't think globals helps me as it does not allow the caller and the callee to access the same active object.

A calls future(B)

B writes to global memory

A reads the global memory so it knows how B is progressing.

I saw your idea about file based communication but I keep feeling I am missing the obvious.

[HenrikBengtsson]

No worries. Comments below:

As I said, Future may not be the right choice given that it is much more powerful since it designed to run across many different architectures.

My understanding is that multiprocess futures will suit your needs very well. Formally, on Unix & macOS these are multicore futures and on Windows they are multisession futures (multiprocess is just a generic alias). Multicore futures forks the current R session (just like parallel::mclapply()) and multisession futures spawns off a new background R session (aka SNOW / PSOCK; just like parallel::makeCluster()).

My case is super simple in that I only care about threads on the same machine but I don't know how to do it.

Just for clarification; there's no parallel framework that runs parallel threads - all run parallel processes.

There's an important distinction in computer science between those two terms. Parallel processes do not share memory (just like your word process cannot share memory with R or the web browser). On the other hand, parallel threads share memory because they are part of the same process. So, if R could parallelize using threads, then you could easily share memory to and have one thread update a variable and another read that variable. But, R is explicitly designed not to do this - that would break the functional property of the language and many other things.

I don't think globals helps me as it does not allow the caller and the callee to access the same active object.

A calls future(B) B writes to global memory A reads the global memory so it knows how B is progressing.

Correct. Global variables won't help you because your main R process and your forked/background R process does not share the same memory and therefore not the same variables etc. The only straightforward way to "communicate" information back from the forked/background R process is via it's return value (when using futures that means the value of the future). That only happens after the forked/background R process completed. This basically how all parallel frameworks in R are designed - you spawn off parallel processes and collect the results when they finish.

I saw your idea about file based communication but I keep feeling I am missing the obvious.

The only solution is to setup another type of communication channel between the forked/background R processes and the main R processes. Using the shared file on your local drive is the easiest. That's the idea of what I propose in Issue #141.

Hope this clarifies.

[PeterVermont]

Very much appreciate you taking the time to so thoroughly and politely answer.

[PeterVermont]

I updated my gist FutureProcessorWithShinyExample to include a Shiny app that uses future

[HenrikBengtsson]

Hi, is this issue/documentation moot since the promises package brings asynchronousity to Shiny using future? The 'Using promises with Shiny' vignette shows how it's done.

Can we close?

[rpodcast]

I would still like to see a vignette for a slightly different take on using future with promises in a Shiny app. My typical use cases are that I have a Shiny app that won't be used by many people at the same time, but I want to optimize the intra-session experience as mentioned by a few people in this issue.

The apps I create at the $day_job often involve using SGE on the backend with either batchtools or clustermq to handle the job submissions. I've found a way to use future combined with promises to at least make the job submission be sent to a background session while the user can do other things in the app. I'd be glad to contribute a proof-of-concept app demonstrating this. This is slightly different than the great example mentioned earlier in this thread.