Where does data live?

[hadley]

If your app need some static data, where should it live? Or if it needs dynamic data, what do you recommend?

[ColinFay]

Hey,

I've written a small part about this here: https://engineering-shiny.org/optim-caveat.html#reading-data

My advice would be (in the context of an application that will be deployed on a server):

• Small datasets that are not subject to change can be used as package data. I say small, because I benchmarked a few months back a Shiny app that bundled in itself "just" 300 mb of data, and when deployed and accessed by a few douzens of users at the same time, the memory is rapidly full

• These 300mb of data should be just fine if the application does not aim at being on a server, but to be used as a package inside an R session: each user would use their own RAM so that should be ok. But in that case the only issue is that using it as an application dataset would make the checking process slower, and the launch of the app also a little slower—it might be a good trick to keep the dataset as a csv or fst or feather dataset, and read the data progressively through the app.

• Otherwise, I would advice for using an external database, even more in the context of deployment: it will make the app faster, and will be more efficient in term of RAM and speed. It might require a little bit of technical knowledge for the devs to set it by themselves but nothing undoable, or maybe they can ask the IT for a small db. And when it comes to interacting with it, I'd advice for {DBI} + {dbplyr} to prevent SQL injection.

[hadley]

It sounds like the main issue with including data in the app is orthogonal to whether or not you use a package — if you have one R session per user, and want to support multiple simultaneous users, you need to be aware of memory usage in a way that you don't normally have to be.