Nathan Sheffield
Nathan Sheffield
@vpnagraj do you want to take a look and see if you can figure out what I'm trying to do there? This would really open the door to higher performance...
@vpnagraj can you comment on this? would it be possible? It would dramatically speed up compute time if containers could pre-load the databases.
yeah, makes sense... So it's a tradeoff of memory and time. What if we kept a few containers pre-loaded, sacrificing a few gigs of memory, but the users would get...
> so for this to work as you've described we'd need to preload all of the regiondbs, also why would we need to preload *all* of them?
@nmagee Yeah. sharing among containers is one thing, and sharing among R processes is an independent thing (which can be solved with maybe with [rredis](https://cran.r-project.org/web/packages/rredis/index.html) or with the [svSocket](https://cran.r-project.org/web/packages/svSocket/index.html) above).
@vpnagraj where are we on the sharing processes/redis idea?
Interesting. I don't suppose it would make sense to do the inverse? (move the smaller query files over and then perform the computation in the svSocket server that already has...
One other thought: would this method dramatically reduce memory required by making it so all the little child processes didn't even need to load the database? In any case, I...
> Mongo actually has an even smaller limit for keys – 16MB I think, and not 512MB Is the key size a problem? or do you mean value size? but...
how about some kind of javascript site tour? Maybe after #30