Distributed data structures

[lukehoban]

We should validate that we have the right primitives for building distributed data structures like shared references, counters, etc. These may be implementable on top of Table or may need new primitives. We may also want do provide a built-in library of some common abstractions.

Some additional notes from @joeduffy:

The other category of things that we should add in here for consideration -- again, not necessarily part of "MVP" -- are things to help with distributed state management. (Remember Jason's suggestions here.)

For instance, maybe a very simple distributed collections library, like pulumi.collections.DistributedSet, pulumi.collections.DistributedHashtable, and so on. There are some pretty cool things we might be able to enable on top of this, like map/filter/reduce that leverages lambda, that approaches Spark and MapReduce. (Stuff like this.)

Even just a Shared<T> would be useful, which lets multiple functions safely and atomically party on a single shared mutable variable.

[joeduffy]

One of the distributed data structures we've discussed is simply a distributed variable. The idea is that you could simply say

let var = new pulumi.collections.Variable<number>(0);
something.subscribe(() => {
    var.set(var.get()+1); // increment!
});

This allows for simple persistence and sharing, presumably backed by a Dynamo table.

I was reading about Brendan Burns's new Metaparticle project https://medium.com/@brendan.d.burns/metaparticle-storage-8c3f6d5f68e5, and it sparked some further ideas.

What if, instead of just a variable, you could have an entire distributed primitive object? And what if we used JavaScript proxies to make the code look nice and clean?

let obj = new pulumi.collections.Object(...);
something.subscribe(() => {
    obj.counter++; // increment!
});

And imagine we used JavaScript proxies to make the code look even cleaner:

And, finally, imagine we made the storage configurable somehow, so you could pick Dynamo, Redis, or your provider of choice given the performance requirements of your usage.