Attribute Distribution

[comnik]

Some relations (such as Datomic's :db/ident, or relations carrying metadata) are comparatively small but can induce massive skew, especially once reverse indices are involved. This can lead to straggling workers and lots of exchange data.

It might therefore be useful to introduce a new dimension of attribute configuration

Distribution := Sharded | Shuffled | Broadcasted

that would allow us to configure small, critical relations as Broadcasted, while keeping their would-be-skewed join partners entirely local, or shuffled randomly across workers.

The query engine might then need to enforce some new rules for what types of distribution are allowed to go together in a join.