comdb2
comdb2 copied to clipboard
bloomberg
Sharding : CREATE TABLE PARTITIONED BY %
This patch : 1.) Adds support for the following queries :
- CREATE TABLE T .... PARTITIONED BY CASE <KEY> % <NUMSHARDS> WHEN 0 THEN 'SHARD1' ... WHEN N THEN 'SHARDN' END
- DROP TABLE T 2.) Adds a test which sets up three databases - one co-ordinator and two shards, populates tables in the individual shards, runs select queries on the co-ordinator and verifies if the rows returned match the rows in the tables on the shards.
Implementation - The design is similar to time partitions. The CREATE statement is forwarded to the leader (via Bpfunc codepath). The leader writes meta data about the sharded table, logs SC_DONE and populates in-memory structures to represent the sharded table. The replicants upon receiving SC_DONE, create their own in-memory structures for the same.
Additionally, everytime a sqlengine is being prepared, a 'CREATE VIEW ' query is constructed as a UNION ALL of all fields from all shards (read from in-mem structures) . This query is run against the sqlite engine resulting in the creation of a view which is technically the "table" on the co-ordinator database.
As it stands, select queries are executed on every shard. A future PR will introduce changes that will filter based on the WHERE clause and the sharding key, and run the selects only on the shards having the relevant rows.
