aztk icon indicating copy to clipboard operation
aztk copied to clipboard

Move cluster config from storage file to storage table

Open jafreck opened this issue 7 years ago • 0 comments

Currently, each cluster has a container with a blob containing it's config.

We should move all cluster configs to a storage table like:

| cluster_id1 | cluster_config dump1 | | cluster_id2 | cluster_config dump2 | | cluster_id3 | cluster_config dump3 |

We can crosscheck the batch pool list with this table to better handle external Batch issues (i.e. an AZTK cluster's job was deleted outside of AZTK).

We can also add a --show-config or similar to aztk spark cluster get to give more detailed information. Since the config shows plugins, the docker image used, etc.

Performance with a single table like this might also be better than getting a blob for each cluster.

jafreck avatar Mar 08 '18 19:03 jafreck