aztk
aztk copied to clipboard
Move cluster config from storage file to storage table
Currently, each cluster has a container with a blob containing it's config.
We should move all cluster configs to a storage table like:
| cluster_id1 | cluster_config dump1 | | cluster_id2 | cluster_config dump2 | | cluster_id3 | cluster_config dump3 |
We can crosscheck the batch pool list with this table to better handle external Batch issues (i.e. an AZTK cluster's job was deleted outside of AZTK).
We can also add a --show-config or similar to aztk spark cluster get to give more detailed information. Since the config shows plugins, the docker image used, etc.
Performance with a single table like this might also be better than getting a blob for each cluster.