feat: Cleanup Old Records When Retiring a Configuration File

[bbernays]

Which problem is this feature request solving?

Currently if you have multiple configuration files all syncing to the same database using the overwrite-delete-stale write mode there is no way to delete all records associated with a configuration file as soon as the config file is no longer in use.

Describe the solution you'd like

Azure supports an empty subscription list... maybe AWS should support an empty account list...

Another potential solution would be to implement a remove or delete command

Pull request (optional)

• [ ] I can submit a pull request

[erezrokah]

Hi @bbernays, I think for starts, we could create an how to guide that explains how to do it. For example in Postgres one could drop all tables where source_name = X do accomplish this, WDYT?

[bbernays]

While I think we could write a tutorial on how to do it for Postgres, but I don't think it will scale to all of our destinations that support. As we have more users that are doing more parallelization at the config.yml I think this is a use case that more users will run into.

As long as we only support this functionality for destinations that support overwrite-delete-stale then it is not a net new capability but more a exposing of an existing functionality that is currently only done implicitly.

[erezrokah]

Could we start with a Postgres guide? If we see that doesn't work we can consider adding the feature. and then probably have more context on it

[shani1998]

We have a similar use case with k8s source, wherein when we delete the k8s cluster with the source name same as a cluster ID, then there is no way to delete the data from Postgres DB corresponding to the deleted cluster. (Discord_Msg_link) Looking forward towards a common solution around this.