Filtering records to be extracted

[prescod]

Please describe the nature of this feature request

• I'm interested in extensions to existing CumulusCI functionality (e.g., broader applicability of an existing Task or Flow).

Does your feature request solve a current problem? Please describe.

When pulling information from persistent orgs (especially production, customer-facing orgs), it is sometimes helpful to be able to extract only a subset.

Describe the solution you'd like A way to specify WHERE clauses for extraction.

Automatically maintaining consistent references is outside of the scope of this project. The WHERE clauses would need to encapsulate that logic.

Context This is on the roadmap of an internal team.

[davisagli]

Automatically maintaining consistent references is outside of the scope of this project.

If someone retrieves e.g. a subset of Account but all Contacts, it might make sense and not be too hard to have a way to skip storing Contacts that have a lookup to an Account that was not retrieved, instead of erroring. So if you don't craft WHERE clauses for the objects that hold references, you'd be retrieving more data that you need to, but still only storing the records that are consistent.

[davidmreed]

Amaxa is implemented to trace references through a hierarchy and maintain referential integrity for partial extracts. I'm not sure to what extent the code is reusable (it's all Bulk API + CSV) but the design and portions of the code may be.

[prescod]

The Frontier Scale team is also interested in using CumulusCI for their general-purpose data-moving processes and thus would like filtering.