[Core feature] Enable pyflyte backfill to ignore cache

[dennisobrien]

Motivation: Why do you think this is important?

When launching a workflow from the web UI, it is possible to select the advanced option "Overwrite cached outputs" in order to have that run ignore any cached values that may be available. This functionality is not available to the pyflyte backfill command.

Ideally when running a backfill, it would be possible to ignore cache for all executions in that backfill. Backfills are sometimes necessary when upstream data had a data quality issue which was later fixed, requiring downstream jobs to re-run on the restated data.

Goal: What should the final outcome look like, ideally?

The pyflyte backfill command should have a flag named something like --ignore-cache or --overwrite-cached-outputs that behaves similar to the web UI when selecting "Overwrite cached outputs".

Describe alternatives you've considered

I don't know of simple alternatives to running a backfill from the command line that has an option to overwrite the cache.

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

• [X] Yes

Have you read the Code of Conduct?

• [X] Yes

[dosubot[bot]]

🤖

It seems like you've got this under control, if you want help or have specific questions, let me know what I can do for you!

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.