Feature Request: Options to rerun pipelines for selective date ranges

[Charantl]

trafficstars

If the pipeline failed in the past few weeks and went unnoticed, we will need an option to rerun the pipeline to fetch the data for specific date ranges.

[chandrashekar-s]

Hi @Charantl, thanks for filing this issue. Based on the current design of fhir-data-pipes, if a pipeline fails then the next pipeline that runs will start picking up records from the point where the previous pipeline was successfully complete (i.e. from the start time of the previous successful run as that would be the cut off time for that run). Based on this design, I think the scenario of running the pipeline for a specific data range might not be needed as there wont be records left out even if the pipeline fails (as they will be picked up in the next run).

Do you still think that this feature is needed? If yes, can you please specify the use case for the same.

[Charantl]

Yes @chandrashekar-s. If data for a specific date(s) is removed due to human error, we need a feature to reload the data for that particular date (range).

[bashir2]

Closing this issue because:

• As @chandrashekar-s mentioned the pipeline automatically fetches resources since last successful run and
• In the second scenario that @Charantl mentioned, you can still go back to a snapshot before the problematic period and remove the snapshots after (such that the next incremental run covers that period). If older snapshots are not available or the period is too long, a full-run can be forced.

Note that we are also planning to add features that cover #1012 and #1013 and with those, this issue becomes even less important. Please feel free to re-open this if there are still practical scenarios that are not covered by above features/comments.