data.gov icon indicating copy to clipboard operation
data.gov copied to clipboard

Published 20 hours ago • verified publisher icon GSA

Create job/complete route

Open jbrown-xentity opened this issue 9 months ago • 1 comments

User Story

In order to run jobs as often as possible, data.gov harvesters want job completion to notify the flask app/controller so that it can start new jobs waiting in the queue.

Acceptance Criteria

  • [ ] GIVEN the route /job/complete exists
    WHEN the /job/complete endpoint for the flask app is called (at the end of a job)
    THEN the controller/flask app kicks off new jobs that are pending.

Background

In order to not wait for harvester to restart, and to keep jobs moving.

Security Considerations (required)

Want to limit this route to authorized only, if possible.

Sketch

Please note there is a race condition here: when the job makes this post/get request, it is still running. It will wait for a response.

Therefore, we want to give an immediate response, wait 1-2 seconds, and then run the analysis. Otherwise we will find the job still running, and nothing will get kicked off.

jbrown-xentity avatar May 20 '24 18:05 jbrown-xentity