backend icon indicating copy to clipboard operation
backend copied to clipboard

submit high priority jobs to extract-and-vector queue from topics-fetch-link jobs

Open hroberts opened this issue 4 years ago • 0 comments

The run_remotely() call to extract_and_vector from topics-fetch-link was mysteriously hanging occasionally, eventually causing the fetch link queue to shrink. I changed the run_remotely() call to an add_to_queue() call followed by polling the database to find out when the extraction is done. It currently works ok, but it hangs the fetch-link queue if there is any backlog in the extractor queue. It would be a lot more robust to be able to stick extractor jobs at the front of the queue, but we need priority support in the python jobs system to be able to do that.

Alternatives:

  • move the extraction code into common (or extractor-base or something) and just do the extraction directly from fetch-link (this is what we were doing pre-containers);
  • add a separate extract_and_vector_for_fetch_link queue to quarantine fetch-link extractor jobs, as we do with topics-mine / topics-mine-public;
  • figure out why run_remotely() is hanging (tried -- it's hard to find the particular job that is causing the problem, and we should have a timeout in any case so that a single bad extractor job does not essentially kill a fetch-link job);
  • re-architect fetch-link so that it goes away and lets another instance of the job try again later after submitting the extractor job, so the fetch-link job is not sitting around.

The first is by far the simplest, but it mucks up the code separation. The second is simple, but it will eat up 32 or whatever db handles at times. The last two are harder.

hroberts avatar Jan 22 '20 20:01 hroberts