edit.tosdr.org icon indicating copy to clipboard operation
edit.tosdr.org copied to clipboard

Implementing Phoenix-side code to crawl documents using OTA engine

Open madoleary opened this issue 1 year ago • 2 comments
trafficstars

  • [x] Are you working on the latest release?
  • [x] Have you tested it locally?
  • [->] Please explain your change

This PR would implement the OTA engine for crawling document text on document creation, update, and for individual crawls made on-demand by the user. This addresses issue https://github.com/tosdr/edit.tosdr.org/issues/1168

It also includes error-handling and removes references to the old crawler system.

madoleary avatar Sep 18 '24 15:09 madoleary

Code is working in development but merging this PR to main is still blocked on https://github.com/tosdr/edit.tosdr.org/issues/1186 since (we think) that would trigger a deploy to production - don't merge this PR until the necessary production environment variables are in place!

michielbdejong avatar Sep 26 '24 11:09 michielbdejong