cugetreg icon indicating copy to clipboard operation
cugetreg copied to clipboard

[BUG] Investigate random scraper freeze

Open bombnp opened this issue 3 years ago • 1 comments

Problem

Sometimes, the scraper just get randomly stuck (like LITERALLY stuck, no errors, no alert, nothing). This causes the courses to be outdated without any of us knowing.

Task Description

So either we should:

  1. Investigate the problem and fix it or
  2. Implement a discord alert (or auto-restart the service) if we detect the scraper is stuck (we should probably do this too anyways).

Additional Context

Right now, we can detect using isScraping property in ScraperService, which if true, would deny scrape action when scrape() is called by scheduler.

Related Teams

  • [ ] Frontend
  • [X] Backend
  • [ ] Data
  • [ ] Design
  • [ ] Infra
  • [ ] QA

Task Advisors

@bombnp

bombnp avatar Nov 09 '22 03:11 bombnp

Actually, im not sure if things will change with @OnFireByte 's new scraper. Will investigate later if it still persists.

bombnp avatar Nov 09 '22 03:11 bombnp