openlibrary icon indicating copy to clipboard operation
openlibrary copied to clipboard

fix: Add wait_for_solr to prevent race condition on startup (#11013)

Open 395ShikharSingh opened this issue 3 weeks ago • 0 comments

closes #11013

Problem

The trending_updater crashes with AssertionError on Docker startup when it queries Solr before Solr is ready.

Solution

Added wait_for_solr() function that polls Solr's /admin/ping endpoint with exponential backoff (up to 12 retries, ~2 min max) before fetching trending scores.

Files changed:

  • scripts/solr_updater/trending_updater_daily.py - Added wait_for_solr() function
  • scripts/solr_updater/trending_updater_hourly.py - Imported and uses wait_for_solr()

Testing

  • Verified with docker compose down -v && docker compose up

Successful startup logs:

solr-updater-1  | [TRENDING-UPDATER] Monitoring job trending_updater_hourly with crontab: 5 * * * *
solr-updater-1  | [TRENDING-UPDATER] Monitoring job trending_updater_daily with crontab: 0 0 * * *
solr-updater-1  | 2025-12-08 23:50:15 [openlibrary.trending-updater] [INFO] Starting trending updater init with offset 2025-12-08T23:05:00
solr-updater-1  | Running hourly trending for 2025-12-08T23:05:00
solr-updater-1  | 0 works with bookshelf books in the given hour
solr-updater-1  | 2025-12-08 23:50:15 [openlibrary.trending-updater-daily] [INFO] Solr is available
solr-updater-1  | Fetching works with existing trending data for hour slot 23
solr-updater-1  | 2025-12-08 23:50:15 [openlibrary.logger] [INFO] solr request: http://solr:8983/solr/openlibrary/export?q=trending_score_hourly_23...
solr-updater-1  | No new trending data found for the current hour.
solr-updater-1  | Caught up to current time, stopping.
solr-updater-1  | [TRENDING-UPDATER] 2 job(s) registered:
solr-updater-1  | [TRENDING-UPDATER] run_hourly_update (trigger: cron[minute='5'], pending)
solr-updater-1  | [TRENDING-UPDATER] run_daily_update (trigger: cron[hour='0', minute='0'], pending)
solr-updater-1  | [TRENDING-UPDATER] Scheduler started.

The key line [INFO] Solr is available confirms the wait logic works, and Scheduler started confirms successful initialization.

@cdrini PTAL

395ShikharSingh avatar Dec 08 '25 23:12 395ShikharSingh