COVID-QA icon indicating copy to clipboard operation
COVID-QA copied to clipboard

Published 20 hours ago • verified publisher icon deepset-ai

CDC Pregnancy page no longer in QA format.

Open mfleming99 opened this issue 4 years ago • 3 comments

https://github.com/deepset-ai/COVID-QA/blob/master/datasources/scrapers/CDC_Pregnancy_scraper.py

The CDC changed this page from a QA style page to a factual page on 7 April 2020. This scraper no longer produces any data when run.

mfleming99 avatar Apr 18 '20 20:04 mfleming99

Hey @mfleming99 I worked on the CDC general scraper that know has many more QA pairs and moved the CDC_Pregnancy_scraper.py to datasources/outdated in #101

Would you be interested in updating the pregnancy scraper yourself so we can add this data to our backend?

Timoeller avatar Apr 20 '20 08:04 Timoeller

Hi @Timoeller I would update the pregnancy scraper, but the CDC pregnancy page is no longer has question answer pairs to scrape. The page was converted into an informative page.

mfleming99 avatar Apr 21 '20 18:04 mfleming99

Ok understood. If you think it is valuable information we should still add it to our service.

If you update the scraper, please also update the manual check for a "?" at the end of each question/statement in our META_scraper.py?

Timoeller avatar Apr 22 '20 07:04 Timoeller