devdocs icon indicating copy to clipboard operation
devdocs copied to clipboard

[low priority refactoring] generalize executing JS during scraping

Open alexch opened this issue 7 years ago • 1 comments

If doc sites use client-side JavaScript to produce or fetch some of their content, we need to launch a headless browser and execute it. Currently scraper subclasses do that individually using capybara, e.g. https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/scrapers/angularjs.rb#L42-L45 but it might be better to extract a helper method or config option, a little more like this: https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/scrapers/webpack.rb#L43-L46

cc @dangeranger

alexch avatar Apr 06 '18 17:04 alexch

We only have three legacy docs requiring capybara: https://devdocs.io/angularjs~1.8/, https://devdocs.io/angular~2/, https://devdocs.io/immutable/

Thus, the refactoring is indeed very low priority.

simon04 avatar Apr 29 '21 22:04 simon04