hyphe
hyphe copied to clipboard
medialab
Websites crawler with built-in exploration and control web interface
When dealing with a high number of pages, the web entity folder view can take a very large amount of time to display a folder view. In my case I...
 the last " is kept from a href="url" parsing
TODO: - [x] download chrome headless + driver - [x] remove phantom binary - [x] plug chrome within selenium in scrapyd spider - [x] include install in build docker -...
The recrawl process first asks the crawl limits (see #158 ) There is also an option to avoid downloading already downloaded page (set to true by default). This process will...
It would be useful to extract a clean textual content from each web page. We could use Boilerpipe for instance https://github.com/kohlschutter/boilerpipe
such as tooltips, or text for each box and entry
We need three settings for web entity crawls: - *depth* : integer or infinity (only if *number of pages* is not infinite) - *number of pages* : integer or infinity...
Which could be justified (as in "0 elements in the elements selected were tagged with that particular tag before you clicked"), but it gets awkward as you can tag the...
