website-evidence-collector icon indicating copy to clipboard operation
website-evidence-collector copied to clipboard

Published 20 hours ago • verified publisher icon EU-EDPS

Added spider/crawler functionality

Open bitnesswise opened this issue 3 years ago • 4 comments

Added a spider capability to the website-evidence-collector, so it can be instructed to gather evidence of all the pages of a website

bitnesswise avatar Jun 25 '21 23:06 bitnesswise

Hi,

how can I install this?

gdmreal avatar Sep 27 '21 16:09 gdmreal

I don't think github offers the possibility to download a zip from the pull-request. But you can download it from the fork I created and on which the pull-request is based : https://github.com/bitnesswise/website-evidence-collector/tree/spider

bitnesswise avatar Sep 27 '21 19:09 bitnesswise

I just notice that both #65 and #66 introduce many many changes. So it is likely better to integrate this feature only after the proposed refactoring.

ghost avatar Oct 04 '21 14:10 ghost

Dear @bitnesswise,

this is just to let you know that I eventually managed to merge #66, which comes with many changes. I do not know how we proceed now best with this PR. I see the following options:

  1. rebase this branch on master
  2. write a separate spider script that uses the modules that WEC exposes now (possibly using a new distinct PR)
  3. abandon this PR for the time being

The option 2) comes possibly with the most flexibility. The new script can live in the WEC repo or in a separate repo. If this feature does not come with a lot of code, it can also be just one more option as it is the case in the current PR.

What's your view on this?

rriemann avatar Jan 06 '22 11:01 rriemann