fastcrawler icon indicating copy to clipboard operation
fastcrawler copied to clipboard

Abstraction layer in Spider

Open ManiMozaffar opened this issue 2 years ago • 0 comments

  • [ ] Define new abstraction, so the process would be unaware of what component is Spider
  • [ ] In result, we should be able to add Parser, Saver and etc
  • [ ] The abstraction layer, may cache data

Final Result Process(Spider() >> Parse(model=ParsingModel) >> Save(to=PostgreSQL)

ManiMozaffar avatar Aug 18 '23 16:08 ManiMozaffar