fastcrawler
fastcrawler copied to clipboard
Abstraction layer in Spider
- [ ] Define new abstraction, so the process would be unaware of what component is Spider
- [ ] In result, we should be able to add Parser, Saver and etc
- [ ] The abstraction layer, may cache data
Final Result Process(Spider() >> Parse(model=ParsingModel) >> Save(to=PostgreSQL)