pipeline-backend icon indicating copy to clipboard operation
pipeline-backend copied to clipboard

Published 20 hours ago • verified publisher icon instill-ai

refactor(web): refactor web operator

Open chuang8511 opened this issue 1 year ago • 2 comments

Because

  • the users feel the web operator is confusing

This commit

  • take scraping ability out from crawler
  • update the document for the confusion
  • add migration

chuang8511 avatar Oct 11 '24 11:10 chuang8511

INS-6568 Fix remove-tags and only-include-tags for TASK_CRAWL_WEBSITE

linear[bot] avatar Oct 11 '24 11:10 linear[bot]

memo Will rebase later.

chuang8511 avatar Oct 11 '24 15:10 chuang8511

@pinglin @GeorgeWilliamStrong I modified all based on your suggestions.

chuang8511 avatar Oct 14 '24 12:10 chuang8511

I will rerun the integration test later to avoid that it achieves the limits.

chuang8511 avatar Oct 14 '24 16:10 chuang8511