antch icon indicating copy to clipboard operation
antch copied to clipboard

Please show another Exit Method ,ths

Open liuzeng01 opened this issue 4 years ago • 1 comments

I had read the example spider, it used the signal chan as the exit method .But it's not very helpfully. Can you show more exit method ? For example , if the spider having done all the scrapy works ,it would exit automatically ?

liuzeng01 avatar May 02 '20 06:05 liuzeng01


func main() {
	startURLs := []string{
		"http://dmoztools.net/Computers/Programming/Languages/Python/Books/",
		"http://dmoztools.net/Computers/Programming/Languages/Python/Resources/",
	}

	quitCh := make(chan struct{})
	crawler := &antch.Crawler{Exit: quitCh}
	spdier := antch.HandlerFunc(func(c chan<- antch.Item, _ *http.Response) {
		c <- nil
	})
	crawler.Handle("*", spdier)

	works := 0
	crawler.UsePipeline(func(next antch.PipelineHandler) antch.PipelineHandler {
		return antch.PipelineHandlerFunc(func(v antch.Item) {
			works++
			if works == len(startURLs) {
				close(quitCh)
			}
		})
	})

	go func() {
		crawler.StartURLs(startURLs)
	}()
	<-crawler.Exit
	fmt.Println("all done")
}

zhengchun avatar May 02 '20 16:05 zhengchun