evine
evine copied to clipboard
Evine will occasionally crash while fetching a lot of results
The fatal error is fatal error: concurrent map writes (generally crashes before reaching 1k results)
This does not happen when running evine with only a single thread
The error occurs in line 1417 of evine.go
// getSource run with each URL and extract links from the URL page
func getSource(uri string) ([]string, error) {
if !urlExcluding(uri) {
return []string{}, nil
}
time.Sleep(time.Duration(OPTIONS.Delay) * time.Millisecond)
text, statusCode, err := request(uri)
if err != nil {
return []string{}, err
}
if !statusCodeExcluding(statusCode) {
return []string{}, nil
}
putting(VIEWS_OBJ["RESPONSE"], " > " + uri)
RESULTS.Pages += text
RESULTS.PageByURL[uri] = text // < -- THE ERROR IS HERE
allURLs := getURLs(text)
allURLs = urlCategory(allURLs)
refStatusLine(fmt.Sprintf("[Elapsed:%fs] | [Obtained:%d] | [%s]", sinceTime(), len(RESULTS.URLs), uri))
return allURLs, nil
}
you have to add a Mutex.Lock(). Im going to make a pull request.
Made pull request that I have verified to fix the issue