langchaingo icon indicating copy to clipboard operation
langchaingo copied to clipboard

scraper: fix concurrent scraped links map access

Open vdpoora opened this issue 1 year ago • 0 comments

PR Checklist

  • [x] Read the Contributing documentation.
  • [x] Read the Code of conduct documentation.
  • [x] Name your Pull Request title clearly, concisely, and prefixed with the name of the primarily affected package you changed according to Good commit messages (such as memory: add interfaces for X, Y or util: add whizzbang helpers).
  • [x] Check that there isn't already a PR that solves the problem the same way to avoid creating a duplicate.
  • [x] Provide a description in this PR that addresses what the PR is solving, or reference the issue that it solves (e.g. Fixes #123).
  • [ ] Describes the source of new concepts.
  • [ ] References existing implementations as appropriate.
  • [ ] Contains test coverage for new functions.
  • [x] Passes all golangci-lint checks.

When running the scraper, there would occasionally be panics for concurrent map access of the scrapedLinks map.
Adding a sync.RWMutex around all accesses of the map fixes this issue.

vdpoora avatar Apr 30 '24 14:04 vdpoora