JobFunnel
JobFunnel copied to clipboard
TFIDF content matching should check inter-scrape
Description
Currently we remove duplicates everywhere but we only remove duplicates by description (TFIDF) between the masterlist and all scrape data.
We should allow masterlist to perform a content match to itself.
Steps to Reproduce
- scrape some jobs to .pkl
- copy-paste a row a few times, only changing the key_id
- run again with --no-scrape
Expected behavior
We should be running TFIDF inter-scrape data and inter-master csv
Actual behavior
Only duplicates in the ncoming dict are identified based on master CSV
Environment
- Build: 3.0.0
Thinking about this... we can simply identify matches in the backwards direction after we perform the matching in the forwards direction? (Currently its incoming job vs master dict, but we want master job vs master dict)
Accidentally tapped close...