probe-scraper icon indicating copy to clipboard operation
probe-scraper copied to clipboard

Gracefully handle commits where files are missing from git

Open relud opened this issue 4 years ago • 0 comments

this code fails when there are commits where a file was deleted:

https://github.com/mozilla/probe-scraper/blob/c71f5dd8b0f4a98a9c2de83b38f22ff4e0b40205/probe_scraper/scrapers/git_scraper.py#L128 https://github.com/mozilla/probe-scraper/blob/c71f5dd8b0f4a98a9c2de83b38f22ff4e0b40205/probe_scraper/scrapers/git_scraper.py#L84-L85

the current workaround requires listing every commit where the file was deleted: https://github.com/mozilla/probe-scraper/blob/c71f5dd8b0f4a98a9c2de83b38f22ff4e0b40205/probe_scraper/scrapers/git_scraper.py#L35-L49 https://github.com/mozilla/probe-scraper/blob/c71f5dd8b0f4a98a9c2de83b38f22ff4e0b40205/probe_scraper/scrapers/git_scraper.py#L103 https://github.com/mozilla/probe-scraper/blob/c71f5dd8b0f4a98a9c2de83b38f22ff4e0b40205/probe_scraper/scrapers/git_scraper.py#L123-L124

That workaround is still needed for invalid files, but for missing files we should add code to handle the exception from git, probably with continue.

relud avatar Oct 27 '21 17:10 relud