Git LFS files modified on another PR messing up runs on other branches

[simondelphia]

Using this action with lfs: true and I create a PR which modifies some images used for snapshot tests, subsequent runs on the same CI machine with different branches that didn't even touch those files will often fail because the image file is not loaded properly. Any suggestions for how to fix this or why this is happening?