difftastic
difftastic copied to clipboard
Wilfred
Died of signal 9 after loading big JSON
When using DFT_BYTE_LIMIT=4000000 git diff I get part of the diff and then:
error: difft died of signal 9
fatal: external diff died, stopping at xxxxxx.json
I actually have a big JSON file (1.1MB / 40k lines) in LFS there. Usually git diff ignores LFS files (it just shows the checksum difference).
Strangely enough, I have a bigger Rust file (3.7MB / 51k lines) and it didn't have any issue.
Difftastic 0.31.0
One workaround I found for now is to use these arguments to ignore the json files: git diff -- . ':!*.json https://maheshwaghmare.com/git/how-to/exclude-files-from-git-diff-command/
Could you share a copy of the JSON file or the git repository? I can't reproduce the crash without it.
(I agree that maybe difftastic should be consistent for git-lfs, but it shouldn't crash.)
Sorry I can't... maybe if I find a way to anonymize the file somehow... :thinking: All I can say is the JSON is in multiline and has been generated with serde.
Though, at the look of it, it seems the program feels up the memory. I have 64G so it's quite surprising!
(I agree that maybe difftastic should be consistent for git-lfs, but it shouldn't crash.)
Actually, imo, it's a feature that it can shows the diff of files that are in LFS. It can be handy!
Oh, I hadn't noticed you've set DFT_BYTE_LIMIT. I'd guess that the JSON parser is OOMing. If you haven't increased DFT_GRAPH_LIMIT then the diffing logic should limit its memory.
Does it still die if you set a very low value of DFT_GRAPH_LIMIT?
With DFT_GRAPH_LIMIT=500000 the memory fills up very close to the limit BUT it worked :sweat_smile:
1/374 --- Text (exceeded DFT_GRAPH_LIMIT)
It is strange that for the Rust file that is even bigger it doesn't fill up the memory at all
Happy to revisit this if you get a repro, but closing as there's nothing I can do at this point.