Doesn't work with large JSON files

[colegleason]

Trying to compare two files of size 701MB results in

FATAL ERROR: CALL_AND_RETRY_0 Allocation failed - process out of memory

[wanderer]

Also its really slow for large files

[andreyvit]

Patches are welcome! ;-)

[miranda-zhang]

My file is only 854K, but it seems to running forever, going to leave it overnight to see if it can finish...

[mistertest]

I think it's working more better in terminal mode than js implementation in browser. But I have not largest files to confirm.

[idanElitzur]

Any news regarding this? I'm facing with the same issue right now With 2 files of 20-30MB each one (almost 1M lines)

[philipborg]

I don't get an out of memory exception with my 60MB json files. It just never finishes and my job agents kills it after an hour, I left it running longer than that locally but after most of a day I called it quits. Performance seems to degrade exponentially with file-size as my 7MB files completes "nicely" after about 150 seconds comparatively. Would be nice if this performed at linear speed with file size.

[mrmianbao]

@philipborg I had the same issue.. it was working before I made changes to my tsconfig.

It just hangs.

@andreyvit I forget exactly what changes I made but it was something like changing my tsconfig to target: "esnext"

[philipborg]

I have found a workaround for my needs I thought I'd share. It seems that comparing large arrays of objects is the issue. Replacing the array with an object, so it only compares key matches, solved it for me. Now it goes decently fast even with 60MiB files. My objects themself contains arrays of strings which doesn't seem to cause any problems.

So instead of

[
    {
        "field1": 2,
        "field2": "foo",
        [...]
    },
    {
        "field1": 1337,
        "field2": "bar",
        [...]
    }
]

I used something more akin to

{
    "MyIdentifier1": {
        "field1": 2,
        "field2": "foo",
        [...]
    },
    "MyIdentifier2": {
        "field1": 1337,
        "field2": "bar",
        [...]
    }
}

This only works if you can find an unique identifier in the data to use as the object key. In my case I had to do a composite key and escape my separators. It worsens the comparison and requires you to mutate the json before comparing, so it's just a workaround and not an issue solution.

Essentially it restricts the search scope for what it will compare objects with as now it only compares objects with matching keys.