Can't load dataset items in total size > 512MB

[valekjo]

Node has 512MB limit on string size, if the response from API is larger, the client fails to parse it.

I guess the issue might also manifest in key-value-stores? Other api endpoints probably never produce this large responses.

Reported on slack: https://apifier.slack.com/archives/C0L33UM7Z/p1652976377122799

[mnmkng]

I don't think this is a bug really. It's quite obvious that strings of unlimited sizes cannot be loaded or parsed. That's why the list functions of client have offsets and limits and the KVS functions have stream options.

I would vote for closing this.

[fnesveda]

@valekjo Where did this bug manifest? Could you link the Slack thread or another discussion where this was reported?

[fnesveda]

It would also be cool if we supported some client.paginate() function, which automatically iterates over the pages in a list returned by the API, like Octokit does it, but that's not really related to this issue specifically.