disk store: bundle activation fails if data.json exceeds what we can write in a single txn

[srenatus]

With a large data.json file, we'll never be able to truncate the disk store on bundle activation: the value we attempt to write is larger than what a single txn can hold; and the commit-and-retry logic will fail: the commit is empty (no problem) but the retry with just yield the same error.

What we could do here is to enhance the bundle iterator to split values according to their value size: if it's above X, we'll add iteratees for the substructures. For objects,

{ "a": ..., "b": ... , ... }

we'll parse that JSON blob, and add iteratees for each value, with path = old_path + "a", path = old_path + "b", etc.

Analogously for arrays.

[stale[bot]]

This issue has been automatically marked as inactive because it has not had any activity in the last 30 days.

[ashutosh-narkar]

Closing this in favor of https://github.com/open-policy-agent/opa/issues/5721