jaxon icon indicating copy to clipboard operation
jaxon copied to clipboard

Support JSON streaming (LDJSON, NDJSON, JSONL)

Open spl opened this issue 3 years ago • 0 comments

There are people who think it's a good idea to “stream” JSON objects. (This is not an Elixir stream; it's string input composed of one JSON object after another with newlines or other whitespace between them.) The JSON streaming describes a few approaches. I wonder if something like the following would be a good addition to Jaxon. I needed this in my project. If it's not useful to add, at least it might help others:

  defp from_object_stream(events) do
    reversed =
      Enum.reduce(events, [:start_array], fn event, acc ->
        case {hd(acc), event} do
          {:end_object, :start_object} -> [:start_object | [:comma | acc]]
          _ -> [event | acc]
        end
      end)
    :lists.reverse([:end_array | reversed])
  end

  defp decode_json_stream(string) do
    case Jaxon.Parser.parse(string) do
      {:ok, events} -> Jaxon.Decoders.Value.decode(from_object_stream(events))
      {:incomplete, _, string} -> {:error, "Incomplete JSON value: #{string}"}
      {:error, _} -> {:error, "JSON parsing error: #{string}"}
    end
  end

spl avatar Aug 27 '21 12:08 spl