Provide a way to stylize/colorize JsonWriter output

[tliron]

Problem solved by the enhancement

It is currently impossible to achieve ANSI colorization via the existing JsonWriter implementation.

Enhancement description

The issue is that the private should_escape function will escape ANSI codes and there does not seem to be a way to get around this.

A straightforward solution might be to add a flag to WriterSettings to not escape ANSI codes. However, I am not sure this is the best solution, because ANSI codes might be present in the actual data, and strictly speaking those should be escaped.

Instead, I would suggest adding a write_unescaped method to JsonWriter. This would allow users to wrap JsonStreamWriter and provide their own ANSI code output before/after writing the actual properly escaped content, e.g.:

impl JsonWriter for MyJsonWriter {
    ...
    // self.writer is a wrapped JsonStreamWriter
    fn string_value(&mut self, value: &str) -> Result<(), io::Error> {
        self.writer.write_unescaped("[some ansi codes"])?;
        self.writer.string_value(value)?;
        self.writer.write_unescaped("[some ansi codes"])
    }
}

Alternatives / workarounds

Writing is ostensibly handled by the JsonStringWriter trait. However, JsonStreamWriter does not provide a way to replace this implementation. There is an internal StringValueWriterImpl that calls a private write_string_value_piece, and no access to the underlying writer, which is owned by JsonStreamWriter.

A workaround would be writing an entire custom JsonWriter implementation.

[Marcono1234]

It is currently impossible to achieve ANSI colorization via the existing JsonWriter implementation. [...] The issue is that the private should_escape function will escape ANSI codes and there does not seem to be a way to get around this.

That is kind of intended; the JSON specification only permits whitespace between JSON values, and within a JSON string value control characters (such as ANSI codes) must be escaped.

But I guess for your use case these restrictions by the JSON specification don't matter because you don't want to store or transmit the JSON data, but instead want to display the colorized data to the user, right? Could you please describe your use case a bit more in detail? Note also that Struson currently does not support customizing pretty-printing; you can only enable or disable it (WriterSettings::pretty_print).

Do you only want to colorize complete JSON values, e.g. true in green, false in red, "test" in blue? Or do you want to colorize parts of a JSON string value? For example in {"fruit": "green apple"} colorize "green"?

• The former might be possible by adding a JsonStreamWriter::writer_mut, similar to the existing JsonStreamReader::reader_mut (though would somehow need a way to flush the internal writer buffer; maybe be doing it implicitly when writer_mut is called).
• The latter will probably be difficult without exposing implementation details. As you mentioned a solution might be to add a WriterSettings option to not escape 0x001B (ESC), but I am not sure about this at the moment, because it would technically emit invalid JSON data and would only be useful for this use case.

[tliron]

You are correct, my use case is presenting JSON to the terminal. So, "pretty" here would mean both indentation and colors.

Also, too your quesiton, my use case is colorizing "complete" JSON values, as you say, but I wonder if this flexibility should be limited to only my use case. There might be value in using Struson in various other environments, i.e. an IDE where you want to automatically underline URLs in any span of a JSON string.

As you point out (I did, too) turning off all escaping is not the best solution. However, it's also not bad once I think of it a bit more. In some cases the data you are writing has already been "sanitized" elsewhere in the code, so you know it's safe for JSON. In such cases you can save some energy by not escaping when you know you don't have to. And it would be easy to implement, a simple boolean.

Better, perhaps, could there be a way to configure exactly which codes to escape? So once can turn off escaping of just ANSI codes, for example.

I have to say, the current architecture is a bit hard for me to follow. I'm not sure what value a separate StringValueWriterImpl provides when it calls internal private methods in JsonStreamWriter. Could you explain the intent? Perhaps I can suggest a way around this. The "best" solution in my view is to allow users to "hook" their own decoration/transformer of the JSON data before it is sent to the writer. I think that's where escaping should happen, too. In your implementation you escape at the lower level, during writing, and that level is inaccessible to users.

[Marcono1234]

an IDE where you want to automatically underline URLs in any span of a JSON string

I think this is not something I am going to support (at least not now). That would probably require exposing an escaping API just for this use case. IDEs normally use special parsers anyway I think, which for example provide better diagnostics for parsing errors, get the position of individual tokens to add syntax highlighting for them, and continue parsing after errors so that syntax highlighting and error detection does not stop after the first parse error. That all is a use case which Struson is currently not designed to cover.

Your use case of colorizing full JSON values seems less extensive and probably easier to support.

---

In your implementation you escape at the lower level, during writing, and that level is inaccessible to users.

That is mostly intentional: The main focus is on the represented JSON data, and there it does not matter whether characters in string values are escaped or not (unless of course the JSON specification requires it), the represented data is identical. That WriterSettings offers escape_all_control_chars and escape_all_non_ascii is already a "bonus" feature which many JSON libraries don't offer, and which is not needed to be compliant with the JSON specification. It is rarely needed to control which other characters are escaped, except maybe for use cases where a checksum is calculated over the JSON data, or when the JSON data is embedded in other data. For example the Java library Gson which Struson's API is partially based on has a feature to escape HTML chars, but I think that is not so relevant nowadays anymore, because the consumer of the JSON data is expected to properly escape it for their use case. For Struson I guess you could achieve this by creating a JsonStreamWriter with a Write which escapes for example all HTML chars before writing them to the underlying writer.

If really needed, maybe I can adjust Struson to further allow customizing which chars to escape during writing, but for your use case it might be possible without this.

---

I have to say, the current architecture is a bit hard for me to follow. I'm not sure what value a separate StringValueWriterImpl provides when it calls internal private methods in JsonStreamWriter. Could you explain the intent?

There might be some misunderstanding here: When I and the documentation mentions "JSON string value", it really only talks about a single "string" value, a part of a complete JSON document. For example in {"name": "value"} the "value" is a "JSON string value". I think this terminology matches what is used by the specification (using the terms "value" and "string"), but it can cause confusion because often the complete JSON data might also be referred to as "JSON string".

Normally you would write these JSON string values with JsonWriter::string_value. However, there are use cases where the string value is huge, e.g. users sometimes store large binary data such as images as Base64 string in JSON. In these cases constructing a String first would be quite memory intensive. Therefore Struson provides JsonWriter::string_value_writer (for which StringValueWriterImpl is part of the implementation); it is a Write which writes the content of a single JSON string value (and only that, no other values) and performs the necessary escaping. For example something like this:

let mut json_writer = JsonStreamWriter::new(...);

let mut string_writer = json_writer.string_value_writer()?;
let mut base64_writer = base64::write::EncoderWriter::new(string_writer, &general_purpose::STANDARD);
// ... write binary data to base64_writer
base64_writer.finish()?;

string_writer.finish_value()?;
json_writer.finish_document()?;

That is also why StringValueWriterImpl is probably not what you are looking for if you want to colorize complete JSON values, and also values other than strings.

[Marcono1234]

Since you only want to colorize complete JSON values, maybe this could be achieved by adding a custom PrettyPrinter trait. The main purpose would be to have more control over indentation and line wrapping, but it could also be used to insert ANSI codes before and after JSON values. What do you think; would that cover your use case? That would probably be similar to your "decoration/transformer" idea.

I will try to create a first draft for it. It will however only be possible to insert text (e.g. indentation or ANSI codes) before and after JSON values (with information about their types, e.g. "boolean", "array", ...); it will not be possible to inspect the values or to modify them before being written.

[tliron]

Thank you for the detailed responses, I look forward to see your implementation.

And I'm sorry, but I still fail to understand the architecture choice. I do understand the different between writing strings and JSON "string values", but you've intermingled JsonStreamWriter and StringValueWriterImpl (which is private) in a way that makes them hard if not impossible to customize. The core issue is that you made your useful escaping code private (write_string_value_piece, write_escaped_char, etc.) so if I want to implement my own custom JsonWriter I would have to copy and paste your code into mine, which is unmaintainable.

I think the best outcome would be to allow users to create their own MyJsonWriter that builds upon the features you provide.

I understand if that's not your goal, but I think it's not far from reach with some redesign.

[Marcono1234]

so if I want to implement my own custom JsonWriter I would have to copy and paste your code into mine, which is unmaintainable

I was hoping that JsonStreamWriter is covering all use cases for writing to a Write. And that implementing your own JsonWriter is rather for cases where you write data to something other than a Write (see for example the custom_json_writer.rs test), or maybe for delegating to a wrapped JsonStreamWriter.

I guess if really needed I could expose write_escaped_char as utility function, but I am not completely sure.

If you want, feel free to open a separate GitHub Issue (or maybe better GitHub Discussion) outlining the API changes you were thinking of, and maybe also explaining the use case why implementing a custom JsonWriter which writes to a Write is necessary (compared to using or wrapping a JsonStreamWriter). If I understand it correctly that is not directly related to your use case of colorizing JSON output here? But I cannot make any promises whether I will make those changes then; I will have to see how extensive they are, and if there aren't other ways to support the use case.

[tliron]

You are correct, I am suggesting an architecture that goes beyond this issue.

To give context, I'm working on porting a library that I originally wrote in Go to Rust. Here it is. Work in progress. :) The actual use of Struson for serialization is here. For parsing, Struson has been wonderful for me, thank you so much for this library.

I might implement my own JsonWriter and copy paste some of your code (with attribution). If I do, I'll update here so you can see it.

[Marcono1234]

Sorry for the delay, a first draft of the pretty printer API is now available in the pretty-printer branch. There are still a few open questions and I am not completely sure whether I will add this API in the end. But any feedback would be useful nonetheless!

In that branch the tests/writer_pretty_printer_test.rs colorizing_pretty_printer shows how you could use it to colorize output, hopefully that is similar to what you had in mind.

[tliron]

Just a quick ping that I did not forget to check your work, just had to devote time to some other things for a while. Will get to it eventually!