Favored way of making JToken?

[drhumlen]

Hi!

I have to do a lot of back and forth to convert my events to the JToken type. Since Newtonsoft's JsonConvert's default converter serializes fsharp types in a very bloated and unnatural way, I have to use FSharpLu.Json's serializer (see the difference here: https://github.com/Microsoft/fsharplu/wiki/fsharplu.json#option-type).

However, FSharpLu.Json convert object directly to string – which means I have to take the result of that and parse it with JToken.Parse to get the desired JToken type. That seems like a lot of unnecessary conversion back and forth. Probably not a huge deal for a modern CPU, but still creates a little unnecessary load – especially considering that our app will grow and eventual have a lot of events that needs processing.

So... Is there a way to pass it a string directly without parsing it first with JToken.Parse? Also, when I read the event, I have to do a .ToString() to convert it back to raw json, so that FSharpLu.Json can convert it to an object again.

Also, I'm really curious: How do you guys create Json? Do you use JsonConvert from Newtonsoft? How do you deal with the awkward json it makes for SumTypes & Options types etc.? Is there a better library than FSharpLu.Json? Chiron has zero documentation it seems, and all other json projects look pretty dead :( It seems that Json + F# is almost a little unsolved...?

[bartelink]

Assorted asides and red herrings:

• Re F# Json.NET converters: https://github.com/jet/Jet.JsonNet.Converters is in active use and works - some people seem to want things more complex by adding tuple converters and things. The only major hole I see is https://github.com/JamesNK/Newtonsoft.Json/issues/1827.

• Re what to use as a common form in a store lib:- In Equinox, the way we resolved the forces at play (EventStore wants UTF-8 arrays, Cosmos wants json.net rendering is to take inputs as byte[], and then bridging with this converter.

[TheAngryByrd]

Regarding the Option/Sumtypes, I added @baronfel's JSON.net converters to the Marten library in this PR recently. So at least in Postgres it will store them sanely.

The CosmoD's library also is using it's own OptionConverter and same with TableStore. It uses the corresponding serialize/deserialize helpers.

Probably would be useful to pick a converter set and go with it or maybe better to allow consumers to be able to pass in their own converters in the settings of a persistent store.

In the meantime you don't have to convert JTokens to string then deserialize them and back, you can use the ToObject<'a> and JToken.FromObject functions on them. You'll just need to make sure you're using the FsharpLu.Json converter

  let serializerSettings = JsonSerializer.CreateDefault(Compact.TupleAsArraySettings().settings)
  let inline toObject<'a> (token : JToken) =
      token.ToObject<'a>(serializerSettings)
  let inline fromObject (o : obj) =
      JToken.FromObject(o,serializerSettings)

This doesn't really fix the double serialization that will happen though and it might persist oddly in the store you're using.

[drhumlen]

Aha. Because it's not really just option that gets converted in a bad way – it's stuff like: type Color = Red | Blue | Green, that gets converted in a very unconventional way by Newtonsoft. I'd like for others to be able to make sense of our events too, not just Cosmostore/Newtonsoft. And that it is why I want a "sane"r json conversion like the one from FSharpLu.Json.

I guess all my problems would be solved if CosmoStore just didn't enforce that Data & Metadata was of type JToken; and just let it be a string instead. That way, you would be free to use whatever serialization you want; you could even use non-json stuff like protobuf if you wanted to.

[bartelink]

Remember CosmosDb ultimately wants json though - if you don't have that, you need to e.g. base64 it.

The problem with 'just' using a string is that the content is usually json, which json.net then needs to double encode "to be safe"

As noted above, UTF8 byte arrays with the verbatim json converter I linked can work well.

However, regardless of that, you should be able to use FSharpLu.json and/or Jet.JsonNetConverters to render to a JToken and then have CosmoStore emit it from there as it stands - its an extra hoop for you to jump through, and it may or may not be the most efficient solution, but ti works and means The CosmoStore impl doesnt need a VerbatimUtf8Converter thing

[kunjee17]

@drhumlen I guess that issue is solved. As of now it is very much store specific. Like liteDB is having bson value and Servicestack (dapper in future) is having long string (blob) value. While inmemory database is having object.

So, have a look and if there is question to any specific store I guess than we can always continue with that. :)