Custom converters required to persist CookieJar and FlurlCookie with System.Text.Json in .NET 5.0

[gitfool]

Following up from https://github.com/tmenier/Flurl/issues/506#issuecomment-727280179.

Specifically, serializing CookieJar (and FlurlCookie) works without requiring custom converters, although Url is serialized as a nested complex type instead of as a simple string, which is not desirable.

However, deserializing the above types fails due to limited functionality and specific requirements of System.Text.Json.

FlurlRepro.zip

FlurlRepro output:

Serializing CookieJar...

Deserializing CookieJar...

System.NotSupportedException: The collection type 'Flurl.Http.CookieJar' is abstract, an interface, or is read only, and could not be instantiated and populated. Path: $ | LineNumber: 0 | BytePositionInLine: 1.
 ---> System.NotSupportedException: The collection type 'Flurl.Http.CookieJar' is abstract, an interface, or is read only, and could not be instantiated and populated.
   --- End of inner exception stack trace ---
   at System.Text.Json.ThrowHelper.ThrowNotSupportedException(ReadStack& state, Utf8JsonReader& reader, NotSupportedException ex)
   at System.Text.Json.ThrowHelper.ThrowNotSupportedException_CannotPopulateCollection(Type type, Utf8JsonReader& reader, ReadStack& state)
   at System.Text.Json.Serialization.Converters.IEnumerableOfTConverter`2.CreateCollection(Utf8JsonReader& reader, ReadStack& state, JsonSerializerOptions options)
   at System.Text.Json.Serialization.Converters.IEnumerableDefaultConverter`2.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.JsonSerializer.ReadCore[TValue](JsonConverter jsonConverter, Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.JsonSerializer.ReadCore[TValue](Utf8JsonReader& reader, Type returnType, JsonSerializerOptions options)
   at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, Type returnType, JsonSerializerOptions options)
   at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, JsonSerializerOptions options)
   at FlurlRepro.Program.Main() in D:\Devel\FlurlRepro\Program.cs:line 47

Deserializing IEnumerable<FlurlCookie>...

System.InvalidOperationException: Each parameter in constructor 'Void .ctor(System.String, System.String, System.String, System.Nullable`1[System.DateTimeOffset])' on type 'Flurl.Http.FlurlCookie' must bind to an object property or field on deserialization. Each parameter name must match with a property or field on the object. The match can be case-insensitive.
   at System.Text.Json.ThrowHelper.ThrowInvalidOperationException_ConstructorParameterIncompleteBinding(ConstructorInfo constructorInfo, Type parentType)
   at System.Text.Json.Serialization.Converters.ObjectWithParameterizedConstructorConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.Converters.IEnumerableDefaultConverter`2.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.JsonSerializer.ReadCore[TValue](JsonConverter jsonConverter, Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.JsonSerializer.ReadCore[TValue](Utf8JsonReader& reader, Type returnType, JsonSerializerOptions options)
   at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, Type returnType, JsonSerializerOptions options)
   at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, JsonSerializerOptions options)
   at FlurlRepro.Program.Main() in D:\Devel\FlurlRepro\Program.cs:line 59

CookieJar.json:

[
  {
    "OriginUrl": {
      "Scheme": "https",
      "UserInfo": "",
      "Host": "boardgamegeek.com",
      "Port": null,
      "Authority": "boardgamegeek.com",
      "Root": "https://boardgamegeek.com",
      "Path": "/login",
      "PathSegments": [
        "login"
      ],
      "Query": "",
      "QueryParams": [],
      "Fragment": "",
      "IsRelative": false,
      "IsSecureScheme": true
    },
    "DateReceived": "2020-11-13T23:22:11.4217197+00:00",
    "Name": "bggusername",
    "Value": "username",
    "Expires": "2020-12-13T23:22:11+00:00",
    "MaxAge": 2592000,
    "Domain": ".boardgamegeek.com",
    "Path": "/",
    "Secure": false,
    "HttpOnly": false,
    "SameSite": null
  },
  {
    "OriginUrl": {
      "Scheme": "https",
      "UserInfo": "",
      "Host": "boardgamegeek.com",
      "Port": null,
      "Authority": "boardgamegeek.com",
      "Root": "https://boardgamegeek.com",
      "Path": "/login",
      "PathSegments": [
        "login"
      ],
      "Query": "",
      "QueryParams": [],
      "Fragment": "",
      "IsRelative": false,
      "IsSecureScheme": true
    },
    "DateReceived": "2020-11-13T23:22:11.4246746+00:00",
    "Name": "bggpassword",
    "Value": "password",
    "Expires": "2020-12-13T23:22:11+00:00",
    "MaxAge": 2592000,
    "Domain": ".boardgamegeek.com",
    "Path": "/",
    "Secure": false,
    "HttpOnly": false,
    "SameSite": null
  }
]

[gitfool]

I think the issues are as follows:

• CookieJar derives from IReadOnlyCollection<FlurlCookie>
  • Perhaps change to derive from ICollection<FlurlCookie> instead?
• FlurlCookie ctor only has parameters for properties that are get-only after construction
  • Remove ctor and make all properties get-set?
• Url may need a custom converter anyway
  • Simply write and read as string

public sealed class UrlConverter : JsonConverter<Url>
{
    public override Url Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
        => Url.Parse(reader.GetString());

    public override void Write(Utf8JsonWriter writer, Url value, JsonSerializerOptions options)
        => writer.WriteStringValue(value);
}

...

var options = new JsonSerializerOptions { Converters = { new UrlConverter() }, WriteIndented = true };

CookieJar.json:

[
  {
    "OriginUrl": "https://boardgamegeek.com/login",
    "DateReceived": "2020-11-13T23:22:11.4217197+00:00",
    "Name": "bggusername",
    "Value": "username",
    "Expires": "2020-12-13T23:22:11+00:00",
    "MaxAge": 2592000,
    "Domain": ".boardgamegeek.com",
    "Path": "/",
    "Secure": false,
    "HttpOnly": false,
    "SameSite": null
  },
  {
    "OriginUrl": "https://boardgamegeek.com/login",
    "DateReceived": "2020-11-13T23:22:11.4246746+00:00",
    "Name": "bggpassword",
    "Value": "password",
    "Expires": "2020-12-13T23:22:11+00:00",
    "MaxAge": 2592000,
    "Domain": ".boardgamegeek.com",
    "Path": "/",
    "Secure": false,
    "HttpOnly": false,
    "SameSite": null
  }
]

Note: the cookies seem to be persisted in reverse order to how they were added in the code and I reordered the json.

[gitfool]

The following hypothetical patch to remove FlurlCookie ctor works:

From 100ae8f24b68b7e6c9716eb811d8768da96d7f56 Mon Sep 17 00:00:00 2001
From: Sean Fausett <[email protected]>
Date: Mon, 16 Nov 2020 14:40:12 +1300
Subject: [PATCH] Remove FlurlCookie ctor

---
 CookieCutter.cs |  2 +-
 CookieJar.cs    | 16 +++++++++++++---
 FlurlCookie.cs  | 21 +++------------------
 Program.cs      | 12 ++++++++++--
 4 files changed, 27 insertions(+), 24 deletions(-)

diff --git a/CookieCutter.cs b/CookieCutter.cs
index 0d47195..b2f3586 100644
--- a/CookieCutter.cs
+++ b/CookieCutter.cs
@@ -37,7 +37,7 @@ namespace Flurl.Http
 			foreach (var pair in GetPairs(headerValue))
 			{
 				if (cookie == null)
-					cookie = new FlurlCookie(pair.Name, Url.Decode(pair.Value.Trim('"'), false), url, DateTimeOffset.UtcNow);
+					cookie = new FlurlCookie { Name = pair.Name, Value = Url.Decode(pair.Value.Trim('"'), false), OriginUrl = url };
 
 				// ordinal string compare is both safest and fastest
 				// https://docs.microsoft.com/en-us/dotnet/standard/base-types/best-practices-strings#recommendations-for-string-usage
diff --git a/CookieJar.cs b/CookieJar.cs
index d51e563..e881431 100644
--- a/CookieJar.cs
+++ b/CookieJar.cs
@@ -22,9 +22,19 @@ namespace Flurl.Http
 		/// <param name="name">Name of the cookie.</param>
 		/// <param name="value">Value of the cookie.</param>
 		/// <param name="originUrl">URL of request that sent the original Set-Cookie header.</param>
-		/// <param name="dateReceived">Date/time that original Set-Cookie header was received. Defaults to current date/time. Important for Max-Age to be enforced correctly.</param>
-		public CookieJar AddOrReplace(string name, object value, string originUrl, DateTimeOffset? dateReceived = null) =>
-			AddOrReplace(new FlurlCookie(name, value.ToInvariantString(), originUrl, dateReceived));
+		public CookieJar AddOrReplace(string name, object value, string originUrl) =>
+			AddOrReplace(new FlurlCookie { Name = name, Value = value.ToInvariantString(), OriginUrl = originUrl });
+
+		/// <summary>
+		/// Adds a cookie to the jar or replaces one with the same Name/Domain/Path.
+		/// Throws InvalidCookieException if cookie is invalid.
+		/// </summary>
+		/// <param name="name">Name of the cookie.</param>
+		/// <param name="value">Value of the cookie.</param>
+		/// <param name="originUrl">URL of request that sent the original Set-Cookie header.</param>
+		/// <param name="dateReceived">Date/time that original Set-Cookie header was received. Important for Max-Age to be enforced correctly.</param>
+		public CookieJar AddOrReplace(string name, object value, string originUrl, DateTimeOffset dateReceived) =>
+			AddOrReplace(new FlurlCookie { Name = name, Value = value.ToInvariantString(), OriginUrl = originUrl, DateReceived = dateReceived });
 
 		/// <summary>
 		/// Adds a cookie to the jar or replaces one with the same Name/Domain/Path.
diff --git a/FlurlCookie.cs b/FlurlCookie.cs
index 55a32c5..8eb6b1b 100644
--- a/FlurlCookie.cs
+++ b/FlurlCookie.cs
@@ -39,37 +39,22 @@ namespace Flurl.Http
 
 		private bool _locked;
 
-		/// <summary>
-		/// Creates a new FlurlCookie.
-		/// </summary>
-		/// <param name="name">Name of the cookie.</param>
-		/// <param name="value">Value of the cookie.</param>
-		/// <param name="originUrl">URL of request that sent the original Set-Cookie header.</param>
-		/// <param name="dateReceived">Date/time that original Set-Cookie header was received. Defaults to current date/time. Important for Max-Age to be enforced correctly.</param>
-		public FlurlCookie(string name, string value, string originUrl = null, DateTimeOffset? dateReceived = null)
-		{
-			Name = name;
-			Value = value;
-			OriginUrl = originUrl;
-			DateReceived = dateReceived ?? DateTimeOffset.UtcNow;
-		}
-
 		/// <summary>
 		/// The URL that originally sent the Set-Cookie response header. If adding to a CookieJar, this is required unless
 		/// both Domain AND Path are specified.
 		/// </summary>
-		public Url OriginUrl { get; }
+		public Url OriginUrl { get; init; }
 
 		/// <summary>
 		/// Date and time the cookie was received. Defaults to date/time this FlurlCookie was created.
 		/// Important for Max-Age to be enforced correctly.
 		/// </summary>
-		public DateTimeOffset DateReceived { get; }
+		public DateTimeOffset DateReceived { get; init ; } = DateTimeOffset.UtcNow;
 
 		/// <summary>
 		/// The cookie name.
 		/// </summary>
-		public string Name { get; }
+		public string Name { get; init; }
 
 		/// <summary>
 		/// The cookie value.
diff --git a/Program.cs b/Program.cs
index f3ff8ca..c2ef31b 100644
--- a/Program.cs
+++ b/Program.cs
@@ -14,8 +14,12 @@ namespace FlurlRepro
         private static async Task Main()
         {
             var cookies = new CookieJar();
-            cookies.AddOrReplace(new FlurlCookie("bggusername", "username", "https://boardgamegeek.com/login", DateTimeOffset.Parse("2020-11-13T23:22:11.4217197+00:00"))
+            cookies.AddOrReplace(new FlurlCookie
             {
+                Name = "bggusername",
+                Value = "username",
+                OriginUrl = "https://boardgamegeek.com/login",
+                DateReceived = DateTimeOffset.Parse("2020-11-13T23:22:11.4217197+00:00"),
                 Expires = DateTimeOffset.Parse("2020-12-13T23:22:11+00:00"),
                 MaxAge = 2592000,
                 Domain = ".boardgamegeek.com",
@@ -24,8 +28,12 @@ namespace FlurlRepro
                 HttpOnly = false,
                 SameSite = null
             });
-            cookies.AddOrReplace(new FlurlCookie("bggpassword", "password", "https://boardgamegeek.com/login", DateTimeOffset.Parse("2020-11-13T23:22:11.4246746+00:00"))
+            cookies.AddOrReplace(new FlurlCookie
             {
+                Name = "bggpassword",
+                Value = "password",
+                OriginUrl = "https://boardgamegeek.com/login",
+                DateReceived = DateTimeOffset.Parse("2020-11-13T23:22:11.4246746+00:00"),
                 Expires = DateTimeOffset.Parse("2020-12-13T23:22:11+00:00"),
                 MaxAge = 2592000,
                 Domain = ".boardgamegeek.com",
-- 
2.29.2.windows.1

Notes:

• Using C# 9 init keyword to enforce same read-only property behavior post construction
• Requires UrlConverter above to avoid the following error (but this is already much simpler than CookieConverter!)

Serializing CookieJar...

Deserializing IEnumerable<FlurlCookie>...

System.NotSupportedException: Deserialization of types without a parameterless constructor, a singular parameterized constructor, or a parameterized constructor annotated with 'JsonConstructorAttribute' is not supported. Type 'Flurl.Url'. Path: $[0].OriginUrl | LineNumber: 2 | BytePositionInLine: 18.
 ---> System.NotSupportedException: Deserialization of types without a parameterless constructor, a singular parameterized constructor, or a parameterized constructor annotated with 'JsonConstructorAttribute' is not supported. Type 'Flurl.Url'.
   --- End of inner exception stack trace ---
   at System.Text.Json.ThrowHelper.ThrowNotSupportedException(ReadStack& state, Utf8JsonReader& reader, NotSupportedException ex)
   at System.Text.Json.ThrowHelper.ThrowNotSupportedException_DeserializeNoConstructor(Type type, Utf8JsonReader& reader, ReadStack& state)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.Converters.IEnumerableDefaultConverter`2.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.JsonSerializer.ReadCore[TValue](JsonConverter jsonConverter, Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.JsonSerializer.ReadCore[TValue](Utf8JsonReader& reader, Type returnType, JsonSerializerOptions options)
   at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, Type returnType, JsonSerializerOptions options)
   at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, JsonSerializerOptions options)
   at FlurlRepro.Program.Main() in D:\Devel\FlurlRepro\Program.cs:line 69

• UrlConverter can be wired up automatically in Flurl by applying JsonConverterAttribute to Url class

using System.Text.Json.Serialization;

[JsonConverter(typeof(UrlConverter))]
public class Url
{
}

@tmenier Not sure what to do with CookieJar due to its composition of ConcurrentDictionary. Let me know your thoughts about that and my other suggestions for FlurlCookie and Url.

[tmenier]

Doing a little spring cleaning and I see this slipped through the cracks long ago. Looks like you solved it for your case but I imagine it's still an "issue" in .NET 6? I doubt I'll make it a very high priority to be honest but at least I'll get it in the backlog so it's not completely off the radar.

[gitfool]

I haven't managed to test with .NET 6 yet but I'd expect the same issues.

[tmenier]

@gitfool, In hindsight, #758 probably could have been marked a duplicate of this issue (and I sort of forgot about this one to be honest), but in any event, I believe the solution I detailed there makes JSON-serialization problems with CookieJar and FlurlCookie moot. If the end goal is saving and reloading CookieJars, I think it's a cleaner and simpler solution. Would you agree?

[tmenier]

Closing because I'm pretty sure #758 addresses the need here. Feel free to re-open for further discussion if you disagree.

[gitfool]

@tmenier I played with the latest prerelease and it worked well re cookies: https://github.com/gitfool/BoardGameGeek.Dungeon/commit/f2cf3c3b42477f1cf6dc8fbff865f345a9d91a42

There was one strange side effect of using the latest prerelease though; the type returned by ReceiveJson<PlayResponse> threw an exception:

[16:06:18 INF] (Recorder) Logging play 2023-10-23: 1x 1
[16:06:18 DBG] (BggService) POST https://boardgamegeek.com/geekplay.php
Flurl.Http.FlurlParsingException: Response could not be deserialized to JSON: POST https://boardgamegeek.com/geekplay.php
     System.Text.Json.JsonException: The JSON value could not be converted to System.Int32. Path: $.playid | LineNumber: 0 | BytePositionInLine: 20.
          System.InvalidOperationException: Cannot get the value of a token type 'String' as a number.
            at void System.Text.Json.ThrowHelper.ThrowInvalidOperationException_ExpectedNumber(JsonTokenType tokenType)
            at bool System.Text.Json.Utf8JsonReader.TryGetInt32(out int value)
            at int System.Text.Json.Utf8JsonReader.GetInt32()
            at bool System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.ReadJsonAndSetMember(object obj, ref ReadStack state, ref Utf8JsonReader reader)
            at bool System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options, ref ReadStack state, out T value)
            at bool System.Text.Json.Serialization.JsonConverter`1.TryRead(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options, ref ReadStack state, out T value)
            at T System.Text.Json.Serialization.JsonConverter`1.ReadCore(ref Utf8JsonReader reader, JsonSerializerOptions options, ref ReadStack state)
       at void System.Text.Json.ThrowHelper.ReThrowWithPath(ref ReadStack state, in Utf8JsonReader reader, Exception ex)
       at T System.Text.Json.Serialization.JsonConverter`1.ReadCore(ref Utf8JsonReader reader, JsonSerializerOptions options, ref ReadStack state)
       at TValue System.Text.Json.JsonSerializer.ContinueDeserialize<TValue>(ref ReadBufferState bufferState, ref JsonReaderState jsonReaderState, ref ReadStack readStack, JsonTypeInfo jsonTypeInfo)
       at TValue System.Text.Json.JsonSerializer.ReadFromStream<TValue>(Stream utf8Json, JsonTypeInfo jsonTypeInfo)
       at T Flurl.Http.Configuration.DefaultJsonSerializer.Deserialize<T>(Stream stream)
       at async Task<T> Flurl.Http.FlurlResponse.GetJsonAsync<T>()
  at async Task<IFlurlResponse> Flurl.Http.FlurlClient.HandleExceptionAsync(FlurlCall call, Exception ex, CancellationToken token)
  at async Task<T> Flurl.Http.FlurlResponse.GetJsonAsync<T>()
  at async Task<T> Flurl.Http.ResponseExtensions.ReceiveJson<T>(Task<IFlurlResponse> response)
  at async Task<Play> BoardGameGeek.Dungeon.Services.BggService.LogUserPlayAsync(Play play) in BggService.cs:221
  at async Task<int> BoardGameGeek.Dungeon.LogPlayCommand.OnExecuteAsync(CommandContext context, Settings settings) in LogPlayCommand.cs:16
  at async Task<int> BoardGameGeek.Dungeon.AsyncCommandBase`1.ExecuteAsync(CommandContext context, TSettings settings) in AsyncCommandBase.cs:15

So I had to change the contract type and convert it to make it work: https://github.com/gitfool/BoardGameGeek.Dungeon/commit/dc01459832d38816aab6c75097ac0a827a83b036

If I capture the response as text using ReceiveString then this is what I see:

{"playid":"76959327","numplays":7,"html":"Plays: <a href=\"\/plays\/thing\/1?userid=12345\">7<\/a>"}

Did something else change in this area? 🤔

[gitfool]

Okay, so what changed in this area was obviously https://github.com/tmenier/Flurl/issues/517, and I'm a bit rusty in this space, sorry. 😊

It turns out System.Text.Json won't do automatic casting of strings to integers like Newtonsoft.Json does but this can be enabled with an attribute and JsonNumberHandling enum: https://github.com/gitfool/BoardGameGeek.Dungeon/commit/26331390d439a764be59462efb74d01a165a44a1 😅

[tmenier]

Excellent. Yeah I can see how the STJ switch may have flown under the radar if this is the first prerelease you've tried, I'll have a loud reminder of it for the full release. 😄