feat(client): JsonSerializable and JsonDeserialized

[jason-ha]

Add pair of type filters for JSON based serialization.

JsonSerializable<T> produces type representing limitations of serializing T. Incompatible elements are transformed to never or SerializationError* types that orignal T is not assignable to.

JsonDeserialized<T> produces type representing result of T being serialized and then deserialized.

JsonSerializable should eventually replace @fluidframework/datastore-definitions's Jsonable. That cannot done be currently as it would be a compile-time breaking change.

AB#6887

Supporting Changes

Add standard test infrastructure

[msfluid-bot]

⯅ @fluid-example/bundle-size-tests: +245 Bytes

Metric Name	Baseline Size	Compare Size	Size Diff
aqueduct.js	460.2 KB	460.24 KB	⯅ +35 Bytes
azureClient.js	558.17 KB	558.21 KB	⯅ +49 Bytes
connectionState.js	680 Bytes	680 Bytes	■ No change
containerRuntime.js	260.96 KB	260.97 KB	⯅ +14 Bytes
fluidFramework.js	401.36 KB	401.37 KB	⯅ +14 Bytes
loader.js	134.24 KB	134.25 KB	⯅ +14 Bytes
map.js	42.44 KB	42.44 KB	⯅ +7 Bytes
matrix.js	146.59 KB	146.59 KB	⯅ +7 Bytes
odspClient.js	525.47 KB	525.52 KB	⯅ +49 Bytes
odspDriver.js	97.72 KB	97.74 KB	⯅ +21 Bytes
odspPrefetchSnapshot.js	42.78 KB	42.79 KB	⯅ +14 Bytes
sharedString.js	163.28 KB	163.29 KB	⯅ +7 Bytes
sharedTree.js	391.82 KB	391.83 KB	⯅ +7 Bytes
Total Size	3.3 MB	3.3 MB	⯅ +245 Bytes

---

Baseline commit: cfa9c386eff95a159941dee41fa37e0d39604035

Generated by :no_entry_sign: dangerJS against 93feaabc9a27f16e86d8185c110f6a1b3aa20930

[jason-ha]

@CraigMacomber and/or @anthony-murphy, would you be able to find time to review these changes? (Per Daniel's suggestion as he has run out of time before vacation.)

[changeset-bot[bot]]

⚠️ No Changeset found

Latest commit: 56bc86498db7ef06e13874c9124320af122ba6d8

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[jason-ha]

Re-opening as refactor from Presence which has been using these types. There is no direct exposure of these types as they are meant for system use.

[pragya91]

@jason-ha , for "JsonSerializable should eventually replace @fluidframework/datastore-definitions's Jsonable" , we should create a backlog item to track it, with details on which version it would be viable to do so.

[markfields]

It would be a cool follow-up to have JsonString<T> type that is type branded string that implies that if you parse it you'll get JsonDeserialized<T> or something (not sure exactly how to leverage your types here). And then we write strongly-typed wrappers for JSON.stringify and JSON.parse to yield/use JsonString so we can have strong typing even after serialization.

I've prototyped this a few times, would be very useful in ContainerRuntime layer where we pass around stringified stuff all the time.

[jason-ha]

It would be a cool follow-up to have JsonString type that is type branded string that implies that if you parse it you'll get JsonDeserialized or something (not sure exactly how to leverage your types here). And then we write strongly-typed wrappers for JSON.stringify and JSON.parse to yield/use JsonString so we can have strong typing even after serialization.

I've prototyped this a few times, would be very useful in ContainerRuntime layer where we pass around stringified stuff all the time.

Maybe I can FHL it. Something like JsonString<T> should be enough and parser/encoder can qualify what is supported for handles and such in their declarations. function encodeWithHandles<T>(v: JsonSerializable<T, { AllowExtensionOf: IFluidHandle }>): JsonString<T> function decodeWithHandles<T>(v: JsonString<T>): JsonDeserialized<T, { AllowExtensionOf: IFluidHandle }> (JsonDeserialized<T, { AllowExtensionOf: [IFluidHandle] }> will just be T for perfectly round-trippable T.)

[jason-ha]

@jason-ha , for "JsonSerializable should eventually replace @fluidframework/datastore-definitions's Jsonable" , we should create a backlog item to track it, with details on which version it would be viable to do so.

Agreed. If the internal uses continue to look good, we could make a change for 3.0. (The Pages codebase may not be able to transition before - there was some cleanup needed when I made fixes to Jsonable a while back.)

[jason-ha]

It would be a cool follow-up to have JsonString type that is type branded string that implies that if you parse it you'll get JsonDeserialized or something (not sure exactly how to leverage your types here). And then we write strongly-typed wrappers for JSON.stringify and JSON.parse to yield/use JsonString so we can have strong typing even after serialization. I've prototyped this a few times, would be very useful in ContainerRuntime layer where we pass around stringified stuff all the time.

Maybe I can FHL it. Something like JsonString<T> should be enough and parser/encoder can qualify what is supported for handles and such in their declarations. function encodeWithHandles<T>(v: JsonSerializable<T, { AllowExtensionOf: IFluidHandle }>): JsonString<T> function decodeWithHandles<T>(v: JsonString<T>): JsonDeserialized<T, { AllowExtensionOf: IFluidHandle }> (JsonDeserialized<T, { AllowExtensionOf: [IFluidHandle] }> will just be T for perfectly round-trippable T.)

I worked on this during FHL which found some interesting use cases (branded strings and explicit unknown), that resulted in all of the additional changes this last week. That FHL work is not complete, but I don't think there will be needs for additional changes to implementation. Going to strive to stop making any more changes outside of critical code review feedback.

[jason-ha]

All of the tests:

• JsonDeserialized

  • positive compilation tests
    • supported primitive types are preserved -✔ boolean -✔ number -✔ string -✔ numeric enum -✔ string enum -✔ const heterogenous enum -✔ computed enum -✔ branded number -✔ branded string
    • unions with unsupported primitive types preserve supported types -✔ string | symbol -✔ bigint | string -✔ bigint | symbol -✔ number | bigint | symbol
    • supported literal types are preserved -✔ true -✔ false -✔ 0 -✔ "string" -✔ null -✔ object with literals -✔ array of literals -✔ tuple of literals -✔ specific numeric enum value -✔ specific string enum value -✔ specific const heterogenous enum value -✔ specific computed enum value
    • arrays -✔ array of supported types (numbers) are preserved -✔ sparse array is filled in with null -✔ array of partially supported (numbers or undefined) is modified with null -✔ array of unknown becomes array of JsonTypeWith<never> -✔ array of partially supported (bigint or basic object) becomes basic object only -✔ array of partially supported (symbols or basic object) is modified with null -✔ array of unsupported (bigint) becomes never[] -✔ array of unsupported (symbols) becomes null[] -✔ array of unsupported (functions) becomes null[] -✔ array of functions with properties becomes ({...}|null)[] -✔ array of objects and functions becomes ({...}|null)[] -✔ array of bigint | symbol becomes null[] -✔ array of number | bigint | symbol becomes (number|null)[] -✔ readonly array of supported types (numbers) are preserved
    • fully supported object types are preserved
      • ✔ empty object
      • ✔ object with boolean
      • ✔ object with number
      • ✔ object with string
      • ✔ object with number key
      • ✔ object with array of supported types (numbers) are preserved
      • ✔ object with sparse array is filled in with null
      • ✔ object with branded number
      • ✔ object with branded string
      • ✔ string indexed record of numbers
      • ✔ string|number indexed record of strings
      • ✔ string indexed record of number|strings with known properties
      • ✔ string|number indexed record of strings with known number property (unassignable)
      • ✔ Partial<> string indexed record of numbers
      • ✔ templated record of numbers
      • ✔ Partial<> templated record of numbers
      • ✔ object with possible type recursion through union
      • ✔ object with optional type recursion
      • ✔ object with deep type recursion
      • ✔ object with alternating type recursion
      • ✔ simple json (JsonTypeWith<never>)
      • ✔ non-const enum
      • ✔ object with readonly
      • ✔ object with getter implemented via value
      • ✔ object with setter implemented via value
      • ✔ object with matched getter and setter implemented via value
      • ✔ object with mismatched getter and setter implemented via value
      • class instance -✔ with public data (propagated)
      • object with optional property (remains optional) -✔ without property -✔ with undefined value (property is removed in value) -✔ with defined value
    • partially supported object types are modified -✔ object (plain object) becomes non-null Json object
    • fully unsupported properties are removed -✔ object with exactly bigint -✔ object with exactly symbol -✔ object with exactly function -✔ object with exactly Function | symbol -✔ object with inherited recursion extended with unsupported properties -✔ object with required exact undefined -✔ object with optional exact undefined -✔ object with exactly never -✔ string indexed record of undefined -✔ string indexed record of undefined and known number property (unassignable)
    • partially unsupported properties become optional for those supported
      • ✔ object with exactly string | symbol
      • ✔ object with exactly bigint | string
      • ✔ object with exactly bigint | symbol
      • ✔ object with exactly number | bigint | symbol
      • ✔ object with symbol key
      • ✔ object with recursion and symbol unrolls 4 times and then has generic Json
      • ✔ object with exactly function with properties
      • ✔ object with exactly object and function
      • ✔ object with function object with recursion
      • ✔ object with object and function with recursion
      • ✔ object with required unknown in recursion when unknown is allowed unrolls 4 times with optional unknown
      • object with undefined -✔ with undefined value -✔ with defined value
    • partially supported array properties are modified like top-level arrays -✔ object with array of partially supported (numbers or undefined) is modified with null -✔ object with array of unknown becomes array of JsonTypeWith<never> -✔ object with array of partially supported (bigint or basic object) becomes basic object only -✔ object with array of partially supported (symbols or basic object) is modified with null -✔ object with array of unsupported (bigint) becomes never[] -✔ object with array of unsupported (symbols) becomes null[] -✔ object with array of unsupported (functions) becomes null[] -✔ object with array of functions with properties becomes ({...}|null)[] -✔ object with array of objects and functions becomes ({...}|null)[] -✔ object with array of bigint | symbol becomes null[] -✔ object with array of number | bigint | symbol becomes (number|null)[] -✔ object with readonly array of supported types (numbers) are preserved
    • function & object intersections preserve object portion -✔ function with properties -✔ object and function -✔ function with class instance with private data -✔ function with class instance with public data -✔ class instance with private data and is function -✔ class instance with public data and is function -✔ function object with recursion -✔ object and function with recursion
    • class instance methods and non-public properties are removed
      • ✔ with public method (removes method)
      • ✔ with private method (removes method)
      • ✔ with private getter (removes getter)
      • ✔ with private setter (removes setter)
      • ✔ with private data (hides private data that propagates)
      • ✔ object with recursion and handle unrolls 4 times listing public properties and then has generic Json
      • for common class instance of -✔ Map -✔ ReadonlyMap -✔ Set -✔ ReadonlySet
    • branded non-primitive types lose branding -✔ branded object becomes just empty -✔ branded object with string
    • unsupported object types
      • known defect expectations
        • ✔ array of numbers with holes
        • getters and setters preserved but do not propagate -✔ object with readonly implemented via getter -✔ object with getter -✔ object with setter -✔ object with matched getter and setter -✔ object with mismatched getter and setter
  • negative compilation tests
    • assumptions -✔ const enums are never readable
    • unsupported types -✔ undefined becomes never -✔ unknown becomes JsonTypeWith<never> -✔ string indexed record of unknown replaced with JsonTypeWith<never> (and becomes optional per current TS behavior) -✔ templated record of unknown replaced with JsonTypeWith<never> (and becomes optional per current TS behavior) -✔ string indexed record of unknown and known properties has unknown replaced with JsonTypeWith<never> (and becomes optional per current TS behavior) -✔ string indexed record of unknown and optional known properties has unknown replaced with JsonTypeWith<never> (and becomes optional per current TS behavior) -✔ string indexed record of unknown and required known unknown has all unknown replaced with JsonTypeWith<never> (and known becomes explicitly optional) -✔ string indexed record of unknown and optional known unknown has all unknown replaced with JsonTypeWith<never> (and becomes optional per current TS behavior) -✔ Partial<> string indexed record of unknown replaced with JsonTypeWith<never> -✔ Partial<> string indexed record of unknown and known properties has unknown replaced with JsonTypeWith<never> -✔ symbol becomes never -✔ unique symbol becomes never -✔ bigint becomes never -✔ function becomes never -✔ void becomes never
  • special cases
    • ✔ explicit any generic limits result type
    • using alternately allowed types
      • are preserved -✔ bigint -✔ object with bigint -✔ object with optional bigint -✔ array of bigints -✔ array of bigint or basic object -✔ object with specific function -✔ IFluidHandle -✔ object with IFluidHandle -✔ object with IFluidHandle and recursion -✔ unknown -✔ object with optional unknown -✔ object with optional unknown and recursion -✔ string indexed record of unknown -✔ templated record of unknown -✔ string indexed record of unknown and known properties -✔ string indexed record of unknown and optional known properties -✔ string indexed record of unknown and optional known unknown -✔ Partial<> string indexed record of unknown -✔ Partial<> string indexed record of unknown and known properties -✔ array of unknown -✔ object with array of unknown
      • still modifies required unknown to become optional -✔ object with required unknown -✔ object with required unknown adjacent to recursion -✔ mixed record of unknown -✔ string indexed record of unknown and required known unknown
      • continue rejecting unsupported that are not alternately allowed -✔ unknown (simple object) becomes JsonTypeWith<bigint> -✔ unknown (with bigint) becomes JsonTypeWith<bigint> -✔ symbol still becomes never -✔ object (plain object) still becomes non-null Json object

• JsonSerializable

  • positive compilation tests
    • supported primitive types -✔ boolean -✔ number -✔ string -✔ numeric enum -✔ string enum -✔ const heterogenous enum -✔ computed enum -✔ branded number -✔ branded string
    • supported literal types -✔ true -✔ false -✔ 0 -✔ "string" -✔ null -✔ object with literals -✔ array of literals -✔ tuple of literals -✔ specific numeric enum value -✔ specific string enum value -✔ specific const heterogenous enum value -✔ specific computed enum value
    • supported array types -✔ array of numbers -✔ readonly array of numbers
    • supported object types
      • ✔ empty object
      • ✔ object with never
      • ✔ object with boolean
      • ✔ object with number
      • ✔ object with string
      • ✔ object with number key
      • ✔ object with array of numbers
      • ✔ readonly array of numbers
      • ✔ object with branded number
      • ✔ object with branded string
      • ✔ string indexed record of numbers
      • ✔ string|number indexed record of strings
      • ✔ templated record of numbers
      • ✔ string indexed record of number|strings with known properties
      • ✔ string|number indexed record of strings with known number property (unassignable)
      • ✔ object with possible type recursion through union
      • ✔ object with optional type recursion
      • ✔ object with deep type recursion
      • ✔ object with alternating type recursion
      • ✔ simple json (JsonTypeWith)
      • ✔ non-const enums
      • ✔ object with readonly
      • ✔ object with getter implemented via value
      • ✔ object with setter implemented via value
      • ✔ object with matched getter and setter implemented via value
      • ✔ object with mismatched getter and setter implemented via value
      • class instance
        • ✔ with public data (just cares about data)
        • with ignore-inaccessible-members -✔ with private method ignores method -✔ with private getter ignores getter -✔ with private setter ignores setter
      • object with optional property -✔ without property -✔ with undefined value -✔ with defined value
    • unsupported object types
      • ✔ object with self reference throws on serialization
      • known defect expectations
        • ✔ sparse array of supported types
        • ✔ object with sparse array of supported types
        • getters and setters allowed but do not propagate -✔ object with readonly implemented via getter -✔ object with getter -✔ object with setter -✔ object with matched getter and setter -✔ object with mismatched getter and setter
        • class instance
          • with ignore-inaccessible-members -✔ with private data ignores private data (that propagates)
  • negative compilation tests
    • assumptions -✔ const enums are never readable
    • unsupported types cause compiler error
      • ✔ undefined
      • ✔ unknown
      • ✔ symbol
      • ✔ unique symbol
      • ✔ bigint
      • ✔ function
      • ✔ function with supported properties
      • ✔ object and function
      • ✔ object with function with supported properties
      • ✔ object with object and function
      • ✔ function with class instance with private data
      • ✔ function with class instance with public data
      • ✔ class instance with private data and is function
      • ✔ class instance with public data and is function
      • ✔ object (plain object)
      • ✔ void
      • ✔ branded object
      • ✔ branded object with string
      • unions with unsupported primitive types -✔ string | symbol -✔ bigint | string -✔ bigint | symbol -✔ number | bigint | symbol
      • array -✔ array of bigints -✔ array of symbols -✔ array of unknown -✔ array of functions -✔ array of functions with properties -✔ array of objects and functions -✔ array of number | undefineds -✔ array of bigint or basic object -✔ array of symbol or basic object -✔ array of bigint | symbols -✔ array of number | bigint | symbols
      • object
        • ✔ object with exactly bigint
        • ✔ object with optional bigint
        • ✔ object with exactly symbol
        • ✔ object with optional symbol
        • ✔ object with exactly function
        • ✔ object with exactly Function | symbol
        • ✔ object with exactly string | symbol
        • ✔ object with exactly bigint | string
        • ✔ object with exactly bigint | symbol
        • ✔ object with exactly number | bigint | symbol
        • ✔ object with array of bigints
        • ✔ object with array of symbols
        • ✔ object with array of unknown
        • ✔ object with array of functions
        • ✔ object with array of functions with properties
        • ✔ object with array of objects and functions
        • ✔ object with array of number | undefineds
        • ✔ object with array of bigint or basic object
        • ✔ object with array of symbol or basic object
        • ✔ object with array of bigint | symbols
        • ✔ object with symbol key
        • ✔ string indexed record of unknown
        • ✔ Partial<> string indexed record of unknown
        • ✔ Partial<> string indexed record of numbers
        • ✔ Partial<> templated record of numbers
        • ✔ object with recursion and symbol
        • ✔ function object with recursion
        • ✔ object and function with recursion
        • ✔ nested function object with recursion
        • ✔ nested object and function with recursion
        • ✔ object with inherited recursion extended with unsupported properties
        • object with undefined -✔ as exact property type -✔ in union property -✔ as exact property type of string indexed record -✔ as exact property type of string indexed record intersected with known number property (unassignable) -✔ as optional exact property type > varies by exactOptionalPropertyTypes setting -✔ under an optional property
        • object with required unknown even though exactly allowed -✔ as exact property type -✔ as exact property type adjacent to recursion -✔ as exact property type in recursion
        • of class instance -✔ with private data -✔ with private method -✔ with private getter -✔ with private setter -✔ with public method
      • common class instances -✔ Map -✔ ReadonlyMap -✔ Set -✔ ReadonlySet
  • special cases
    • ✔ explicit any generic still limits allowed types
    • number edge cases
      • supported -✔ MIN_SAFE_INTEGER -✔ MAX_SAFE_INTEGER -✔ MIN_VALUE -✔ MAX_VALUE
      • resulting in null -✔ NaN -✔ +Infinity -✔ -Infinity
    • using alternately allowed types
      • are supported -✔ bigint -✔ object with bigint -✔ object with optional bigint -✔ array of bigints -✔ array of bigint or basic object -✔ object with specific alternately allowed function -✔ IFluidHandle -✔ object with IFluidHandle -✔ object with IFluidHandle and recursion -✔ unknown -✔ array of unknown -✔ object with array of unknown -✔ object with optional unknown -✔ string indexed record of unknown -✔ templated record of unknown -✔ string indexed record of unknown and known properties -✔ string indexed record of unknown and optional known properties -✔ string indexed record of unknown and optional known unknown -✔ Partial<> string indexed record of unknown -✔ Partial<> string indexed record of unknown and known properties -✔ object with optional unknown adjacent to recursion -✔ object with optional unknown in recursion
      • continue rejecting unsupported that are not alternately allowed -✔ unknown (simple object) expects JsonTypeWith<bigint> -✔ unknown (with bigint) expects JsonTypeWith<bigint> -✔ symbol still becomes never -✔ object (plain object) still becomes non-null Json object -✔ object with non-alternately allowed too generic function -✔ object with non-alternately allowed too input permissive function -✔ object with non-alternately allowed more restrictive output function -✔ object with supported or non-supported function union -✔ string indexed record of unknown and required known unknown that must be optional -✔ mixed record of unknown

• JsonSerializable under exactOptionalPropertyTypes=true

  • negative compilation tests
    • unsupported types cause compiler error
      • object
        • object with undefined
          • ✔ as optional exact property type