Serialization / ES integration re type names

[bartelink]

I want to build a service that has mutiple aggregates and build appropriate routing and/or splitting of processing into approriately separate agents using NES. While I could use the NES bucket to keep all the events for a given aggregate together and then have several indendent projections per bucket/aggregate, I believe that NES' checkpoint/token mechanism for building a projection is generally intended to operate across aggregates within a BC. (EDIT Confirmed in https://jabbr.net/#/rooms/DDD-CQRS-ES at https://jabbr.net/#/rooms/DDD-CQRS-ES 14:00 UTC on 2014-8-8)

(ASIDE: I will log a separate issue and/or stand up a PR which addesses the need for the code to be work with >1 DU/Aggregate -- i.e. at present there are explicit references to Game in Serialization?)

Context: json.net v6 DU support - http://james.newtonking.com/archive/2014/02/01/json-net-6-0-release-1-%E2%80%93-jsonpath-and-f-support

At present (assuming I read the code right), you

1. take the DU and are makign sure not to redundantly store the Case Name in the serialized body
2. are only storing the Case name in the ES type field

For the same kind of reasons that we arrived at the resolution that it is correct to redundantly maintain a GameIdwithin an Event in order to allow contextless processing and/or restreaming of events, I believe it would be more correct to:

1. store the full type name e.g. Namespace.Module.Events.DU+Case or similar as the ES type of the message
2. Make the serialization stuff work more with the type name and/or lean on the V6 features/capabilities vs dealing explicitly in Case Names as aprt of the serialization code.

Whether it is easy to get json.net and/or Serlalizer to instead of:

//{
//   "Case": "Rectangle",
//   "Fields": [
//     1.3,
//     10.0
//   ]
// }

yield/store:

//   [
//     1.3,
//     10.0
//   ]

and then use the ES type to guide the deserialization appropaitely is another matter.

Questions:

1. do you agree that the type should be more than the just the case
2. do you look at each aggregate as being independent from a projection POV (i.e. to you have a clear picture I'm just not aware of re e.g. having a 1 agg : 1 "Handler agent" : n "Projection agent"s in a system) ? (I guess I could divine that from the code)
3. would you be concerned with changing the code to use json.net native serializaiton and let it redundantly store the case (i.e. esp given 1. will already know the case) (Or can you see a quick way to resolve that?

I realize this is a wall of text but wanted to get my thoughs straightened out, Perhaps we need a https://jabbr.net/#/rooms/Fester to discuss Functional Event Sourcing [TERing] stuff like this in a more interactive manner :D

[bartelink]

As discussed in https://jabbr.net/#/rooms/FunctionalEventSourcing the key bit I missed is that I didnt read the stuff I typed above...

The json.net native DU serialization does not emit field names. This makes the result useless for GES but from my perspective it's a pretty brittle (reordering fields not supported).

What is as yet unresolved (either as code or as a journey article) is whether any BC/subsystem/aggregate metadata should be kept alongside the events (and the reasoning behind that).

i.e., for an event of type App.Service.Events.DU+Case :

• should the type be modified in any way (e.g. strip namespace?)
• should the Case be kept and be used as the (only) event name/type for routing purposes and if so how do we ensure they are not ambiguous when e.g. emitted to other streams in GES
  • should the name always have the Aggregate name included as part of this scheme as a way to make them context-free as part of good domain design anyway
  • is it better to have a module and an eventName and explicitly don't event maintain a type (or is it safe/appropriate to eg. put module|eventName into type ?)

(NES has a property bag and no particular constraints to I'd probably apply the same metadata arrangement in there)

[thinkbeforecoding]

From a DDD system point of view, the fact that the event was actually emitted by an aggregate is an implementation detail. You can spot this from the fact that all given events refer to the same aggregate but it doesn't really matter. When doing Event Storming, Aggregate definition comes late in the process, but Events still have their meaning.

Using only unambiguous Event names is this flavor.

But it can be practical for implementation to indicate more explicitly what aggregate produce them.. it's actually already quite explicitly/implicitly in the stream name, but it can be added either to the event name (like module.eventuate) or to metadata.

[bartelink]

I think in the jabbr you mentioned that refactoring can change the aggregate that will generate/note/emit a given event type too.

The problem is I want one answer to rule then all :)

If using a flat structure, the Domain should make the Events a single DU (i.e., it needs to move out of Game.fs) and FsUno.Domain.Game.Event+GameStarted will become FsUno.Domain.Event+GameStarted

So, in terms of proposing a metadata structure, does it make sense to say that instead of the serialization only coping with the case name (and being hardwired to the Event type) and using the full namespaced typename in the GES type there should be 2 schemes which one can choose without anyone being the magic blessed variant:

1. FsUno.Domain.Game.Event+GameStarted => type=Game.GameStarted
2. FsUno.Domain.Event+GameStarted => type=GameStarted

And the deserialization will need to deal with taking a 1 or 2 part type and allowing something external to map Game to the type FsUno.Domain.Game.Event (and the case name searching to find the concrete nested type (FsUno.Domain.Game.Event+GameStarted or FsUno.Domain.Event+GameStarted) from the string "GameStarted".

Perhaps this boils down to

1. serializeAggUnion<'aggEventType> (aggName:string) instance:'aggEventType : byte[],storedType: string
2. serializeUnion<'subsystemEventType> (instance:'subsystemEventType) : byte[],storedType: string // storedType is a case name only
3. deserializeAggUnion<'aggEventType> (aggToEventDu:string->Type) (storedType:string) (instance:obj) // storedType is split by the same name encoder used in (1)
4. deserializeUnion<'subsystemEventType> (storedType:string) (instance:byte[]) // storedType is a case name only

[bartelink]

One big thing you lose if one consolidates all the Events in the domain into one DU is that you end up visiting all the match expressions whenever you add an event (esp in the apply impl).

module Game =
    type Events =
        | GameStarted
        | GameEvent2

module Agg2 =
    type Events =
        | EventX

I guess one could do a hybrid -

• not keep an Agg name alongside the event, just the case name

• have an Events DU in each Agg

• have a reverse lookup function:-

  let selectAgg = function 
      | "GameStarted"  -> typeof<Game.GameStarted>
      | "GameEvent2"  -> typeof<Game.GameEvent2>
      | "EventX"         -> typeof<Agg2.EventX>
  

I'm thinking this seems the best general strategy (Best Practice :D) as you get a clear events index and not a big tangled ball of mud Events.fs (and the routing is clear).

BUT Any reflection based reverse lookups are bad news and defy static analysis. Perhaps:

let selectAgg = function 
    | "GameStarted" | "GameEvent2" -> typeof<Game.Evetns>
    | "EventX"                       -> typeof<Agg2.Events>

works well in terms of static analysis?

IOW I'm trying to surface the mappings a DI container (via Convention based registrations) and/or Serialization.fs (via reflection) could/should (NOT!) be doing.

[bartelink]

Most recent thoughts (after twitter discussion and some playing in https://github.com/thinkbeforecoding/FsUno.Prod/pull/8 ) are that now Game has:

type Event =
| GameStarted of GameStartedEvent
| CardPlayed of CardPlayedEvent
| PlayerPlayedAtWrongTurn of PlayerPlayedAtWrongTurnEvent
| PlayerPlayedWrongCard of PlayerPlayedWrongCardEvent
| DirectionChanged of DirectionChangedEvent

The serialization and tests use this DU and save the caseName (GameStarted).

I think if the serialization saves only the case body (as it already does) and maintains the DU case body type as the GES type, then tryDeserialize 'Du typeName blob : 'Du option could:

• identify the case type names of the DU presented (not sure if that's intrinsic, might be reflection)
• for each item being deserialized, deserialize as the DU type with the case type matching the event type
• for any not found, assume the impl is not interested (either emitting in impl or consuming in evolve) in that event anymore and that's OK

As you probably agree, no namespaces should be kept (the DUs track the mapping to where the event body type lives - be that in Game, Deck or Domain).

And then all the junk required for projection is already done:-

type WrongEvent = 
| Turn of PlayerPlayedAtWrongTurnEvent
| Card of PlayerPlayedWrongCardEvent

type StartEvent = 
| Start of GameStartedEvent

let project = 
    let wrongEvents = tryDeserializer<WrongEvent> // cache the reflection of the case types 
    let starts = tryDeserializer<StartEvent> // cache the reflection of the case types 

    fun (events: (typeName: string* eventBlob: byte[]) list) ->
        events 
        |> List.choose wrongEvents
        |> List.iter <| function
              | Turn _ -> printfn "Bad turn"
              | Card _ -> printfn "Bad card"
        events 
        |> List.choose starts 
        |> List.iter primerAgent.dispatch

[bartelink]

Implemented DU body emission high level concept as mentioned in previous comment in FunDomain

[bartelink]

Complete impl of the concept pushed to FunDomain.

@thinkbeforecoding Raises the questions:

1. is this something you're interested in using in FsUno.Prod as a replacement for the existing case name scheme (I could do a PR pretty quick or you can reimpl (or perhaps review first in context of FunDomain))
2. If not as a replacement, might be a good idea to add it to Serialization.fs anyway (slight consistency issue in that case - would prob encapsulate the (case name, encoded bytes) tuple in a record as with the new stuff)