[Enhancement]: Everything we know about `AnyDataHolder` is wrong

[jonbarrow]

Checked Existing

• [X] I have checked the repository for duplicate issues.

What enhancement would you like to see?

Currently we implement AnyDataHolder as documented by Kinnay originally https://nintendo-wiki.pretendo.network/docs/nex/types#anydataholder

This is wrong, however, and doesn't make sense in places like the matchmaking protocols where AnyDataHolder<Gathering> and the like is frequently seen (since Gathering does not inherit from Data). We should probably change this to be more accurate, as well as update the docs to reflect this

There are actually several classes/templates used here for this and AnyDataHolder is NOT a real type. The notes here will be broken up into relevant sections

All data is taken from either Xenoblade or Minecraft on the Wii U

This was initially discovered in July 2024 and discussion/research/drafts continued into August of 2024 however we never published these findings outside of our private Discord channel. This issue is designed to bring these notes input the public eye so further discussion can happen with them

Classes/concepts

• AnyObjectHolder - Template class used to hold any object. Takes in 2 types, the base class for the type to be held and a 2nd "identifier" type. In practice the "identifier" type is always nn::nex::String, and is likely used to write the data we currently document as Type name (it's possible that other types are supported here but was never officially used)
• AnyObjectAdapter - Seems to be just a barebones adapter class?
• AnyDataAdapter - Seems to be an adapter class to conform to Data?
• AnyGatheringAdapter - Seems to be an adapter class to conform to Gathering?
• *Holder - Class-specific holders (explained later)
• CustomDataHolder - Simplified holder (explained later)

Signature examples

As an example, the RMC method SecureConnection::RegisterEx has the signature CallRegisterEx__Q3_2nn3nex30SecureConnectionProtocolClientFPQ3_2nn3nex19ProtocolCallContextPQ3_2nn3nex7qResultRCQ3_2nn3nex36qList__tm__23_Q3_2nn3nex10StationURLPUiPQ3_2nn3nex10StationURLRCQ3_2nn3nex56AnyObjectHolder__tm__33_Q3_2nn3nex4DataQ3_2nn3nex6String. The important part here being 2nn3nex56AnyObjectHolder__tm__33_Q3_2nn3nex4DataQ3_2nn3nex6String, which decodes to nn::nex::AnyObjectHolder<nn::nex::Data, nn::nex::String> and represents the hCustomData field of the request

However in other protocols, this is not the case. For example the RMC method MatchmakeExtension::BrowseMatchmakeSession has the signature BrowseMatchmakeSession__Q3_2nn3nex24MatchmakeExtensionClientFPQ3_2nn3nex19ProtocolCallContextRCQ3_2nn3nex30MatchmakeSessionSearchCriteriaRCQ3_2nn3nex11ResultRangePQ3_2nn3nex87qList__tm__74_Q3_2nn3nex61AnyObjectHolder__tm__38_Q3_2nn3nex9GatheringQ3_2nn3nex6StringPQ3_2nn3nex39qList__tm__26_Q3_2nn3nex13GatheringURLs. The important part here being 2nn3nex61AnyObjectHolder__tm__38_Q3_2nn3nex9GatheringQ3_2nn3nex6String, which decodes to nn::nex::AnyObjectHolder<nn::nex::Gathering, nn::nex::String> and represents the lstGathering field of the response

This means that the current documentation and implementation is wrong, AnyDataHolder does NOT exist and it does NOT hold types like Gathering. Instead a base template class called AnyObjectHolder is used to create these type holders. The confusion likely came about because the DDLs just refer to this as any, so Kinnay likely just assumed that any was all the same data type

Adapter classes

These seem to just be part of standard adapter design, based on the class names and some of the functions? Unsure what they're ACTUALLY used for in NEX but they show up in signatures. Not super relevant for us I think

*Holder classes

To expand on this further, @DaniElectra later discovered some debug logs still present in some games which give us more hints as to how this works. The debug logs are used to warn developers that an "obsolete" RMC method is being used:

nn::nex::Platform::WarnObsoleteMethod(L"BackEndServices::Disconnect(CallContext*)", L"BackEndServices::Logout(CallContext*, Credentials*)");

nn::nex::Platform::WarnObsoleteMethod(L"MatchMakingClient::FindByDescriptionRegex(ProtocolCallContext*,const String&,const Re sultRange&,qList<GatheringHolder>*)", L"MatchMakingClient::FindByDescriptionLike(ProtocolCallContext*,const String&,const Res ultRange&,qList<GatheringHolder>*)");

This means that the original implementation seems to have expanded on AnyObjectHolder for more specific classes. In this case GatheringHolder which holds any class that inherits from Gathering. We can use this to assume that in places like SecureConnection::RegisterEx a class called DataHolder is used, which matches up more closely to the original documentation

2nd "length" field

@DaniElectra also discovered that the documentation for the structure is slightly wrong. The "second length" field IS part of a Buffer class. We implement it like this already for convenience, but the documentation is wrong and we have a comment saying the implementation is wrong:

void StreamOut__Q3_2nn3nex19AnyGatheringAdapterCFPQ3_2nn3nex7MessagePPQ3_2nn3nex9Gathering
               (undefined4 param_1,Message *message,int *param_3)

{
  int iVar1;
  bool bVar2;
  Buffer *msgBuffer;
  uint size;
  String identifier;
  Buffer buffer;
  Message msg;
  
  nn::nex::String::String(&identifier);
  nn::nex::String::Extract(message,&identifier);
  iVar1 = CreateObject__Q3_2nn3nex19AnyGatheringAdapterCFRCQ3_2nn3nex6String
                    (param_1,(uint **)&identifier);
  *param_3 = iVar1;
  nn::nex::Buffer::Buffer(&buffer);
  size = 0;
  bVar2 = nn::nex::ByteStream::ExtractPrimitive<uint>((ByteStream *)message,&size);
  if ((bVar2) &&
     (bVar2 = nn::nex::ByteStream::ValidateBufferLimit((ByteStream *)message,size), bVar2)) {
    msgBuffer = nn::nex::Message::GetBuffer(message);
    nn::nex::Buffer::AppendData
              (&buffer,msgBuffer->dataBuffer + message->offset + msgBuffer->shiftSize,size,
               0xffffffff);
    nn::nex::ByteStream::SetPosition((ByteStream *)message,message->offset + size);
  }
  nn::nex::Message::Message(&msg,&buffer);
  (**(code **)(*(int *)(*param_3 + 0x28) + 0x3c))(*param_3,&msg);
  nn::nex::Message::~Message(&msg,2);
  nn::nex::Buffer::~Buffer((int)&buffer,2);
  nn::nex::String::~String(&identifier,2);
  return;
}

I later then checked the corresponding StreamIn method to confirm this:

void StreamIn__Q3_2nn3nex19AnyGatheringAdapterCFPQ3_2nn3nex7MessageRCQ3_2nn3nex9Gathering
               (undefined4 param_1,nn::nex::Message *message,int gathering)

{
  int iVar1;
  nn::nex::Buffer *buffer;
  nn::nex::String *type_name;
  uint32_t size;
  undefined message_2 [72];
  
  (**(code **)(*(int *)(gathering + 0x2c) + 0x1c))(gathering,&type_name);
  iVar1 = IsEqual__Q3_2nn3nex6StringSFPCwT1(type_name,L"PersistentGathering");
  if (iVar1 != 0) {
    __as__Q3_2nn3nex6StringFPCw(&type_name,L"Community");
  }
  nn::nex::_Type_string::Add(message,(nn::nex::String *)&type_name);
  __ct__Q3_2nn3nex7MessageFv(message_2);
  (**(code **)(*(int *)(gathering + 0x2c) + 0x34))(gathering,message_2);
  buffer = (nn::nex::Buffer *)GetBuffer__Q3_2nn3nex7MessageFv((nn::nex::Message *)message_2);
  size = buffer->offset;
  nn::nex::ByteStream::Append((uchar *)message,(uint)&size,4);
  nn::nex::ByteStream::Append((uchar *)message,buffer->data + buffer->head_shift_size,1);
  __dt__Q3_2nn3nex7MessageFv(message_2,2);
  __dt__Q3_2nn3nex6StringFv(&type_name,2);
  return;
}

(Side note: This seems to confirm something I had previously thought to be the case, that PersistentGathering and Community are interchangeable concepts. Nintendo probably changed the name at some point, but they're the exact same thing internally)

CustomDataHolder

I later discovered CustomDataHolder, which seems to be a simplified abstraction on these holders. Where the holders previously could take in a 2nd class type, CustomDataHolder only takes in the class type it is intended to be holding. This class builds off DataHolder and always sets the 2nd class type to String (AnyObjectHolder<nn::nex::Data, nn::nex::String>), which means no CustomDataHolder could ever have a different "identifier" type

The Messaging protocol makes use of this to store messages (nn::nex::CustomDataHolder<nn::nex::UserMessage>. UserMessage inherits from Data which makes it compatible with DataHolder and thus usable in CustomDataHolder, and both TextMessage and BinaryMessage inherit from UserMessage). This is used, for example, in Messaging::DeliverMessage as the oUserMessage field of the request

It seems like for all intents and purposes, CustomDataHolder and DataHolder function identically. They just seem to get compiled differently and thus their signatures appear different in functions? However more research into this needs to be done to be sure

Any other details to share? (OPTIONAL)

It looks like at some point maybe Kinnay noticed this as well and changed his common types (https://github.com/kinnay/NintendoClients/blob/e2b673bac6d7781e83f83ae2c5d0c34a2092d72e/nintendo/nex/common.py#L121-L155)? However this only seems to partially account for DataHolder and nothing else mentioned here

An implementation of this in Go has been partially drafted, but nothing has been set in stone and all implementations require some level of hacks/compromises to work. All issues stem from the fact that Go does not have inheritance, and thus either types can be much too strict (which breaks cases like child classes) or too permissive (which prevents strict narrowing). A version of narrowing WITH child class support CAN be achieved by abusing interfaces however, but it's somewhat hacky

Below are the draft implementations as they evolved. These are provided for historical purposes, to see what has already been tried. I am not 100% sold on these, as they are somewhat hacky, so I am open to further design suggestions. However this may be as good as we are going to get:

8/13/24 (no child structs)

My first draft which allowed for type narrowing, but disallowed child structs:

package main

import "fmt"

type RVType interface{}

type AnyObjectHolder[T RVType] struct {
    object T
}

type Gathering struct{}
type MatchmakeSession struct{ Gathering }

type GatheringHolder = AnyObjectHolder[Gathering]

func main() {
    holder1 := GatheringHolder{Gathering{}}
    holder2 := GatheringHolder{MatchmakeSession{}} // Fails to compile because MatchmakeSession is not Gathering

    test(holder1)
    test(holder2)
}

func test(input GatheringHolder) {
    fmt.Println(input)
}

8/13/24 ("identification struct method")

My second draft which allowed for both child structs and type narrowing, at the cost of interface abuse. This method uses the concept of a "identification struct method" on an interface to "trick" Go into allowing types which inherit from each other. This concept (hack, really) continues in all further implementations:

package main

import "fmt"

type RVType interface{}

type AnyObjectHolder[T RVType] struct {
    object T
}

type String = string
type Gathering struct{}

func (g Gathering) isGathering() {} // Anything that embeds Gathering will ALSO have this method by default

type MatchmakeSession struct{ Gathering }

// This interface narrows types down to only those which have the isGathering() method
type GatheringInterface interface {
    isGathering()
}

type GatheringHolder = AnyObjectHolder[GatheringInterface] // Only allow types which conform to GatheringInterface

func main() {
    var str String = "test"
    holder1 := GatheringHolder{Gathering{}}
    holder2 := GatheringHolder{MatchmakeSession{}} // Since MatchmakeSession embeds Gathering and has the isGathering() method, it conforms
    holder3 := GatheringHolder{str}                // Does not have the isGathering() method, does not conform

    test(holder1)
    test(holder2)
    test(holder3)
}

func test(input GatheringHolder) {
    fmt.Println(input)
}

8/18/24 (adds in the "identification" type support)

Dani later modified my 2nd draft to add in support for the "identifier" type:

package main

import "fmt"

type RVType interface{}

type AnyObjectHolder[T RVType, I RVType] struct {
    object     T
    identifier I
}

type String string
type Gathering struct{}

func (g Gathering) GatheringObjectIdentifier() String {
    return "Gathering"
} // Anything that embeds Gathering will ALSO have this method by default

type MatchmakeSession struct{ Gathering }

func (ms MatchmakeSession) GatheringObjectIdentifier() String {
    return "MatchmakeSession"
}

// This interface narrows types down to only those which have the isGathering() method
type GatheringInterface interface {
    GatheringObjectIdentifier() String
}

type GatheringHolder = AnyObjectHolder[GatheringInterface, String] // Only allow types which conform to GatheringInterface

func main() {
    holder1 := GatheringHolder{object: Gathering{}}
    holder2 := GatheringHolder{object: MatchmakeSession{}} // Since MatchmakeSession embeds Gathering and has the GatheringIdentifier() method, it conforms

    test(holder1)
    test(holder2)
}

func test(input GatheringHolder) {
    fmt.Println(input.object.GatheringObjectIdentifier())
}

8/18/24 ("HeldObject")

This is a modified version of Dani's implementation I made which adds in the concept of HeldObject to try and help further centralize and narrow our types. However this has the issue of making the "identification struct method" private. This was done to prevent types from outside the package from being valid, but this design prevents expansion so it should probably be changed if used:

package main

import "fmt"

type RVType interface{}

// This name ("HeldObject") is dumb, just to get the idea across.
// Restrict AnyObjectHolder only to types which have ID methods
type HeldObject interface {
    RVType
    ObjectStringID() String
    // We can expand on this to add more Object ID methods (like numbers), if we see others being used
}

type AnyObjectHolder[T HeldObject, I RVType] struct { // Now this only allows for `HeldObject` types to be held
    object     T
    identifier I
}

// Centralized object writing. Stubbed
func (aoh AnyObjectHolder[T, I]) WriteTo() {
    fmt.Println(aoh.object.ObjectStringID())
    fmt.Println(aoh.object)
}

type String string
type Gathering struct{}

// This makes the type conform to HeldObject
func (g Gathering) ObjectStringID() String {
    return g.gatheringObjectID()
}

// This method uniquely identifies the type as being valid for GatheringHolder.
// Anything that embeds Gathering will ALSO have this method by default.
// Make it private so that structs from outside this package can't conform to GatheringHolder
func (g Gathering) gatheringObjectID() String {
    return "Gathering"
}

type MatchmakeSession struct{ Gathering }

// Need to redefine these otherwise the Gathering ones are used
func (ms MatchmakeSession) ObjectStringID() String {
    return ms.gatheringObjectID()
}

func (ms MatchmakeSession) gatheringObjectID() String {
    return "MatchmakeSession"
}

// This interface narrows types down to only those which have the gatheringObjectID() method
type GatheringInterface interface {
    HeldObject
    gatheringObjectID() String
}

type GatheringHolder = AnyObjectHolder[GatheringInterface, String] // Only allow types which conform to GatheringInterface. This is more like a "CustomDataHolder" in terms of implementation since it always uses "String" just with a different name, which I THINK is correct usage? But this could be wrong

func main() {
    holder1 := GatheringHolder{object: Gathering{}}
    holder2 := GatheringHolder{object: MatchmakeSession{}} // Since MatchmakeSession embeds Gathering and has the gatheringObjectID() method, it conforms

    test(holder1)
    test(holder2)
}

func test(input GatheringHolder) {
    input.WriteTo() // Demo of writing the type
}

8/18/24 (CustomDataHolder)

Below is the first draft of an implementation of CustomDataHolder, building off the previous designs:

package main

import "fmt"

type RVType interface{}

// HeldObject defines a common interface for types which can be placed in AnyObjectHolder
type HeldObject interface {
    RVType
    ObjectStringID() String
}

type String string

// AnyObjectHolder can hold a reference to any RVType which can be held.
// Takes in an additional identification type.
type AnyObjectHolder[T HeldObject, I RVType] struct {
    object     T
    identifier I
}

func (aoh AnyObjectHolder[T, I]) WriteTo() {
    fmt.Println(aoh.object.ObjectStringID())
    fmt.Println(aoh.object)
}

// Data is a base struct. It's only purpose is to be the parent of other
// types to allow them to be placed in DataHolder.
type Data struct{}

func (d Data) ObjectStringID() String {
    return d.dataObjectID()
}

func (d Data) dataObjectID() String {
    return "Data"
}

// DataInterface defines an interface to track types which have Data anywhere
// in their parent tree.
type DataInterface interface {
    HeldObject
    dataObjectID() String
}

// DataHolder is an AnyObjectHolder for types which embed Data
type DataHolder[T DataInterface] AnyObjectHolder[T, String]

// CustomDataHolder customizes DataHolder for a specific type of data.
type CustomDataHolder[T DataInterface] struct {
    DataHolder[T]
}

// NewCustomDataHolder initializes a CustomDataHolder with the given type.
func NewCustomDataHolder[T DataInterface](obj T) CustomDataHolder[T] {
    return CustomDataHolder[T]{DataHolder[T]{object: obj}}
}

// UserMessage represents a specific data type.
type UserMessage struct{ Data }

func (um UserMessage) ObjectStringID() String {
    return um.dataObjectID()
}

func (um UserMessage) dataObjectID() String {
    return "UserMessage"
}

// TextMessage and BinaryMessage are types that extend UserMessage.
type TextMessage struct{ UserMessage }

func (tm TextMessage) ObjectStringID() String {
    return tm.dataObjectID()
}

func (tm TextMessage) dataObjectID() String {
    return "TextMessage"
}

type BinaryMessage struct{ UserMessage }

func (bm BinaryMessage) ObjectStringID() String {
    return bm.dataObjectID()
}

func (bm BinaryMessage) dataObjectID() String {
    return "BinaryMessage"
}

func main() {
    holder1 := NewCustomDataHolder(UserMessage{})
    holder2 := NewCustomDataHolder(TextMessage{})
    holder3 := NewCustomDataHolder(BinaryMessage{})

    fmt.Println(holder1.DataHolder.object.ObjectStringID()) // UserMessage
    fmt.Println(holder2.DataHolder.object.ObjectStringID()) // TextMessage
    fmt.Println(holder3.DataHolder.object.ObjectStringID()) // BinaryMessage
}

@DaniElectra then later suggested a modified New function:

// NewCustomDataHolder initializes a CustomDataHolder with the given type.
func NewCustomDataHolder[T DataInterface](obj T) CustomDataHolder[T] {
    return CustomDataHolder[T]{DataHolder[T]{object: obj, identifier: obj.dataObjectID()}}
}

[ashquarky]

Haven't read and understood everything yet but just to check youre using ghs-demangle for those C++ types right? Tools meant for gcc (ghidra, c++filt) can give incorrect decoding results when used on ghs symbols

[jonbarrow]

Yes, I use https://github.com/Chadderz121/ghs-demangle for everything. My Ghidra setup doesn't even automatically demangle Wii U symbols at all

[DaniElectra]

I did some testing with the latest proposal and it seems that you would not be able to call CustomDataHolder.WriteTo() directly because DataHolder is defined as a type definition for AnyObjectHolder, and not a type alias.

type DataHolder[T DataInterface] AnyObjectHolder[T, String]

This means that to call WriteTo we'd have to do the following:

holder1 := NewCustomDataHolder(UserMessage{})
holder11 := AnyObjectHolder[UserMessage, String](holder1.DataHolder)
holder11.WriteTo()

Using a type alias for a generic type is possible, and would make the code look better:

type DataHolder[T DataInterface] = AnyObjectHolder[T, String]

holder1 := NewCustomDataHolder(UserMessage{})
holder1.WriteTo()

However, this feature is locked in Go behind an experimental flag GOEXPERIMENT=aliastypeparams. If we set the flag it seems to work though

Also, I'm not sure how would this work for reading an object holder? If we do AnyObjectHolder[Data, String], then we would only be able to store the Data part of the object, but we would be missing the rest of the object that would inherit Data.

[jonbarrow]

Due to the above mentioned issues, and in the spirit of #73 to reduce "1:1" ports in favor of "follow the intentions" implementations, I've opted to change this proposal to be simpler than the way Nintendo/Quazal implemented this. Our basic usage requirements are:

• Replace usage of AnyDataHolder with either a specific holder designed for a parent type or AnyObjectHolder if the parent type is unknown
• We've only ever seen string identifiers in the wild, so just assume it's always a string still to simplify things (not sure how we'd handle this anyway, global config?)
• Don't bother implementing all the tons of sub-holder types like CustomDataHolder or the adapters. We assume a string identifier anyway so it would be pointless to keep making sub-holder types since the Custom* holders seem to ONLY exist as a way to make a holder that assumes a string identifier?

With this in mind, here is an updated demo. This demo:

• Demos reading and writing (uses dummy data, but should still work with real data)
• Only implements AnyObjectHolder and the specific holders for a parent type
• Assumes identifiers are strings at all times
• Removed the generic from the parent holder aliases, removing the need for GOEXPERIMENT=aliastypeparams
• Makes the "ID methods" public because the idea of keeping them private was stupid and prevented things like types in the protocols lib from being valid here

I think this demonstrates everything we'd need this to do, while not being 1:1 with the decomp? Which I think is a good first step to #73?

package main

import "fmt"

type RVType interface{}

type HoldableObject interface {
	RVType
	ObjectID() any
}

var AnyObjectHolderObjects = make(map[string]HoldableObject)

type AnyObjectHolder[T HoldableObject] struct {
	Object T
}

func (aoh AnyObjectHolder[T]) WriteTo() {
	fmt.Println(aoh.Object.ObjectID())
	fmt.Printf("%+v\n", aoh.Object)
}

func (aoh *AnyObjectHolder[T]) ExtractFrom() {
	objectName := "UserMessage"                  // * Simulate reading the string here
	object := AnyObjectHolderObjects[objectName] // * Simulate the "CopyRef()"/"Deref()" calls

	aoh.Object = object.(T)
}

type Data struct{}

func (d Data) ObjectID() any {
	return "Data"
}

type DataInterface interface {
	HoldableObject
	DataObjectID() any
}

type DataHolder = AnyObjectHolder[DataInterface]

type UserMessage struct {
	Data
	m_uiID uint32
}

func (um UserMessage) ObjectID() any {
	return um.DataObjectID()
}

func (um UserMessage) DataObjectID() any {
	return "UserMessage"
}

type Gathering struct {
	m_idMyself uint32
}

func (g Gathering) ObjectID() any {
	return g.GatheringObjectID()
}

func (g Gathering) GatheringObjectID() any {
	return "Gathering"
}

type GatheringInterface interface {
	HoldableObject
	GatheringObjectID() any
}

type GatheringHolder = AnyObjectHolder[GatheringInterface]

type MatchmakeSession struct {
	Gathering
	m_GameMode uint32
}

func (ms MatchmakeSession) ObjectID() any {
	return ms.GatheringObjectID()
}

func (ms MatchmakeSession) GatheringObjectID() any {
	return "MatchmakeSession"
}

func main() {
	AnyObjectHolderObjects["UserMessage"] = UserMessage{}
	AnyObjectHolderObjects["MatchmakeSession"] = MatchmakeSession{}

	ReadDemo()
	WriteDemo()
}

func ReadDemo() {
	// * Like in the request for TicketGranting::DeliverMessageMultiTarget
	oUserMessage := DataHolder{}

	oUserMessage.ExtractFrom()

	fmt.Printf("%+v\n", oUserMessage) // * {Object:{Data:{} m_uiID:0}}
}

func WriteDemo() {
	// * Like in the response for MatchmakeExtension::AutoMatchmake_Postpone
	joinedGathering := GatheringHolder{
		Object: MatchmakeSession{}, // * Dummy data
	}

	joinedGathering.WriteTo() // * Logs:
	// * MatchmakeSession
	// * {Gathering:{m_idMyself:0} m_GameMode:0}

	joinedGathering = GatheringHolder{
		Object: Gathering{}, // * Dummy data
	}

	joinedGathering.WriteTo() // * Logs:
	// * Gathering
	// * {m_idMyself:0}
}

[DaniElectra]

This looks good, though I think we can still handle multiple identifiers for the sake of compatibility (in case there's some random QRV game that doesn't use strings) by using a single map[RVType]HoldableObject without adding too much complexity. This also removes the ambiguous any return type for the ObjectID:

package main

import "fmt"

type RVType interface{}

type String string

type HoldableObject interface {
	RVType
	ObjectID() RVType
}

var AnyObjectHolderObjects = make(map[RVType]HoldableObject)

type AnyObjectHolder[T HoldableObject] struct {
	Object T
}

func (aoh AnyObjectHolder[T]) WriteTo() {
	fmt.Println(aoh.Object.ObjectID())
	fmt.Printf("%+v\n", aoh.Object)
}

func (aoh *AnyObjectHolder[T]) ExtractFrom() {
	objectName := String("UserMessage")          // * Simulate reading the string here
	object := AnyObjectHolderObjects[objectName] // * Simulate the "CopyRef()"/"Deref()" calls

	aoh.Object = object.(T)
}

type Data struct{}

func (d Data) ObjectID() RVType {
	return d.DataObjectID()
}

func (d Data) DataObjectID() RVType {
	return String("Data")
}

type DataInterface interface {
	HoldableObject
	DataObjectID() RVType
}

type DataHolder = AnyObjectHolder[DataInterface]

type UserMessage struct {
	Data
	m_uiID uint32
}

func (um UserMessage) ObjectID() RVType {
	return um.DataObjectID()
}

func (um UserMessage) DataObjectID() RVType {
	return String("UserMessage")
}

type Gathering struct {
	m_idMyself uint32
}

func (g Gathering) ObjectID() RVType {
	return g.GatheringObjectID()
}

func (g Gathering) GatheringObjectID() RVType {
	return String("Gathering")
}

type GatheringInterface interface {
	HoldableObject
	GatheringObjectID() RVType
}

type GatheringHolder = AnyObjectHolder[GatheringInterface]

type MatchmakeSession struct {
	Gathering
	m_GameMode uint32
}

func (ms MatchmakeSession) ObjectID() RVType {
	return ms.GatheringObjectID()
}

func (ms MatchmakeSession) GatheringObjectID() RVType {
	return String("MatchmakeSession")
}

func main() {
	AnyObjectHolderObjects[String("UserMessage")] = UserMessage{}
	AnyObjectHolderObjects[String("MatchmakeSession")] = MatchmakeSession{}

	ReadDemo()
	WriteDemo()
}

func ReadDemo() {
	// * Like in the request for TicketGranting::DeliverMessageMultiTarget
	oUserMessage := DataHolder{}

	oUserMessage.ExtractFrom()

	fmt.Printf("%+v\n", oUserMessage) // * {Object:{Data:{} m_uiID:0}}
}

func WriteDemo() {
	// * Like in the response for MatchmakeExtension::AutoMatchmake_Postpone
	joinedGathering := GatheringHolder{
		Object: MatchmakeSession{}, // * Dummy data
	}

	joinedGathering.WriteTo() // * Logs:
	// * MatchmakeSession
	// * {Gathering:{m_idMyself:0} m_GameMode:0}

	joinedGathering = GatheringHolder{
		Object: Gathering{}, // * Dummy data
	}

	joinedGathering.WriteTo() // * Logs:
	// * Gathering
	// * {m_idMyself:0}
}

[jonbarrow]

My main concern with using multiple identifiers is how we actually tell the server which one to use. This isn't so much of an issue when writing objects, but for reading them it is. The reader needs to know beforehand what type of identifier to use, otherwise it won't be able to make the copies it needs to. This can't be done on a per-object basis either because you need to know what object is being used before doing anything

The only thing I can think of is, like, a global config? Like we do other stream settings like the PID size? But then how would that API look?

The any was put there as a bit of "future proofing" in case we did add support for multiple identifiers to be clear, but your implementation works just fine for that as well

[DaniElectra]

Oh right I missed that. In that case we can also include the identifier on the generics like originally intended without too much hassle, like shown in the following code. If we think this is getting too complex, I'm fine with going back to your original design

package main

import "fmt"

type RVType interface {
	CopyRef() RVTypePtr
}

type RVTypePtr interface {
	RVType
	ExtractFrom()
	Deref() RVType
}

type String string

func (s *String) ExtractFrom() {
	*s = String("UserMessage")
}

func (s String) CopyRef() RVTypePtr {
	var copied String = s
	return &copied
}

func (s *String) Deref() RVType {
	return *s
}

type HoldableObject interface {
	RVType
	ObjectID() RVType
}

var AnyObjectHolderObjects = make(map[RVType]HoldableObject)

type AnyObjectHolder[T HoldableObject, I RVType] struct {
	Object T
	ObjectID I
}

func (aoh AnyObjectHolder[T, I]) WriteTo() {
	fmt.Println(aoh.ObjectID)
	fmt.Printf("%+v\n", aoh.Object)
}

func (aoh *AnyObjectHolder[T, I]) ExtractFrom() {
	objectID := aoh.ObjectID.CopyRef()             // * Simulate reading the string here
	objectID.ExtractFrom()
	aoh.ObjectID = objectID.Deref().(I)

	object := AnyObjectHolderObjects[aoh.ObjectID] // * Simulate the "CopyRef()"/"Deref()" calls

	aoh.Object = object.(T)
}

type Data struct{}

func (d Data) ObjectID() RVType {
	return d.DataObjectID()
}

func (d Data) DataObjectID() RVType {
	return String("Data")
}

type DataInterface interface {
	HoldableObject
	DataObjectID() RVType
}

type DataHolder = AnyObjectHolder[DataInterface, String]

type UserMessage struct {
	Data
	m_uiID uint32
}

func (um UserMessage) ObjectID() RVType {
	return um.DataObjectID()
}

func (um UserMessage) DataObjectID() RVType {
	return String("UserMessage")
}

func (um *UserMessage) ExtractFrom() {}

func (um *UserMessage) Deref() RVType {
	return *um
}

func (um UserMessage) CopyRef() RVTypePtr {
	copied := um
	return &copied
}

type Gathering struct {
	m_idMyself uint32
}

func (g Gathering) ObjectID() RVType {
	return g.GatheringObjectID()
}

func (g Gathering) GatheringObjectID() RVType {
	return String("Gathering")
}

func (g *Gathering) ExtractFrom() {}

func (g *Gathering) Deref() RVType {
	return *g
}

func (g Gathering) CopyRef() RVTypePtr {
	copied := g
	return &copied
}

type GatheringInterface interface {
	HoldableObject
	GatheringObjectID() RVType
}

type GatheringHolder = AnyObjectHolder[GatheringInterface, String]

type MatchmakeSession struct {
	Gathering
	m_GameMode uint32
}

func (ms MatchmakeSession) ObjectID() RVType {
	return ms.GatheringObjectID()
}

func (ms MatchmakeSession) GatheringObjectID() RVType {
	return String("MatchmakeSession")
}

func (ms *MatchmakeSession) ExtractFrom() {}

func (ms *MatchmakeSession) Deref() RVType {
	return *ms
}

func (ms MatchmakeSession) CopyRef() RVTypePtr {
	copied := ms
	return &copied
}

func main() {
	AnyObjectHolderObjects[String("UserMessage")] = UserMessage{}
	AnyObjectHolderObjects[String("MatchmakeSession")] = MatchmakeSession{}

	ReadDemo()
	WriteDemo()
}

func ReadDemo() {
	// * Like in the request for TicketGranting::DeliverMessageMultiTarget
	oUserMessage := DataHolder{}

	oUserMessage.ExtractFrom()

	fmt.Printf("%+v\n", oUserMessage) // * {Object:{Data:{} m_uiID:0}}
}

func WriteDemo() {
	matchmakeSession := MatchmakeSession{}

	// * Like in the response for MatchmakeExtension::AutoMatchmake_Postpone
	joinedGathering := GatheringHolder{
		Object: matchmakeSession, // * Dummy data
		ObjectID: matchmakeSession.ObjectID().(String),
	}

	joinedGathering.WriteTo() // * Logs:
	// * MatchmakeSession
	// * {Gathering:{m_idMyself:0} m_GameMode:0}


	gathering := Gathering{}

	joinedGathering = GatheringHolder{
		Object: gathering, // * Dummy data
		ObjectID: gathering.ObjectID().(String),
	}

	joinedGathering.WriteTo() // * Logs:
	// * Gathering
	// * {m_idMyself:0}
}

[jonbarrow]

I guess part of the issue here stems from the fact that we haven't actually seen anything besides strings being used here. Which means we have to make assumptions on how this works

There's 2 possibilities here:

1. The identifier is global. Which means every single object that is stored in a holder, including built in objects, all uses the same type of identifier
2. The identifier is per object type. Which means built in objects would always use strings, but custom objects may not

That changes the implementation here

If the way this works is option 2, then what you've done here should work perfectly. It would allow the built in objects to remain how they are, but custom ones to do whatever is they do. At least, that's the case for using the raw AnyObjectHolder type I think. Stuff like GatheringHolder still always assumes a string

However if it's option 1, then what you've done here wouldn't work since it still always assumes a string for the built in types

Option 1 is how I assumed things worked, which is why I was mentioning a "global config" so that we could change how built in types work. But it seems like you assumed it worked like option 2, where we just always use strings but custom types may use something else

It might be worth making a "needs additional research" issue here for how the identifiers work, and then holding off on implementing "custom identifiers" until later, and always assume strings for now. We can still keep some of the foundations of this though, like letting the map of registered types take in RVType as the key still and replacing any with RVType

[DaniElectra]

That sounds good to me, we can go with your design for now and use RVType instead of any