firestore: design doc for custom datatypes

[tritone]

Issues: #1438 #1863 #1740 #1475 #1267 #2643

Past attempts at fixes from the community: #2481 , https://code-review.googlesource.com/c/gocloud/+/50430 (mentions something like datastore.PropertyLoadSaver as a potential model for a solution.

We've seen a collection of issues (linked above) related to using user-defined data types with the firestore library. This is definitely a gap and something that's impacted the usability of the library. This issue covers making a design doc for a fix:

• Consolidate and synthesize all the issues raised in the above threads
• Define the criteria for what needs to be accomplished in a fix.
• Propose solution(s) with a basic definition of what the surface will look like.

We'll need to review the design before moving forward to implementation. The design must be not constitute a breaking change.

[roboncode]

Here is a file I have created to convert structs to maps - https://gist.github.com/roboncode/d8592d3a6ca3704a92ff2c35baf8456a

[mfreeman451]

Any updates on this?

[subimage]

Just ran into this…completely unintuitive. Is there an ETA on when this will be fixed? Two years later and nothing?

[treeder]

3+ years...

[trakhimenok]

Here is a workaround I came with to be able to store a JSON-like content to one of "extra" fields:

https://github.com/sneat-co/sneat-go-backend/blob/main/src/modules/assetus/models4assetus/asset_extra.go

// WithAssetExtraField defines and `Extra` field to store extension data
type WithAssetExtraField struct {
	ExtraType AssetExtraType `json:"extraType" firestore:"extraType"`
	Extra     map[string]any `json:"extra" firestore:"extra"`
	extra     AssetExtra
}

func (v *WithAssetExtraField) SetExtra(extra AssetExtra) (err error) {
	v.extra = extra
	if extra == nil {
		v.Extra = make(map[string]any)
	} else {
		var b []byte
		if b, err = json.Marshal(extra); err != nil {
			return fmt.Errorf("failed to marshal extra data to JSON: %w", err)
		}
		if err = json.Unmarshal(b, &v.Extra); err != nil {
			return fmt.Errorf("failed to unmarshal JSON data to extra type %t: %w", extra, err)
		}
	}
	return nil
}

func (v *WithAssetExtraField) GetExtra() (extra AssetExtra, err error) {
	var b []byte
	if v.extra == nil {
		switch v.ExtraType {
		case AssetExtraTypeVehicle:
			v.extra = new(AssetVehicleExtra)
		case AssetExtraTypeDwelling:
			v.extra = new(AssetDwellingExtra)
		case AssetExtraTypeDocument:
			v.extra = new(AssetDocumentExtra)
		default:
			return nil, fmt.Errorf("unsupported extra type: %s", v.ExtraType)
		}
	}
	if len(v.Extra) == 0 {
		return v.extra, nil
	}
	if b, err = json.Marshal(v.Extra); err != nil {
		return nil, fmt.Errorf("failed to marshal extra data to JSON: %w", err)
	}

	if err = json.Unmarshal(b, &v.extra); err != nil {
		return nil, err
	}
	return v.extra, nil
}

Ideally I'd like to have an interface like:

type CustomFieldObjectsProvider interface {
	GetCustomFieldObjects(fields map[string]any) 
}

that should be called after all other non-custom fields have been umarshaled to the instance and the implementation on consumer type would be something like:

func (v *MyDocument) GetCustomFieldObjects(fields map[string]any) {
	for field := range fields {
		switch field {
		case "field1":
			switch v.SomeFlg {
			case "a":
				fields[field] = new(A)
			case "b":
				fields[field] = new(B)
			}
		}
	}
	return
}

After calling this method Firestore lib can unmarshall the custom fields into the target objects.