Support for structs in properties

[notdodo]

Feature Request:

Add support for this kind of properties definition:

type Sub struct {
	A int
	B string
	C []SubSub
}

type SubSub struct {
	D int
}

type VertexA struct {
	gogm.BaseNode
	// provides required node fields

	TestField     string            `gogm:"name=test_field"`
	TestStruct         Sub               `gogm:"name=test_struct;properties"`
	MapProperty   map[string]string `gogm:"name=map_property;properties"`
	SliceProperty []string          `gogm:"name=slice_property;properties"`
	SingleA       *VertexB          `gogm:"direction=incoming;relationship=test_rel"`
	ManyA         []*VertexB        `gogm:"direction=incoming;relationship=testm2o"`
	MultiA        []*VertexB        `gogm:"direction=incoming;relationship=multib"`
	SingleSpecA   *EdgeC            `gogm:"direction=outgoing;relationship=special_single"`
	MultiSpecA    []*EdgeC          `gogm:"direction=outgoing;relationship=special_multi"`
}

It should be useful to store custom defined structs inside the node properties; as a side note a possible way to achieve this could be to flatten the struct into something like TestStruct_A, TestStruct_B, TestStruct_C_D_0.

Context

This feature could be used to directly upload complex struct into the node (i.e. coming from a JSON response/data).

Alternatives

I don't it is possible without manually rewriting the node structure with the required fields since the library only support primitive types.

Would you be interested in implementing this feature?

Yes

[nikitawootten]

Hi, I like this idea, and it would simplify some things that we do at MindStand internally. So far we've stuck relatively close in features to the Neo4j Java OGM, but there's nothing stopping this project from branching out and implementing interesting features like this.

I will say that the approach of flattening a struct into a properties map is a lot more elegant then our initial idea, for this exact problem, which was to serialize the offending field to json and back. One drawback I see is that more complex structs that involve sub-maps or slices would lead to all sorts of interesting edge cases. This solution would also have to have limits so that super nested structs aren't allowed.

[notdodo]

Hi!

Thank you for the interest. I've created an awful routine to recursively flatten structs, maps, array/slices and primitive types.

func flattenStruct(strct interface{}, prefix string, remove string) (item map[string]interface{}) {
	item = make(map[string]interface{})
	structValue := reflect.ValueOf(strct)
	kind := structValue.Kind()

	// Return the real type immediately if pointer or interface
	if kind == reflect.Ptr || kind == reflect.Interface {
		structValue = reflect.Indirect(structValue)
		kind = structValue.Kind()
	}

	switch kind {
	case reflect.Array, reflect.Slice:
		for i := 0; i < structValue.Len(); i++ {
			for k, v := range flattenStruct(structValue.Index(i).Interface(), prefix, remove) {
				k = removeDuplicates(k, prefix, remove)
				item[k+prefix+strconv.Itoa(i)] = v
			}
		}
	case reflect.Map:
		for _, mapKey := range structValue.MapKeys() {
			mapValue := structValue.MapIndex(mapKey)
			cleanKey := strings.Replace(mapKey.String(), ":", strings.Repeat(prefix, 2), -1) // TODO: doc
			cleanKey = strings.Replace(cleanKey, "-", strings.Repeat(prefix, 3), -1)         // TODO: doc
			for k, v := range flattenStruct(mapValue.Interface(), prefix, remove) {
				val := fmt.Sprintf("%v", v)
				if k == "string" {
					item[cleanKey] = val
				} else {
					k = removeDuplicates(k+prefix+cleanKey, prefix, remove)
					item[k] = val
				}
			}
		}
	case reflect.String:
		item[reflect.TypeOf(strct).Name()] = structValue.String()
	case reflect.Struct:
		for structIterator := 0; structIterator < structValue.NumField(); structIterator++ {
			field := structValue.Field(structIterator)
			// Only exported fields
			if field.CanInterface() {
				key := structValue.Type().Field(structIterator).Name
				value := field.Interface()
				switch valueCasted := value.(type) {
				case string, bool, time.Time, int, int32:
					item[key] = valueCasted
				case *string:
					item[key] = aws.ToString(valueCasted)
				case *time.Time:
					if valueCasted != nil {
						item[key] = *valueCasted
					}
				case *int32:
					if valueCasted != nil {
						item[key] = *valueCasted
					}
				case *bool:
					item[key] = aws.ToBool(valueCasted)
				default:
					// Another struct, recursively return the fields
					for k, v := range flattenStruct(value, prefix, remove) {
						k = removeDuplicates(key+prefix+k, prefix, remove)
						item[k] = v
					}
				}
			}
		}
	}
	return
}

this is ugly but it's working to map AWS API JSON structs output into neo4j node properties...may be this could be a starting point

[nikitawootten]

That's a great solution, the resulting data should still be perfectly queriable from neo4j without having to resort to something like using APOC.

There are some interesting edge cases at play, like how we should handle loops (and for that matter if we have two references to the same struct, do we duplicate the data in the resulting structure). If we limit the property structs to a certain depth, how should we handle structs that exceed the depth? Should we error out, or truncate the output, or should we check if a struct is valid when the decorators are being parsed? Definitely looking for feedback here (@erictg :eyes:).

Would you be interested in implementing this idea? We can definitely provide you with pointers on the encoder/decoder as well as the struct decorator.

[notdodo]

Would you be interested in implementing this idea? We can definitely provide you with pointers on the encoder/decoder as well as the struct decorator.

I'd like to give it try if you can point me where to put the code and lead me on how the data is flowing in the library!

do we duplicate the data in the resulting structure

I think it's the user's responsibility to pass/create non-duplicated fields in the struct.

how should we handle structs that exceed the depth?

is it really a problem here? The only caveat that I see it's the computational time and eventually recursive structs mistakenly created by the user (i.e. a field that reference the struct itself, I don't if it's even possible).

I'm open to discuss all of the above points :)

[erictg]

Hey @notdodo, I apologize for not responding sooner! I've been super busy. First off I want to say I really like this proposal.

Approaches

So off the top of my head I can think of 2 approaches you could take to serializing this.

1. You could marshal the whole field to json on save then unmarshal the field when it is loaded. However given that the decoding is done in reflect I could anticipate some issues in decoding that.
2. You can save it similarly to how we currently handle maps. When saving maps the key is structured as `<field>.<key>`. With this you could structure the field name as `<field>.<json-path>`. This approach would also give you the bonus of reassembling the structure on decode and potentially using the key as part of your query.

I would personally chose option 2. Tbh the more I think about this the more I think option 1 isn't practical.

Areas of code to be updated to support this

1. Decorator logic

• In the decorator validate logic here you would want to update the way it checks for both properties and relationships. If I remember correctly it doesn't like structs to be used for anything but relationships. So you would have to validate that a struct property does not have any relationship related tags on it in order to pass.
• You also would want to add some stuff to decoratorConfig and propConfig. The decoratorConfig object essentially stores all of the information you would need in save for the system to understand the objects its reading in. You would specifically want to add to propConfig since it already holds information about property maps. You could potentially expand this to hold information about the json structure you are trying to serialize. You may also want to create an additional structure defining the layers of json to be used in the decode logic. When I developed gogm I added to this as I realized I needed information elsewhere. I'd recommend adding what you think you need, but don't be afraid to update this as you go. Once you're in save and decode all you have to go by is what is stored in this config. The key of the decorator portion is to use reflect to extract all necessary info about structs we're dealing with in order to avoid additional uses of reflect at runtime.

2. Save logic

• This piece is essentially the reverse of the decode logic, you'd have to figure out how to convert that struct json into property values that neo4j can then save.
• The bulk of this piece would actually be in util.go in the function toCypherParamsMap(). Essentially this takes a value and turns its fields into a map that the save logic then uses in create/merge queries.

3. Decode logic

• The main function in decode that you would need to update in decode is convertToValue(). This function takes in the raw response from neo4j which is basically just a map of the properties in the node and converts it into a reflect.Value based on the the information generated from the decorator logic. This information is accessed via the gogm object. You would need to figure out here how to turn the property values for the json object back into the objects they are designated to be then ensure that they are set to the node correctly.
• A good starting point would be to see how the property map is handled, then go from there.

In my opinion, the hardest part for this will probably be the decode logic, since you would have to assemble up rather the recursively build out the fields. I would also recommend trying to store as much information about the whole object tree in the propConfig object I mentioned in the decode section.

I definitely think this is worth doing, feel free to reach out here for help. I will be much more responsive on this going forward. Additionally thank you for your support in gogm and willingness to implement a cool feature like this :)

[erictg]

In the coming weeks i'll be adding more test and comment coverage, which should definitely help. If you come across code that doesn't make sense, feel free to ask what it does!

[notdodo]

Hi @erictg!

I took another approach: since all structures in Golang can be marshalled to a JSON string I've created a library that given a JSON input string it returns a JSON string with all flatten fields. The code is simpler than the one used for structs.

The library is here: https://github.com/notdodo/goflat

I think that this library (or just copy/paste the code) could be integrated here to do what we want to achieve.

[SomniVertix]

@notdodo Is this lib something that you would use in conjunction with gogm, or would you have to edit the gogm logic itself for this?

Im running into this same thing and wondering how to do this:

type IndexTemplate struct {
	gogm.BaseNode

	Name string `gogm:"name=Name" json:"index_pattern"`
	Spec struct {
		Mappings struct {
			Meta struct {
				Description string `gogm:"name=description" json:"description"`
			} `json:"_meta"`
		} `json:"mappings"`
		Retention             string `gogm:"name=Retention" json:"Retention"`
		PolicyName            string `gogm:"name=PolicyName" json:"PolicyName"`
		ComponentTemplateName string `gogm:"name=ComponentTemplateName" json:"ComponentTemplateName"`
		Environment           string `gogm:"name=Environment" json:"Environment"`
		TotalDocCount         int    `gogm:"name=TotalDocCount" json:"TotalDocCount"`
		TotalIndexCount       int    `gogm:"name=TotalIndexCount" json:"TotalIndexCount"`
		TotalShardCount       int    `gogm:"name=TotalShardCount" json:"TotalShardCount"`
	} `gogm:"name=Spec;properties" json:"spec"`

	ComponentTemplate *ComponentTemplate `gogm:"direction=incoming;relationship=MANAGES" json:"-"`
	Policy            *LifecyclePolicy   `gogm:"direction=incoming;relationship=MANAGES" json:"-"`
}