goavro icon indicating copy to clipboard operation
goavro copied to clipboard

Advice request to encode complex structure

Open maeglindeveloper opened this issue 4 years ago • 6 comments

Hi everyone,

I'm actually new in linkedin/goavro lib, and I'm trying to use the lib to encode a complex structure. I have the following code to encode a nested User structure:

package main

import (
	"encoding/json"
	"fmt"
	"time"

	hambda "github.com/hamba/avro"
	"github.com/linkedin/goavro"
)

type Phone struct {
	PhoneNumber string `avro:"phoneNumber" json:"phoneNumber"`
}

type Base struct {
	ID        string    `avro:"id" json:"id"`
	CreatedAt time.Time `avro:"createdAt" json:"createdAt"`
}

type User struct {
	Base
	Firstname string `avro:"firstName" json:"firstName"`
	Lastname  string `avro:"lastName" json:"lastName"`
	Count     int    `avro:"count" json:"count"`
	Phone     *Phone `avro:"phone" json:"phone"`
}

var mockUser = User{
	Firstname: "Maeglin",
	Lastname:  "Dev",
}

func main() {
	schema, err := hambda.ParseFiles("schema.avsc")
	if err != nil {
		panic(err)
	}
	codec, err := goavro.NewCodec(schema.String())
	if err != nil {
		panic(err)
	}

	textual, err := json.Marshal(mockUser)
	if err != nil {
		panic(err)
	}

	native, _, err := codec.NativeFromTextual(textual)
	if err != nil {
		panic(err)
	}

	value, err := codec.BinaryFromNative(nil, native)
	if err != nil {
		panic(err)
	}

	fmt.Println(value)
}

and the following schema.avsc file:

{
  "type": "record",
  "name": "User",
  "namespace": "mycompany.avro.types",
  "fields": [
    {
      "name": "id",
      "type": "string"
    },
    {
      "name": "createdAt",
      "type": "string"
    },
    {
      "name": "firstName",
      "type": "string"
    },
    {
      "name": "lastName",
      "type": "string"
    },
    {
      "name": "count",
      "type": "int"
    },
    {
      "name": "phone",
      "type": ["null", {
        "type": "record",
        "name": "Phone",
        "namespace": "mycompany.avro.types",
        "fields": [
          {
            "name": "phoneNumber",
            "type": "string"
          }
        ]
      }]
    }
  ]
}

Of course, that code works, but I feel like I have to much steps to build the binary avro data ...

  • First I need to marshal the data to json.
  • Second I convert my json to native, getting a map[string]interface{}
  • Third I can then convert the native map to binary avro []byte.

But is there any other way (faster one?) to convert my User to that avro binary without using json marshal step ?

Thanks by advance ! :) @karrick

maeglindeveloper avatar Jan 16 '20 14:01 maeglindeveloper