avro-serde-php icon indicating copy to clipboard operation
avro-serde-php copied to clipboard

schemas registry and refernce to schemas

Open lubagov opened this issue 4 years ago • 5 comments

I'm looking at the library for now, trying to figure out the possibility of using schema references. Is this supported in the library? For example I have 2 files:

com.myapp.Children.avsc:

{
  "namespace": "com.myapp",
  "type": "record",
  "name": "Children",
  "fields": [
    {"name": "first_name", "type": "string"},
    {"name": "last_name", "type": "string"}
  ]
}

com.myapp.People.avsc:

{
  "namespace": "com.myapp",
  "type": "record",
  "name": "People",
  "fields": [
    {"name": "id", "type": "string"},
    {"name": "first_name", "type": "string"},
    {"name": "last_name", "type": "string"},
    {
      "name": "children",
      "type": ["null",
        {
          "type": "array",
          "items": "Children"
        }],
      "doc": ""
    }
  ]
}

And I want to get a schema, that matches People.avsc.

My test code:

        $schemaRegistryClient = new CachedRegistry(
            new PromisingRegistry(
                new Client(['base_uri' => 'http://****:8081'])
            ),
            new AvroObjectCacheAdapter()
        );

        $recordSerializer = new RecordSerializer(
            $schemaRegistryClient,
            [
                // If you want to auto-register missing schemas set this to true
                RecordSerializer::OPTION_REGISTER_MISSING_SCHEMAS => false,
                // If you want to auto-register missing subjects set this to true
                RecordSerializer::OPTION_REGISTER_MISSING_SUBJECTS => false,
            ]
        );

        $baseDir = "./avro";

        $inflector = static function ($record, bool $isKey) {
            echo "Searching for:".get_class($record);
            $ext = $isKey ? '.key.avsc' : '.avsc';
            $fileName = is_object($record)
                ? str_replace('\\', '.', get_class($record))
                : 'default';

            return $fileName . $ext;
        };

        $resolver = new FileResolver($baseDir, $inflector);
        $normalizer = new GetSetMethodNormalizer();
        $encoder = new AvroSerDeEncoder($recordSerializer);

        $symfonySerializer = new Serializer([$normalizer], [$encoder]);

        $children=new Children("Nikolai","Lubiagov");

        $people=new People("Nikolai","Lubiagov", [$children]);

        $schemac= $resolver->valueSchemaFor($children);
        $schemap= $resolver->valueSchemaFor($people);

I don't see any code at all to resolve the link. Or how to transfer the already obtained Children scheme to People, is this supported?

I also can't get already registread schema, with refernces over: $sc=$schemaRegistryClient->schemaForSubjectAndVersion("phptopic-value",1)->wait();

{ "subject":"phptopic-value", "version":1, "id":70, "references":[ { "name":"Children", "subject":"Children", "version":1 } ], "schema":"{"type":"record","name":"People","namespace":"com.myapp","fields":[{"name":"id","type":"string"},{"name":"first_name","type":"string"},{"name":"last_name","type":"string"},{"name":"children","type":["null",{"type":"array","items":"Children"}],"doc":""}]}" }

I found this https://github.com/flix-tech/avro-serde-php/commit/322fcd66d2ae11572920ed0c91bf58f533851bad for previously defined types, but if type defined in another subject, it is not fit...?

lubagov avatar Oct 17 '21 17:10 lubagov

Is that what you mean ? There are some good examples in the test Filder.

https://github.com/flix-tech/avro-serde-php/blob/master/test/Objects/Schema/Generation/Fixture/RecordWithRecordType.php

BeyerJC avatar Oct 29 '21 18:10 BeyerJC

I mean schema references. When one schema is link to another schema. This example forms one schema with 2 types. I also need Protobuf and it's not there.

(I was under the impression that it's easier to make your own library than trying to adapt that)

The annotated method also seems to be of little use when using several clients implemented in different programming languages? Seems, no sense to describe it, it's easier to describe the once in AVRO / Protobuf, and generate the code...

lubagov avatar Oct 29 '21 20:10 lubagov

@lubagov you are correct in that the tying this library does not support schema references from other topics right now. In general I am planning on adding these features, not only references, but also Protobuf & JSON schemas. It was on the plate for over a year now, but I believe that I can finally find time for the implementation.

I will leave this open as a reference.

tPl0ch avatar Oct 30 '21 10:10 tPl0ch

@tPl0ch any update on possibily to reference schemas from other topics?

AlexOstrovsky avatar Sep 11 '23 15:09 AlexOstrovsky

@AlexOstrovsky currently there is no time for me to work on this, I am happily accepting PRs that will add features, but this project currently has no priority. Sorry. :(

tPl0ch avatar Jan 26 '24 13:01 tPl0ch