question: linking different AVRO schemas with OWL/RDFS ontologies

[nicolaemarasoiu]

Hello, I like your project very much as bridging the gap from AVRO to semantic.

I would have a question, could you please help me with this one?

We use Apache Kafka and microservices publishing and subscribing to the data exposed by other microservices. We use AVRO. Different microservices publish one or more topics like events topics or facts/entities/current state topics. And consume from others.

Currently the data is a bit heterogenous and a lot of guesswork happens in terms of "is there any microservice publishing the data i need?", "where to find a data source for customer like entities?", "what data does team x publish?".

We are exploring semantic web & ontologies for:

1. linking different topics AVRO schemas with OWL/RDFS ontologies

2. marking metadata on topics to be searched for

3. giving more semantics to data and allow more ways to navigate it for users across the enterprise

It seems to me that the problem starts perhaps with proper modelling. Since AVRO is already a schema (for JSON documents), I would like to explore ways to go into the semantic layer of relating various concepts exposed by these microservices so that later on people and microservices can query this set of relations and find data sources that they would not normally be aware of, thus increasing productivity.

Do you know of any project that integrated Kafka or AVRO with OWL Ontologies?

Or more generally in the land of microservices do you know of pragmatic and nice application of ontologies to help offer visibility on data and various parties can search for "customer like entities" for instance, across the organisation?

Do you see other technologies for achieving our goals of increasing transparency of domains and their interconnections?

Thank you very much, for more context:

https://www.quora.com/unanswered/How-can-AVRO-schemas-used-in-Kafka-topics-be-reused-to-generate-OWL-ontologies-and-how-can-OWL-ontologies-be-used-to-generate-AVRO-schemas-What-are-other-ways-to-inter-op

https://www.quora.com/unanswered/How-would-you-mix-OWL-ontologies-with-AVRO-schemas-needed-in-Apache-Kafka-to-describe-the-same-domain-without-much-duplication

https://www.quora.com/unanswered/What-are-the-ways-to-define-a-domain-model-that-can-be-freely-shared-across-organisations-and-be-laterally-developed-in-a-decentralised-fashion-allow-for-different-ways-of-modelling-the-Customer-Is-OWL-ontologies-a

https://www.quora.com/What-kind-of-metadata-would-you-use-to-associate-data-sources-like-Kafka-topics-or-db-tables-that-they-represent-Entity-X-and-they-are-owned-by-team-Y-Would-OWL-Ontologies-be-ok-to-model-entities-across

Thank you very much,

Nicu

Thank you, Nicolae Marasoiu Scala Engineer Orion, OVO Group

https://stackoverflow.com/questions/61270853/how-can-we-use-owl-ontologies-to-link-avro-schemas-from-different-microservices

[dfornika]

Hi @nicolaemarasoiu, I'm not sure if this is the best place to have an open-ended discussion about this, but I'm also interested in this topic. If there isn't much discussion here, maybe it could be discussed in the schema_salad gitter or somewhere in the Confluent Community Slack workspace?