linkedin
LongLongColumn, BigInteger and Avro serialization
Hello,
While setting things up Databus with MySQL this week, I came across a problem. This happens while running the MySQL example
I am using MySQL 5.7, but I think it's not relevant to this problem.
At this line in ORListener.java, the value of a LongLongColumn is being converted into a BigInteger. And it looks like that that was a fairly recent change. Earlier, it was taken as a Long value.
This seems to be causing problems with Avro serialization. For ex. the schema declares the id field as [ "long", "null" ], which gets mapped to a Long typed field in Person_V1 class.
Since the types don't match, we get an avro exception.
It's of course possible that I am missing something here. But could you please throw some light on this? Should I send a PR that reverts the mapping back to Long?
Hi,
It looks like the change was to support unsigned long integers. Since Java does not support unsigned integers, the change stores the value in BigInteger so it can fit 64-bit unsigned numbers.
You are right, it's probably best to revert to the old logic for signed integers and use a BigInteger only when there is a signed long that cannot fit.
On Sun, Feb 5, 2017 at 12:18 AM, Bhashit [email protected] wrote:
Hello,
While setting things up Databus with MySQL this week, I came across a problem. This happens while running the MySQL example https://github.com/linkedin/databus/wiki/Databus-for-MySQL
I am using MySQL 5.7, but I think it's not relevant to this problem.
At this line in ORListener.java https://github.com/linkedin/databus/blob/master/databus2-relay/databus2-event-producer-or/src/main/java/com/linkedin/databus2/producers/ORListener.java#L615, the value of a LongLongColumn is being converted into a BigInteger. And it looks like that that was a fairly recent change. Earlier, it was taken as a Long value.
This seems to be causing problems with Avro serialization. For ex. the schema declares the id field as [ "long", "null" ], which gets mapped to a Long typed field in Person_V1 class https://github.com/linkedin/databus/blob/b775aeb2a0bd5448983607dc636c39f17227969e/databus2-example/databus2-example-person/src/main/java/com/linkedin/events/example/person/Person_V1.java#L32 .
Since the types don't match, we get an avro exception.
It's of course possible that I am missing something here. But could you please throw some light on this? Should I send a PR that reverts the mapping back to Long?
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/linkedin/databus/issues/88, or mute the thread https://github.com/notifications/unsubscribe-auth/AA4sG3gTI6NaNV_1gtzkEJHSrJi9xEHBks5rZYXPgaJpZM4L3bhu .
when i run this demo it throw the exception
UnsupportedKeyException: Bad key type: java.math.BigInteger
017-04-29 23:54:04,663 +18541 [ORListener_person] (INFO) {DatabusThreadBase} Starting source: or_test.person
2017-04-29 23:54:04,664 +18542 [ORListener_person] (INFO) {DatabusThreadBase} UPDATE or_test.person: 1
2017-04-29 23:54:04,670 +18548 [ORListener_person] (ERROR) {OpenReplicator_person} Got UnsupportedKeyException :
com.linkedin.databus.core.UnsupportedKeyException: Bad key type: java.math.BigInteger
at com.linkedin.databus.core.DbusEventKey.
@769344359
I found it is because the bigint type of id column in the table defined in create_person.sql, and this exception is gone after I replace the bigint with int in create_person.sql.
Hope it helps.
https://github.com/linkedin/databus/pull/98 with this branch,the example run properly
