xstevens
Formatting of WAL data
I am searching for a way to trigger a message to Kafka on INSERT, UPDATE, and DELETE to postgres Tables where the Data inserted / Updated / Deleted is contained as JSON in Kafka Message (for Data Replication into NoSQL Couchbase). Your project seems ideally conceptualized to accomplish this.
In the notes, you wrote "I could not find any material on how to actually decode that data. If you know how to decode WAL data from PostgreSQL " ... This note suggests that the data currently contained in messages is all binary. As I look at your https://github.com/xstevens/decoderbufs project though, it seems as though you have made progress on this ...
From your perspective, would the combination of these 2 projects represent a path to achieve the NoSQL replication we are seeking to achieve?
This project was created before I knew about Postgres 9.4 Logical WAL Decoding.
pg_kafka is geared toward pushing on a trigger or calling manually from SQL. This approach has a number of downsides because it's push based and you would have to figure out what to do with messages when your destination (kafka) is down/unreachable.
decoderbufs is designed to be a Logical WAL Decoding plugin that transforms the data into a Protocol Buffer format. If you're going the logical decoding route you'll also need a way to connect to Postgres and consume the logical decoding stream. See pg_recvlogical as an example.
Let me know if you have any other questions.
