Jack Vanlightly

Indeed generalising node roles is not an easy thing to design well. I think for now I will play with using existing workloads and proxying. BookKeeper is after all, log...

The above looks like an elegant design. This leaves open the question of consumer offset tracking. Without competing consumers, manually managed consumer offsets are easy. Just periodically write the last...

Great, currently I have only implemented the methods I need at work, but I will start implementing the rest of the API methods. The code is currently here: https://github.com/Vanlightly/r.aws.secretsmanager. Once...

Cool, thanks @leeper, I'm sure it will be helpful!

Check my blog post: https://jack-vanlightly.com/blog/2018/4/28/sql-server-cdc-to-redshift-pipeline It explains the different strategies that either respect global or table level transactional boundaries and those that simply stream changes as they occur. Reading from...

It is a tempting idea as it makes the solution simpler, but it does have two drawbacks: 1. You'll lose the natural order of the events. From my experiments I...

I've never used them, but products such as Attunity and Debezium read the transaction log directly, they might offer some functionality that you need.

Changing the routing without some kind of migration will result in effective data loss. Plus an md5 hash is relatively expensive. The issue here is that ledger ids are all...