dbeam
dbeam copied to clipboard
spotify
DBeam exports SQL tables into Avro files using JDBC and Apache Beam
When an Avro user-supplied Avro schema doesn't correspond to actual columns returned by SQL SELECT statement, dbeam produces avro data files, which cause exceptions be thrown when users try to...
Is there any partucular reason why dbeam releases library as Java 8 artifact? https://github.com/spotify/dbeam/blob/1668c7d78aad4a7aa741e82184b8ad10adc84da4/pom.xml#L101 ``` 8 ``` Especially given thefact that dbeam requires at least JDK 11 for compilation https://github.com/spotify/dbeam/blob/1668c7d78aad4a7aa741e82184b8ad10adc84da4/pom.xml#L532-L534...
Postgres has a non-standard SQL concept called comment: https://www.postgresql.org/docs/current/sql-comment.html It shows up when you type `\d+` in `psql` and a few other places and is a pretty nice way of...
Hey 😊 I want to contribute the following test: Test that `actual.sqlQueryWithLimitOne()` is equal to `"SELECT * FROM (SELECT * FROM COFFEES WHERE SIZE > 10) as user_sql_query WHERE 1=1...
Requires more testing. Works locally in IDEA. Related to #209 .
1. Encode JDBC ResultSet into ByteBuffer using `directBinaryEncoder`, write using `appendEncoded()`. This avoids a bit of copying bytes between buffers. 2. Use a `BlockingQueue` to asynchronously read from JDBC and...
I understand dbeam officially does not support S3. I kinda of hopping it might work out of the box because Beam supports S3 as its file io. However I got...
Hello, I was trying out this tool and however could not compile on windows. It will be great if you could help on this. I was also trying to get...
It would be great if dbeam allows direct streaming into BQ. Current approach requires second job to accomplish this, loading from gcs to bq.
It would be good to provide [GitHub release artifacts](https://github.com/spotify/dbeam/releases) from sbt-pack task running on travis. There is some good example of this in ratatool: https://github.com/spotify/ratatool/blob/master/.travis.yml#L27