kafka-connect-jdbc
kafka-connect-jdbc copied to clipboard
confluentinc
Support Postgres partitioned table #1092
Problem
Postgres partitioned tables are not supported after this change in the driver.
#1092
Solution
Map PARTITIONED TABLE as a valid value;
not yet enabled by default, although that could be added still.
Does this solution apply anywhere else?
- [ ] yes
- [x] no
Test Strategy
Testing done:
- [x] Unit tests
- [x] Integration tests
- [ ] System tests
- [ ] Manual tests
Release Plan
? Merge to master I guess; no plans for backports unless specifically requested.
As indicated I've not (yet) run a test using this branch as a plugin in kafka-connect. If this has a chance of getting merged soon-ish I could look into further manual testing.
@ddasarathan (or anyone else) would you mind reviewing this pull request?
Any updates on this issue? We would like to see this code merged as we currently are forced to use version 42.2.10 of the driver which contains a few reported security vulnerabilities, including:
- https://access.redhat.com/security/cve/CVE-2022-26520
- https://access.redhat.com/security/cve/CVE-2020-13692
- https://access.redhat.com/security/cve/CVE-2022-21724 If we can provide any assistance with testing or anything else please let me know. Thanks!
I've not yet tested this, as I had been waiting for any kind of response from Confluent before I spend more effort on this. If you're willing and able to test these changes that should at least give more confidence for a merge if Confluent get involved. And any feedback on whether partitioned tables should be support by default would be appreciated as well.
I just merged any changes from master into my branch https://github.com/confluentinc/kafka-connect-jdbc/pull/1156/commits/a7190c9ac0d399bd65bea84c7c56df65c82178e5 & signed the CLA.
We are also facing the same issue partition table with postgresql-42.3.3.jar and with io.confluent:kafka-connect-jdbc:10.4.0?
Any plan to merge this one?
@sagarrao12 I've added a PostgresPartitionedIT. Could you help update the docs to see this through?
@rajdangwal or @sagarrao12 could either of you review and help finish this one?
Any news when this could be merged? Would be great if it can be fixed this week as we are going live next week
indeed that would be great, that took me a while to find out the error I have was due to this bug.
Awesome, glad to see this merged. Any hints on how to activate the property?
Awesome, glad to see this merged. Any hints on how to activate the property?
I suppose it would be table.types: PARTITIONED_TABLE,TABLE
but it gives the error
message: |- PUT /connectors/vmaas-vlog-dev-jdbc-sink/config returned 400 (Bad Request): Connector configuration is invalid and contains the following 1 error(s): Invalid value PARTITIONED_TABLE for configuration table.types: Invalid enumerator You can also find the above list of errors at the endpoint/connector-plugins/{connectorType}/config/validate
@beer1970 From what I can see we still have a wait for a 10.6.0 release, as the latest is still 10.5.2.
@rajdangwal did a couple of releases two days ago, but no release yet of either 10.5.x or 10.6.x. When can we expect those?
@beer1970 There's a new confluentinc/kafka-connect-jdbc:10.6.0 release that you can try as well; I managed to deploy my connectors with "table.types": "PARTITIONED TABLE,TABLE" as configuration. Note how its using a space rather than underscore, as opposed to your error above. Still waiting to see if records will get pushed through to the partitioned table :crossed_fingers:.
have you resolve it ? i meet same problem kafka-connect-jdbc-10.5.3.jar postgresql-42.3.3.jar, configure "table.types":"PARTITIONED TABLE,TABLE"
