DBZ-4315 Addition of Oracle connector

[mdrillin]

Draft PR - (in progress) Addition of Oracle connector to the UI

[gunnarmorling]

So let's move forward here:

• Use placeholder icon for Oracle connector
• Show in README how to create a custom container image with the driver

[mdrillin]

Pushed changes to utilize a placeholder image for Oracle. I could use some help with the build setup. Currently there are dependencies in the backend pom - I'm assuming those would need to be removed? Also, should there be some flag or other so that the oracle connector type is not shown as an option if driver is not available?

[Naros]

Also, should there be some flag or other so that the oracle connector type is not shown as an option if driver is not available?

I think it would be sufficient on the Integrator interface to have a method such as isAvailable() that has a default implementation that returns true and then Oracle can override this and verify the existence of the driver on the classpath? We could use this to toggle visibility for now, and it's generic enough that other connectors could rely on this in the future based on some other reason.

But did we find a solution to the Quarkus issue with the classpath for users adding the driver after the fact? If you recall we had issues just adding the jar to the /lib directory of the image due to the classpath being specified in the runner's manifest IIRC which is precisely what lead us to use the pom for the driver. I wonder if the dependency is marked provided does Quarkus still add it to the list of classpath dependencies in the manifest? If it does, then theoretically users dropping the jar into the /lib folder would work but if it isn't; we'd face the same problem we had originally when we decided to directly put the driver in the POM dependencies list.

[rk3rn3r]

@Naros in the light of DBZ-4395 where we want to remove the UI backend at all in favor of the Debezium Connect REST extension, I think we should not invest too much into that topic. That provided dependency solution is a good way to solve the issue for the time being. Do you think you could do that @mdrillin?

• adding the driver as a provided dependency
• add some lines in the README on how to provide that file in the Quarkus dir (at the moment it's the jar file dir as we have/had to switch to the uber-jar. When we return to the fast-jar, we need to move the docs to point to the Quarkus app directory. Config example for a docker-composer setup.

[mdrillin]

pushed update for oracle driver dependency.

[gunnarmorling]

@mdrillin, so what's needed to un-draft and complete this one? Are @indraraj and/or @uidoyen aware of the required steps?

[mdrillin]

@gunnarmorling - I went ahead and undrafted it. The UI parts of it had previously been tested - I think the only thing remaining are build / build instructions which @rk3rn3r is probably most familiar with.

[indraraj]

Hey @mdrillin I remember you have shared the values of the required fields to test UI, I am not able to find them now. Can you share them here again?

[mdrillin]

@indraraj - I just emailed you and @uidoyen the connection properties I've been using for all the connector types, including oracle. Chris helped me to set up the Oracle container and here are the properties I have been using for that

Namespace: server1 Hostname: (determined from local ip) Port: 1521 User: c##dbzuser Database: ORCLCDB Log Mining Strategy: redo_log_catalog Kafka broker addresses: kafka:9092 Database history topic name: schema-changes.inventory

I am not sure all of these are strictly required, but was working for me.

[indraraj]

Thanks, @mdrillin :)

[uidoyen]

@rk3rn3r @Naros How do we run Oracle db to connect with? I couldn't see a container running.

[indraraj]

FYI @uidoyen recently, we have relaced _ with & in handling properties name coming from the backend response. You may need to update the same in this PR

[Naros]

@ani-sha i thought we fixed testSqlServerCreateConnectorEndpoint as I rebased this PR against main today.

[ani-sha]

@Naros Yes. That's strange. Please create a Jira, I will take a look into it.

[Naros]

Hi @rk3rn3r @uidoyen @indraraj so I've pushed a small commit to this PR that does the following:

1. Added a new docker-compose-oracle file useful for building the UI with Oracle. This uses the private repository on quay.io to get the stable Oracle image. This image cannot be redistributed and requires authentication so if you cannot access the repository, please let me know.
2. Added ojdbc8.jar to the backend dependencies as a required dependency. I explicitly remove this from the standard backend build performed by the docker-compose YAML file, the oracle flavor leaves this file intact. Per a discussion with Alexy from Quarkus team, this is our only option. This allows the backend to communicate with Oracle via validation or any other backend tasks.
3. Added a special connect docker build to pull in ojdbc8 from maven central when using Oracle compose file. This is required so that the connect container has the JDBC driver as well.
4. Added a special oracle build to add the initializing script to the database image when using Oracle compose file. The namespace is called "DEBEZIUM" rather than "INVENTORY" but I kept all the table object names identical to that of SQL Server.

In terms of test properties, here is what should work and I have validated this with the PR changes:

Namespace: server1 Hostname: dbzui-db-oracle (this is now a service when using the Oracle compose file, can just use it) Port: 1521 User: c##dbzuser Database: ORCLCDB PDB name: ORCLPDB1 Log Mining Strategy: online_catalog Kafka broker addresses: kafka:9092 Database history topic name: schema-changes.inventory

Let me know if you guys have any questions or want any further changes.

[Naros]

@uidoyen I've added a small improvement for the Filters for the Oracle connector. As I mentioned in the comment on the commit, this will only improve the performance for PDB installations, we need to expose a method publically on the connection class so that we can take advantage of the best way across non-PDB installations. I'll work to get that ASAP.

[Naros]

@uidoyen, I've rebased this on the changes for database history stuff to work, lets see if it now works.

[uidoyen]

@uidoyen, I've rebased this on the changes for database history stuff to work, lets see if it now works.

Thanks, @Naros! I will test the UI and let you know.

[uidoyen]

@Naros still can't see database.history.kafka.bootstrap.servers?

[uidoyen]

@Naros I tried creating the Oracle connector and It seems to be working fine. E2E tests are fine too after updating the workflows .yaml.

I have a question though, Do we need to maintain both Dockerfiles and docker-compose for Oracle and other connectors?

[Naros]

This now depends on https://github.com/debezium/debezium-ui/pull/649 since the changes were merged to core @uidoyen. I'll check with @vjuranek to see where we stand on the draft PR.

[vjuranek]

This now depends on #649 since the changes were merged to core @uidoyen. I'll check with @vjuranek to see where we stand on the draft PR.

this PR waits for https://github.com/debezium/debezium/pull/3862. If/once this PR is merged, #649 should be ready.

[Naros]

I think this is now dependant on PR #655 and PR #656.

[Naros]

Looks like we're back to a green build @uidoyen so I'm looking at the connector deployment, now.

[Naros]

I also tested the connector with the following configs: And connect spun up the connector fine:

So I think we're good to merge this, if this looks fine w/you @uidoyen .