vector icon indicating copy to clipboard operation
vector copied to clipboard
verified publisher icon vectordotdev

feat(odbc source): Add ODBC source implementation

Open powerumc opened this issue 2 months ago • 4 comments

Summary

This PR implemented a new ODBC(Open Database Connectivity) Source.

Vector configuration

Manual configuration example

[sources.odbc]
type = "odbc"
connection_string = "driver={MySQL ODBC 8.0 ANSI Driver};server=localhost;port=13306;database=vector;uid=vector;pwd=vector;"
statement = "SELECT * FROM odbc_table WHERE id > ? LIMIT 1;"
schedule = "*/5 * * * * *"
schedule_timezone = "UTC"
last_run_metadata_path = "odbc_tracking.json"
tracking_columns = ["id"]
statement_init_params = { id = "0" }

[sinks.console]
type = "console"
inputs = ["odbc"]
encoding.codec = "json"

Create example sql file

cat << EOF > example.sql
DROP TABLE IF EXISTS odbc_table;
CREATE TABLE odbc_table
(
    id int auto_increment primary key,
    name varchar(255) null,
    datetime datetime null
);

INSERT INTO odbc_table (name, datetime) VALUES
('test1', now()),
('test2', now()),
('test3', now()),
('test4', now()),
('test5', now());
EOF

Configure ODBC and MariaDB Driver on MacOS

brew install unixodbc
brew install mariadb-connector-odbc

cat << EOF >> /opt/homebrew/etc/odbcinst.ini
[MariaDB Unicode]
Description=MariaDB Connector/ODBC v.3.0
Driver=/opt/homebrew/Cellar/mariadb-connector-odbc/3.2.7/lib/mariadb/libmaodbc.dylib
EOF

Run MariaDB docker container

docker run \
  --rm \
  --name mariadb \
  -e MYSQL_ROOT_PASSWORD=vector \
  -e MYSQL_USER=vector \
  -e MYSQL_PASSWORD=vector \
  -e MYSQL_DATABASE=vector_db \
  -v $(pwd)/example.sql:/docker-entrypoint-initdb.d/example.sql:ro \
  -p 3306:3306 \
  mariadb:latest

How did you test this PR?

I tested it with integration tests of two databases: MariaDB and Postgresql. (Testing the MySQL container integration in an ARM64 architecture is not simple, so I tested it with MariaDB. Instead, I manually tested MySQL locally.)

cargo vdev integration test odbc-mariadb
cargo vdev integration test odbc-postgresql

Change Type

  • [ ] Bug fix
  • [x] New feature
  • [ ] Non-functional (chore, refactoring, docs)
  • [ ] Performance

Is this a breaking change?

  • [ ] Yes
  • [x] No

Does this PR include user facing changes?

  • [x] Yes. Please add a changelog fragment based on our guidelines.
  • [ ] No. A maintainer will apply the no-changelog label to this PR.

References

  • Related: #22931

Notes

  • Please read our Vector contributor resources.
  • Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
  • Some CI checks run only after we manually approve them.
    • We recommend adding a pre-push hook, please see this template.
    • Alternatively, we recommend running the following locally before pushing to the remote branch:
      • make fmt
      • make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
      • make test
  • After a review is requested, please avoid force pushes to help us review incrementally.
    • Feel free to push as many commits as you want. They will be squashed into one before merging.
    • For example, you can run git merge origin master and git push.
  • If this PR introduces changes Vector dependencies (modifies Cargo.lock), please run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.

powerumc avatar Oct 22 '25 13:10 powerumc

Hi @pront @aramperes 👋 I’d really appreciate it if someone could take a look at this PR when you have a chance. Please let me know if there’s anything I can improve or adjust — happy to make changes based on your feedback. Thanks a lot!

powerumc avatar Oct 30 '25 09:10 powerumc

Hi @powerumc, please resolve the merge conflicts and look at the failing checks. We will take another look once those are addressed.

pront avatar Nov 12 '25 20:11 pront

Hi @powerumc, please resolve the merge conflicts and look at the failing checks. We will take another look once those are addressed.

This integration test was working fine before the merge on my mac. However, after merging the refactored code and running the tests locally, I’m now encountering failures.

Of course, it builds fine locally on my Mac. (cargo build --release --features sources-odbc,odbc-integration-tests)

I’m not sure yet whether the issue is caused by my changes or something else, and I’m continuing to investigate by trying different approaches. If you’ve seen similar cases or have any insights, I’d really appreciate your guidance.

Run `cargo vdev integration test odbc-mariadb -v` 
error: could not compile `vector` (lib test) due to 1 previous error
error: command `/usr/local/rustup/toolchains/1.91-aarch64-unknown-linux-gnu/bin/cargo test --no-run --message-format json-render-diagnostics --lib --features all-integration-tests --no-default-features` exited with code 101

You can see more detailed error information by running the command inside the Docker container vector-test-runner-1.91.

# /usr/local/rustup/toolchains/1.91-aarch64-unknown-linux-gnu/bin/cargo test --no-run --message-format json-render-diagnostics --lib --features all-integration-tests --no-default-features

...
info: installing component 'rustc'
info: installing component 'rustfmt'
error[E0514]: found crate `lalrpop` compiled by an incompatible version of rustc
 --> /usr/local/cargo/git/checkouts/vrl-7b350af036089ddd/1f0d8a5/build.rs:1:1
  |
1 | extern crate lalrpop;
  | ^^^^^^^^^^^^^^^^^^^^^
  |
  = note: the following crate versions were found:
          crate `lalrpop` compiled by rustc 1.91.1 (ed61e7d7e 2025-11-07): /home/vector/target/debug/deps/liblalrpop-e5c02c887748f8d8.rlib
  = help: please recompile that crate using this compiler (rustc 1.90.0 (1159e78c4 2025-09-14)) (consider running `cargo clean` first)

error[E0514]: found crate `ua_parser` compiled by an incompatible version of rustc
  --> /usr/local/cargo/git/checkouts/vrl-7b350af036089ddd/1f0d8a5/build.rs:11:5
   |
11 | use ua_parser::device::Flag;
   |     ^^^^^^^^^
   |
   = note: the following crate versions were found:
           crate `ua_parser` compiled by rustc 1.91.1 (ed61e7d7e 2025-11-07): /home/vector/target/debug/deps/libua_parser-0068263c251d7689.rlib
   = help: please recompile that crate using this compiler (rustc 1.90.0 (1159e78c4 2025-09-14)) (consider running `cargo clean` first)

error[E0514]: found crate `ua_parser` compiled by an incompatible version of rustc
  --> /usr/local/cargo/git/checkouts/vrl-7b350af036089ddd/1f0d8a5/build.rs:69:18
   |
69 |     let regexes: ua_parser::Regexes =
   |                  ^^^^^^^^^
   |
   = note: the following crate versions were found:
           crate `ua_parser` compiled by rustc 1.91.1 (ed61e7d7e 2025-11-07): /home/vector/target/debug/deps/libua_parser-0068263c251d7689.rlib
   = help: please recompile that crate using this compiler (rustc 1.90.0 (1159e78c4 2025-09-14)) (consider running `cargo clean` first)

error[E0514]: found crate `serde_yaml` compiled by an incompatible version of rustc
  --> /usr/local/cargo/git/checkouts/vrl-7b350af036089ddd/1f0d8a5/build.rs:70:9
   |
70 |         serde_yaml::from_slice(&regexes).expect("Regex file is not valid yaml");
   |         ^^^^^^^^^^
   |
   = note: the following crate versions were found:
           crate `serde_yaml` compiled by rustc 1.91.1 (ed61e7d7e 2025-11-07): /home/vector/target/debug/deps/libserde_yaml-b0a611a1255d908c.rlib
   = help: please recompile that crate using this compiler (rustc 1.90.0 (1159e78c4 2025-09-14)) (consider running `cargo clean` first)

For more information about this error, try `rustc --explain E0514`.
error: could not compile `vrl` (build script) due to 4 previous errors
error[E0514]: found crate `unicode_segmentation` compiled by an incompatible version of rustc
 --> /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/convert_case-0.7.1/src/boundary.rs:1:5
  |
1 | use unicode_segmentation::UnicodeSegmentation;
  |     ^^^^^^^^^^^^^^^^^^^^
  |
  = note: the following crate versions were found:
          crate `unicode_segmentation` compiled by <unknown rustc version>: /home/vector/target/debug/deps/libunicode_segmentation-06119734fac1cf78.rmeta
  = help: please recompile that crate using this compiler (rustc 1.83.0 (90b35a623 2024-11-26)) (consider running `cargo clean` first)

error[E0514]: found crate `unicode_segmentation` compiled by an incompatible version of rustc
 --> /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/convert_case-0.7.1/src/pattern.rs:6:5
  |
6 | use unicode_segmentation::UnicodeSegmentation;
  |     ^^^^^^^^^^^^^^^^^^^^
  |
  = note: the following crate versions were found:
          crate `unicode_segmentation` compiled by <unknown rustc version>: /home/vector/target/debug/deps/libunicode_segmentation-06119734fac1cf78.rmeta
  = help: please recompile that crate using this compiler (rustc 1.83.0 (90b35a623 2024-11-26)) (consider running `cargo clean` first)

error[E0599]: no method named `grapheme_indices` found for reference `&str` in the current scope
   --> /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/convert_case-0.7.1/src/boundary.rs:420:52
    |
420 |     let (indices, graphemes): (Vec<_>, Vec<_>) = s.grapheme_indices(true).unzip();
    |                                                    ^^^^^^^^^^^^^^^^
    |
help: there is a method `char_indices` with a similar name, but with different arguments
   --> /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/str/mod.rs:896:5

error[E0599]: no method named `graphemes` found for reference `&str` in the current scope
  --> /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/convert_case-0.7.1/src/pattern.rs:23:38
   |
23 |                 let mut chars = word.graphemes(true);
   |                                      ^^^^^^^^^ method not found in `&str`

error[E0599]: no method named `graphemes` found for reference `&str` in the current scope
  --> /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/convert_case-0.7.1/src/pattern.rs:31:38
   |
31 |                 let mut chars = word.graphemes(true);
   |                                      ^^^^^^^^^ method not found in `&str`

Some errors have detailed explanations: E0514, E0599.
For more information about an error, try `rustc --explain E0514`.
error: could not compile `convert_case` (lib) due to 5 previous errors
{"reason":"build-script-executed","package_id":"registry+https://github.com/rust-lang/crates.io-index#[email protected]","linked_libs":["static=ssl","static=crypto"],"linked_paths":["native=/home/vector/target/debug/build/openssl-sys-bb6c5d7608652f5d/out/openssl-build/install/lib"],"cfgs":["osslconf=\"OPENSSL_NO_IDEA\"","osslconf=\"OPENSSL_NO_CAMELLIA\"","osslconf=\"OPENSSL_NO_COMP\"","osslconf=\"OPENSSL_NO_SSL3_METHOD\"","osslconf=\"OPENSSL_NO_SEED\"","openssl","ossl340","ossl330","ossl320","ossl300","ossl101","ossl102","ossl102f","ossl102h","ossl110","ossl110f","ossl110g","ossl110h","ossl111","ossl111b","ossl111c","ossl111d"],"env":[],"out_dir":"/home/vector/target/debug/build/openssl-sys-bb6c5d7608652f5d/out"}
{"reason":"build-finished","success":false}

powerumc avatar Nov 13 '25 09:11 powerumc

Hi @powerumc, please resolve the merge conflicts and look at the failing checks. We will take another look once those are addressed.

This integration test was working fine before the merge on my mac. However, after merging the refactored code and running the tests locally, I’m now encountering failures.

Of course, it builds fine locally on my Mac. (cargo build --release --features sources-odbc,odbc-integration-tests)

I’m not sure yet whether the issue is caused by my changes or something else, and I’m continuing to investigate by trying different approaches. If you’ve seen similar cases or have any insights, I’d really appreciate your guidance.

Run cargo vdev integration test odbc-mariadb -v

The issue was caused by the missing unixodbc-dev package. (b599b75) After installing it, the integration tests are now passing.

powerumc avatar Nov 16 '25 12:11 powerumc

Hi, @pront, I’ve now resolved the merge conflicts and fixed all failing checks. I’d appreciate it if you could review the PR again when you have time.

Thank you!

powerumc avatar Dec 19 '25 00:12 powerumc