erddap icon indicating copy to clipboard operation
erddap copied to clipboard

Add support for direct reads from duckdb

Open rmendels opened this issue 8 months ago • 2 comments

Support reading using duckdb directly not using JDBC, and not just .db files. This opens up support for a number of other formats that duckdb can read from including Apache iceberg

rmendels avatar Apr 24 '25 14:04 rmendels

For Java I only see JDBC in their client list:

https://duckdb.org/docs/stable/clients/overview.html

srstsavage avatar Apr 24 '25 17:04 srstsavage

@srstsavage I believe you are correct, but the issue has other do with how duckdb works and how ERDDAP accesses things in its JDBC handling. The duckdb JDBC can only read from .db files, which defeats the whole purpose, you would have to reformat things, rather than have duckdb handle reading other formats. However you can "trick" duckdb by setting up a .db file that does nothing else but point to the files you actually want to read, which are in a different format that duckdb handles. The problem at least so far as I have been able to work out. (and I am not very knowledgeable about this), is when you do this ERDDAP doesn't like it, as it can't find what it considers to be a table in the .db file that is pointing to the other file. I am still working on this when I am not being inundated with meetings. I have done one small example where a straight Java application using the JDBC reads this indirect file just fine, while ERDDAP throws an error. Could I be doing something stupid in ERDDAP, very possibly, but if there is a way to get this to work duckDB is very fast and supports any number of other formats, so we get those for "free" in a sense.

rmendels avatar Apr 24 '25 18:04 rmendels