trino
trino copied to clipboard
trinodb
Add support of JDBC store for the dynamic catalog feature
Description
Based on the new dynamic catalog, compatible with file and memory. I've try to add a compatibility with a jdbc database for store them. The main feature of dynamic catalog come from this Issue : https://github.com/trinodb/trino/issues/12709#issuecomment-1290290147
Additional context and related issues
Goal is to avoid the need of a persistant volume with a persistent dynamic catalog.
Release notes
( ) This is not user-visible or docs only and no release notes are required. ( ) Release notes are required, please propose a release note for me. (x) Release notes are required, with the following suggested text:
# DynamicCatalog
* Add support for storing the dynamic catalog through a jdbc connector
Due to problem of duplicate classes I've had some exclusion inside other plugin. I'm not sure it's the right way to deal with that problem
This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
@dain can you look and chime in to see if this approach is in line with the planned features?
@trinodb/devrel could one of you find someone to give this PR a look while @dain is out? Just to verify best practices are being followed.
On my side, we are actively using it in production right now, will be happy to go further on it.
On my side, we are actively using it in production right now, will be happy to go further on it.
@cdrappier Hi, thanks for your work. But have you encountered the situation like these issues while using dynamic catalogs? https://github.com/trinodb/trino/issues/18040 Unknown handle id: hive:hive:XXX:io.trino.plugin.hive.HiveTableHandle when using dynamic catalogs https://github.com/trinodb/trino/issues/18053 Dynamic Catalogs - Sync Catalog Properties to Worker Node
Can you rebase this PR @cdrappier
@dain you mentioned that this feature might be a good add ... can you help with providing guidance with the PR?
This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua
@cdrappier could you potentially rebase so we can verify it against CI and then we can try to get reviewers involved again to get this to merge.
This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua
Closing this pull request, as it has been stale for six weeks. Feel free to re-open at any time.
I am reopening because I think @dain mentioned that he would like to have a look.
This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua
@cdrappier is it possible to sync the catalogs periodically? as there might be more than 1 clusters, the catalog created on one cluster might also be used on another cluster.
@jerryleooo there is no multi-cluster support in Trino as such at this stage. Therefore this feature should also not take this idea into account.
At least not in the initial implementation .. later down the track this could make sense in Trino itself.. or in orchestration from some other system, including potentially Trino Gateway
Fyi @dprophet .. you could use this as a base to expand and improve and work with @dain and @cdrappier
This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua
At this point, I think this PR is abandoned. Trino now has an official SPI interface for catalog storage and a new implementation should be created using that interface.
