trino Pinot insert

Implement insert into pinot. TODO: add disk based sort, extract relevant commits, currently it is based off of #7160 (drop table)

Mar 04 '21 18:03 elonazoulay

:wave: @elonazoulay - this PR has become inactive. If you're still interested in working on it, please let us know, and we can try to get reviewers to help with that.

We're working on closing out old and inactive PRs, so if you're too busy or this has too many merge conflicts to be worth picking back up, we'll be making another pass to close it out in a few weeks.

Oct 21 '22 21:10 bitsondatadev

Gonna close this one out due to inactivity, please reopen if you would like to continue doing work here.

Nov 18 '22 22:11 bitsondatadev

Sure, I am still actively working on this one. I will push changes to it later today.

Nov 21 '22 16:11 elonazoulay

@bitsondatadev can we review please?

Nov 27 '22 05:11 jatin5251

Sure! Go right ahead. We always welcome reviews.

Nov 27 '22 10:11 bitsondatadev

@elonazoulay how's this looking? Is it ready for review?

Jan 26 '23 17:01 bitsondatadev

Hey, I just resolved a dependency issue w new version. Waiting for the tests to complete and it should be good to go! Will likely break into multiple commits, whatever the reviewer thinks is the easiest way to review:

Break out support for timestamp and boolean array types
Minor fix to array types (handle null array element)
Fix time boundary handling + test

Just want to make sure the tests are all good first:)

Jan 26 '23 18:01 elonazoulay

Tests passed! Ready for review. Any initial comments about large refactoring are welcome, will be happy to do anything you suggest. One more note: Added failsafe retry policy to the testInsertIntoHybridTableWithTimestamp test since there appears to be latency between uploading segments and servers downloading them so the data is available. This only happened on ci, maybe due to constrained resources. Locally all tests pass repeatedly.

Jan 27 '23 21:01 elonazoulay

@bitsondatadev I have reviewed the PR and LGTM. We are running trino in production from this branch for more than 6 months now. can we request review from trino guys?

Apr 06 '23 19:04 jatin5251

Thanks @jatin5251! Let me see if @martint can take a look.

Apr 06 '23 20:04 bitsondatadev

@bitsondatadev @martint can you guys have a look?

Jun 14 '23 10:06 jatin5251

Hey @jatin5251 gonna tag @trinodb/devrel to include Manfred and Cole to follow up.

Jun 15 '23 13:06 bitsondatadev

@bitsondatadev any luck?

Sep 12 '23 19:09 jatin5251

@elonazoulay can you rebase and get this reviewed?

Sep 12 '23 20:09 mosabua

Are you looking at including the upgrade to Apache Pinto 1.0.0 as part of this PR? https://docs.pinot.apache.org/basics/releases/1.0.0

Sep 22 '23 19:09 mosabua

Update: the failing tests in ci all passed locally with the same command as failing step, i.e. ./mvnw test -pl plugin/trino-pinot. It appears the github runner does not have enough resources to run the tests.

Sep 23 '23 07:09 elonazoulay

Are you looking at including the upgrade to Apache Pinto 1.0.0 as part of this PR? https://docs.pinot.apache.org/basics/releases/1.0.0

That would be done in a separate pr. Will push that and let you know.

Sep 23 '23 07:09 elonazoulay

Update: the tests now fail after ~47m: increasing heap to 3g ,i.e. using MAVEN_INSTALL_OPTIONS. Before they were failing after about 22-23mins.

Trying to increase the flush interval which reduces resources on the pinot server test containers.

Sep 23 '23 18:09 elonazoulay

Are you looking at including the upgrade to Apache Pinto 1.0.0 as part of this PR? https://docs.pinot.apache.org/basics/releases/1.0.0

I created #19136 for upgrading to pinot 1.0.0 - cc @mosabua

Sep 24 '23 06:09 elonazoulay

FYI - the reason checks were failing was because too many parallel pinot smoke tests were running. I removed the legacy client pinot tests test verify this. Will add them back, testing how to do this without overloading the gh runner.

Sep 24 '23 07:09 elonazoulay

@mosabua @bitsondatadev can you please get it reviewed , its pending for long?

Oct 01 '23 19:10 jatin5251

@mosabua @bitsondatadev can you please get it reviewed , its pending for long?

@elonazoulay can work directly with the maintainers who know the Pinot connector.

Oct 02 '23 19:10 mosabua

:wave: @elonazoulay - this PR has become inactive. If you're still interested in working on it, please let us know.

We're working on closing out old and inactive PRs, so if you're too busy or this has too many merge conflicts to be worth picking back up, we'll be making another pass to close it out in a few weeks.

Jan 11 '24 00:01 mosabua

Hi, this is still active. I'm waiting for this issue to be resolved, likely in the next minor release of Pinot.

Jan 15 '24 17:01 elonazoulay

This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua

Sep 05 '24 17:09 github-actions[bot]

Looks like you can proceed on this @elonazoulay since the linked issue seems resolved and a new release is out.

Sep 05 '24 17:09 mosabua

This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua

Sep 30 '24 17:09 github-actions[bot]

Closing this pull request, as it has been stale for six weeks. Feel free to re-open at any time.

Oct 22 '24 17:10 github-actions[bot]

I think @elonazoulay will pick this up again eventually or someone else can use this as a base for implementing independently. Next step would be a rebase.

I added the stale-ignore label to ensure we dont loose sight of this improvement.

Oct 22 '24 19:10 mosabua

also fyi @robertzych

Jan 08 '25 22:01 mosabua

trino trino copied to clipboard

Pinot insert

trino
trino copied to clipboard