starrocks-kubernetes-operator icon indicating copy to clipboard operation
starrocks-kubernetes-operator copied to clipboard

Enhance CREATE PIPE to process INSERT Select from starrocks tables as well

Open AsaiWiz opened this issue 1 year ago • 6 comments

CREATE PIPE is awesome help to load data from s3 compatible object storage. It basically processes INSERT SELECT from FILES. An enhancement to it to allow INSERT SELECT form TABLES/VIEWS would make starrocks very powerful and cater to 10x times use cases in implementing systems like Data Vault 2.0 data warehouses.

Please consider this at a priority as high as possible.

AsaiWiz avatar Dec 24 '23 04:12 AsaiWiz

I believe you can already do this. https://docs.starrocks.io/docs/loading/InsertInto/#insert-data-via-insert-into-select

alberttwong avatar Dec 24 '23 23:12 alberttwong

Hi Albert, I'm asking for a PIPE for it. CREATE PIPE ..INSERT ...SELECT ..from <starrocks table/view> is not supported currently.

CREATE PIPE ..INSERT ...SELECT ..from FILES( ) is supported now. Thanks, Pugazendhi Asaimuthu

On Sun, Dec 24, 2023 at 6:12 PM Albert T. Wong @.***> wrote:

I believe you can already do this. https://docs.starrocks.io/docs/loading/InsertInto/#insert-data-via-insert-into-select

— Reply to this email directly, view it on GitHub https://github.com/StarRocks/starrocks-kubernetes-operator/issues/382#issuecomment-1868609549, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALQ7IASN5HMBKTSMAD574GLYLCZG7AVCNFSM6AAAAABBBIPAJ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRYGYYDSNJUHE . You are receiving this because you authored the thread.Message ID: @.***>

AsaiWiz avatar Dec 26 '23 05:12 AsaiWiz

Since PIPE just got released in 3.2, it may take another release for this feature to be implemented.

alberttwong avatar Dec 26 '23 18:12 alberttwong

Thanks for being on it. Can you give a date so I can plan my development around it.

AsaiWiz avatar Dec 27 '23 05:12 AsaiWiz

Since StarRocks is a community project, dates come from the committers and vendors who write code for the project.

alberttwong avatar Dec 27 '23 17:12 alberttwong

‌Hello, I am not sure if the latest version of StarRocks has resolved your issue. If it hasn't been resolved, I suggest you can go to https://github.com/StarRocks/starrocks and raise this issue again. Because this project is the StarRocks Kubernetes Operator, it mainly deals with issues related to deployment and Kubernetes. @AsaiWiz

yandongxiao avatar May 30 '24 01:05 yandongxiao