pulsar icon indicating copy to clipboard operation
pulsar copied to clipboard

[feature][CLI] Add commands to run 'scanOffloadedLedgers'.

Open Technoboy- opened this issue 3 years ago • 8 comments

Master Issue: #16070

Motivation

Add pulsar-admin command to execute scanOffloadedLedgers, introduced with #14930

Documentation

  • [x] doc-not-needed

Technoboy- avatar Jul 06 '22 05:07 Technoboy-

@Technoboy- Need update docs?

shibd avatar Jul 06 '22 10:07 shibd

@Technoboy- Need update docs?

No need.

Technoboy- avatar Jul 06 '22 12:07 Technoboy-

I had started to work on a prototype https://github.com/apache/pulsar/compare/master...eolivelli:pulsar:impl/pulsar-admin-scanoffloaded

eolivelli avatar Jul 14 '22 13:07 eolivelli

I suggest to not change the "StreamingOutput" part, maybe there is a better way to do it, but I did not find. There should be some way in the client to do Streaming Reads

eolivelli avatar Jul 14 '22 14:07 eolivelli

I prefer to use the original streaming way to return the scan results. Because it will be huge and will cost very long time to scan.

ok.

Technoboy- avatar Jul 15 '22 01:07 Technoboy-

@eolivelli @hangc0276 I think the streaming style you are talking about is similar to https://trino.io/docs/current/develop/client-protocol.html ? The client can get part of the data for each request and perform the next request based on the last response nextUrl. I'm not sure how https://github.com/apache/pulsar/pull/14930 can resolve the issue of the server-side run for a long time, but the client-side will not get a timeout.

codelipenghui avatar Jul 18 '22 01:07 codelipenghui

in #14930 we send the data in a streaming fashion, without accumulating the results on the server.

now the problem is to read it in a streaming way on the client

eolivelli avatar Jul 29 '22 11:07 eolivelli

The pr had no activity for 30 days, mark with Stale label.

github-actions[bot] avatar Aug 29 '22 02:08 github-actions[bot]