Didip Kerabat comments

Results 55 comments of


                                            Didip Kerabat

Use standard library to correctly glob and stop at the correct folder structure when filtering cloud objects

@gianm @abhishekagarwal87 Any further thought?

Use standard library to correctly glob and stop at the correct folder structure when filtering cloud objects

I really think using `**.parquet` is a small price to pay. Especially considering that this is a standard Java syntax.

Use standard library to correctly glob and stop at the correct folder structure when filtering cloud objects

@gianm I definitely like calling this feature as a new property because blob store path is apparently different enough than regular file system path. I am down calling it `objectGlob`....

Use standard library to correctly glob and stop at the correct folder structure when filtering cloud objects

ok, @gianm what do you think now?

Use standard library to correctly glob and stop at the correct folder structure when filtering cloud objects

@abhishekagarwal87 @gianm thought?

Use standard library to correctly glob and stop at the correct folder structure when filtering cloud objects

@gianm @abhishekagarwal87 ok, I addressed the 3 comments. We'll see what the CI results look like.

Use standard library to correctly glob and stop at the correct folder structure when filtering cloud objects

@gianm @abhishekagarwal87 ok, the rest of the failures are due to unrelated Python errors. What do you think now?

Use standard library to correctly glob and stop at the correct folder structure when filtering cloud objects

An unrelated typescript check failed: ``` FAIL e2e-tests/multi-stage-query.spec.ts (302.447 s) ● Multi-stage query › runs a query that reads external data thrown: "Exceeded timeout of 300000 ms for a test....

Task logs UI is empty for the new Indexer daemon.

Unfortunately, there are quite a number of classes to whitelist into log4j2 file. ``` org.apache.druid.indexing.worker.WorkerTaskManager # important org.apache.druid.segment.loading.SegmentLocalCacheManager org.apache.druid.indexing.common.task.AbstractBatchIndexTask org.apache.druid.discovery.DruidLeaderClient org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask org.apache.druid.indexing.common.task.batch.parallel.TaskMonitor # important org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexPhaseRunner org.apache.druid.segment.realtime.appenderator.AppenderatorImpl org.apache.parquet.hadoop.InternalParquetRecordReader org.apache.druid.indexing.overlord.ThreadingTaskRunner ```

Task logs UI is empty for the new Indexer daemon.

So, `ThreadingTaskRunner.java` already knows the `taskDir`: ``` final File taskFile = new File(taskDir, "task.json"); ``` If `ThreadingTaskRunner.java` can `tee` the above class logs into: ``` final File taskFile = new...