bhat-vinay

Rebase and resolve conflicts. Fix a bug related to MOR tables with secondary index.

Moved away from using `HoodieUnMergedLogRecordScanner`. Added new buffer in `HoodieMergedLogRecordScanner` (based on SpillableDiskMap) to handle non-unique keys (secondary keys)

Thanks for the review @bvaradar. @codope pointed that the failing tests could be fixed by https://github.com/apache/hudi/pull/10381. Rebased past it to see if I can get a clean run.

> Hi @bhat-vinay! Is this design of secondary index through MDT is the only one to be implemented or there plans to make some other Index Types? As I remember...

This is likely not an issue, but a gap in understanding the feature. The issue is that `spark.read.format("hudi").load(PATH).createOrReplaceTempView(TABLE_NAME)` creates a temporary view (similar to the one that is created using...

Thanks for trying it out. This seems like a different issue (rather than the spark analysis error that you were seeing earlier). The error reported now is that the base...

``` 24/03/27 14:14:10 ERROR HoodieBackedTableMetadataWriter: Bootstrap on func_index_ partition failed for file:/Users/soumilshah/Desktop/hudidemo/.hoodie/metadata org.apache.spark.sql.AnalysisException: [PATH_NOT_FOUND] Path does not exist: file:/f113c72e-9bf6-4ab1-84b4-b2e9467142a8-0_11-68-0_20240327141353398.parquet. ``` The above lines indicate that the basepath for the file...

Thanks for trying all the suggestions. I am unable to reproduce this in my environment. I will spend some time next week to setup the exact environment/versions you are using....

Tagging @codope for an initial review. Some design elements in RFC are left-out as 'Implementation Detail' and will be filled as the implementation nears completion

cc: @codope