[HUDI-7146] [RFC-77] RFC for secondary index

[bhat-vinay]

Abstract: In this RFC, we propose implementing Secondary Indexes (SI), a new capability in Hudi's metadata table (MDT) based indexing system. SI are indexes defined on user specified columns of the table. Similar to record level indexes, SI will improve query performance when the query predicate contains secondary keys. The number of files that a query needs to scan can be pruned down using secondary indexes.

Change Logs

New RFC

Impact

NA

Risk level (write none, low medium or high below)

NA

Documentation Update

NA

Contributor's checklist

• [ ] Read through contributor's guide
• [ ] Change Logs and Impact were stated clearly
• [ ] Adequate tests were added if applicable
• [ ] CI passed

[bhat-vinay]

Tagging @codope for an initial review. Some design elements in RFC are left-out as 'Implementation Detail' and will be filled as the implementation nears completion