[HUDI-7146] Implement secondary index write path

[codope]

Change Logs

Main changes in this PR are for secondary index write path:

• New index type added in MetadataPartitionType
• Initialization of the new index in HoodieBackedTableMetadataWriter
• Util methods to support index creation and update in HoodieTableMetadataUtil
• Changes to HoodieBackedTableMetadataWriter to handle update and deletes for secondary index.
• New APIs in HoodieTableMetadata and their implementation in BaseTableMetadata and HoodieBackedTableMetadata to load secondary index.
• Changes in HoodieMergedLogRecordScanner to merge secondary index payloads.

Another PR will follow this PR, wherein we will have the secondary index reader-side integration and more e2e tests.

Impact

Support secondary index creation and update.

Risk level (write none, low medium or high below)

medium

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".

• The config description must be updated if new configs are added or the default value of the configs are changed
• Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the ticket number here and follow the instruction to make changes to the website.

Contributor's checklist

• [ ] Read through contributor's guide
• [ ] Change Logs and Impact were stated clearly
• [ ] Adequate tests were added if applicable
• [ ] CI passed

[hudi-bot]

CI report:

• 470bc5f44e7a6658a8717ef1b77e92afcdd90087 UNKNOWN
• 43f73661f79eb87ac52d29fa153b996a15f29b99 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build