Extensibility Hooks for a new Hyperspace Index

[rapoth]

Describe the problem Adding a new index type into Hyperspace is currently undocumented. A developer interested in contributing will have to reverse engineer and figure out how to do this. In addition to the covering index that Hyperspace supports today, there are many other indexes that are highly beneficial for users.

Describe your proposed solution To solve this, we would have to refactor the codebase to allow easy addition of new indexes and provide sufficient support for leveraging these indexes at query time (e.g., through query hints, optimizer heuristics etc.).

In addition, as part of this effort, we also need to write a document (not necessarily a design document) that uses an example index to fully describe what all needs to be considered and what all needs to be done to add a new index support into Hyperspace. It is best if the guide targets an incremental development scenario i.e., the developer does not need to support every single feature of Hyperspace (e.g., index maintenance etc.) in one-shot.

The document needs to cover at least the following:

• What to keep in mind when introducing a new index into Hyperspace?
• Thinking about Meta-data for your Index
• Hooking into Hyperspace Index Abstraction (i.e., what classes to modify)
• Optimizer changes
• Debugging
• Hooking into Hyperspace Explain
• Testing
• Avoiding Regressions

Describe alternatives you've considered Let developers explore the code base. However, this is cumbersome and requires a lot of hand-holding.

Additional context N/A

[rapoth]

@apoorvedave1 @pirz @imback82 If there is anything else that needs to be covered, let's discuss it here.

[imback82]

Overall it seems fine, but I would experiment adding one ourselves first before publishing this doc since the internal hooks are still evolving.

Btw, I think we need to add the spec for index metadata first?

[sezruby]

I'll update the document when I go through #161. Thanks!