hyperspace icon indicating copy to clipboard operation
hyperspace copied to clipboard

Add plan signature validation for Hybrid Scan

Open sezruby opened this issue 5 years ago • 2 comments

Describe the issue

Currently, Hyperspace only supports creating indexes on a logical relation node. In order to support arbitrary logical plans, plan signature comparison is required for getting candidate indexes. This work is done by #76. In case of hybrid scan, we cannot utilize the "index" signature value in IndexLogEntry which is a composite of FileBasedSignature + PlanSignature, because

  • Index signature calculation overhead during the query time
  • Delete support - need to identify the deleted files and exclude them in the index signature calculation

#153 handles this issue by checking the metadata of source files directly without using the index signature. As a follow-up, we need to add plan signature validation for hybrid scan accordingly. This could be done by following:

  1. Store the plan signature of the source plan separately in IndexLogEntry
  2. Compare plan signatures in getCandidateIndex

To Reproduce

Expected behavior

Environment

sezruby avatar Sep 14 '20 01:09 sezruby

@sezruby Can you check if this issue can be closed?

imback82 avatar Jan 29 '21 18:01 imback82

Since we only allow 1 relation and the current plan signature of non-hybrid scan is just using "Relation" node, I think we don't need to consider the query plan(Relation) for Hybrid Scan - until we support #95.

sezruby avatar Feb 02 '21 07:02 sezruby