Add rank of hit from first-phase as a rank-feature for second-phase

[jobergum]

TLDR; Add new rank-feature firstPhaseRank that returns the rank of the hit to be used in second-phase.

We already support the firstPhase rank-feature which returns the score of the hit from the first-phase ranking. We have several times been asked to add a firstPhaseRank feature that returns the rank of the hit after sorting the hits after the first-phase. This feature can only be used in second-phase, similar to firstPhase.

[tkaessmann]

That rank-feature would great for us! We read in multiple papers that the rank of the first-phase is a important rank-feature for re-ranking-models. In our cases we already proofed an increase of the quality with that feature of our personalization model and for now it wasn't possible to implement this feature on our own. Really looking forward to have this!

[geirst]

The firstPhaseRank feature is available in Vespa 8.348.21.

Documentation: https://docs.vespa.ai/en/reference/rank-features.html#firstPhaseRank System test: https://github.com/vespa-engine/system-test/pull/4014 PRs: https://github.com/vespa-engine/vespa/pull/31297, https://github.com/vespa-engine/vespa/pull/31312

[geirst]

@tkaessmann FYI

[tkaessmann]

Thanks a lot! Great work!

[tomglk]

Hi, encountered some surprises while working with the new feature today:

rank-profiles:

rank-profile test_working {
    first-phase { expression: 42 }
    second-phase { expression: firstPhase }
    
    match-features { firstPhase firstPhaseRank }
    summary-features { firstPhase firstPhaseRank }
}

rank-profile test_not_working {
    first-phase { expression: 42 }
    # no second-phase
    
    match-features { firstPhase firstPhaseRank }
    summary-features { firstPhase firstPhaseRank }
}

Surprises

The timeout matters
http://localhost:8080/search/?yql=select%20*%20from%20data%20where%20true&ranking.profile=test_working&timeout=10

{
  ... 
  "matchfeatures": {
    "firstPhase": 42.0,
    "firstPhaseRank": 2.0
  },
  "summaryfeatures": {
    "firstPhase": 42.0,
    "firstPhaseRank": 2.0,
    "vespa.summaryFeatures.cached": 0.0
  }
}

( Slight confusion: summaryfeatures for firstPhaseRank should be max double as doc states ? It also works for a rank-profile without match-features. )

http://localhost:8080/search/?yql=select%20*%20from%20data%20where%20true&ranking.profile=test_working&timeout=11 (timeout 11, not 10)

{
  ...
  "matchfeatures": {
    "firstPhase": 42.0,
    "firstPhaseRank": 2.0
  },
  "summaryfeatures": {
    "firstPhase": 42.0,
    "firstPhaseRank": 1.7976931348623157E308,
    "vespa.summaryFeatures.cached": 0.0
  }
}

Expectation: the timeout should not make a difference

Differences using trace.level=3:

• additional log for 11: "Query timeout (10994 ms) > max query cache timeout (10000 ms). Disabling query cache."
• difference in log:
  • timeout=10: "Not resending query during document summary fetching"
  • timeout=11: "ProtoBuf: Resending query during document summary fetching"

The 2 firstPhase features behave differently
http://localhost:8080/search/?yql=select%20*%20from%20data%20where%20true&ranking.profile=test_non_working

{
  ...
  "matchfeatures": {
    "firstPhase": 42.0,
    "firstPhaseRank": 1.7976931348623157E308
  },
  "summaryfeatures": {
    "firstPhase": 42.0,
    "firstPhaseRank": 1.7976931348623157E308,
    "vespa.summaryFeatures.cached": 0.0
  }
}

Expectations:

• both have values without a second-phase, since they only rely on the first-phase info
• both have / don't have values in the same way

Technically, you could argue that they are not selected for a second-phase if there is no second-phase. But as a user I was surprised.

Questions

I know that I should not go to the summaryfeatures for firstPhaseRank. However, the inconsistency between so similar features was unexpected.
It was also surprising that I could in fact use the summaryfeatures for firstPhaseRank (if my rank-profile defines second-phase) even though the docs say no.

Is the difference between the features intended? If yes, do you mind explaining why?
Is it expected that summaryfeatures is giving a value for firstPhaseRank? If yes, why do the docs call it an unsupported case?
Is the difference for timeout=10 / timeout=11 expected?

[Optional / Out of curiosity]
Why is the default for firstPhaseRank max double and not 0, like for all other Rank scores? The count starts at 1, so 0 would also not occur normally. Only to prevent developers from assuming 0 means first one?

[jobergum]

As clearly documented, this does not work in summary-features for reasons.

[jobergum]

It does not work with summary-features because the order at that protocol phase is not well-defined. If there are concrete questions regarding the value for match-features ( I didn't read the entire post since I was you were randomly testing stuff using summary-features),feel free to ask.

on

[Optional / Out of curiosity] Why is the default for firstPhaseRank max double and not 0, like for all other Rank scores? The count starts at 1, so 0 would also not occur normally. Only to prevent developers from assuming 0 means first one?

Yes .