tempo icon indicating copy to clipboard operation
tempo copied to clipboard

Published 20 hours ago • verified publisher icon databrickslabs

Improve AS OF Join Performance through Max Lookback parameter

Open rportilla-databricks opened this issue 2 years ago • 1 comments

Currently, AS OF joins require a window to be retained in memory per partition key (key for the Spark window). Improve may benefit from only looking back a certain number of records (or using a range if appropriate). This ticket should have:

  1. Tests showing perf improvement
  2. Prototype with a lookback parameter option (records lookback or range)
  3. Feedback from a user to see whether this type of approximation is acceptable.

rportilla-databricks avatar Aug 19 '22 17:08 rportilla-databricks

Would this be covered by the new tolerance parameter added by @nina-hu in #304?

tnixon avatar Apr 24 '23 22:04 tnixon