fdb-record-layer icon indicating copy to clipboard operation
fdb-record-layer copied to clipboard

Allowing JoinedRecordTypes to have two different KeyExpressions for each side

Open scottfines opened this issue 2 years ago • 17 comments

for use when constructing the join and also when constructing the index

scottfines avatar Oct 12 '22 21:10 scottfines

it appears that there might be a race condition when writing and/or deleting lucene data, causing one of the tests to fail here.

scottfines avatar Oct 13 '22 18:10 scottfines

Result of fdb-record-layer-pr-proto3 on Linux CentOS 7

  • Commit ID: ac27eff998e9274022492828de2ef0f0d7d123a5
  • Duration 0:19:39
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Oct 13 '22 18:10 foundationdb-ci

Result of fdb-record-layer-pr-proto2 on Linux CentOS 7

  • Commit ID: 5730ba8ceb32fe989c97e9df86cd30745a2d803f
  • Duration 0:15:34
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Oct 14 '22 14:10 foundationdb-ci

Result of fdb-record-layer-pr-proto3 on Linux CentOS 7

  • Commit ID: 5730ba8ceb32fe989c97e9df86cd30745a2d803f
  • Duration 0:16:06
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Oct 14 '22 14:10 foundationdb-ci

Result of fdb-record-layer-pr-proto2 on Linux CentOS 7

  • Commit ID: 2db6220747b464004ab9ec79b79b05b59fa4434a
  • Duration 0:17:16
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Oct 14 '22 14:10 foundationdb-ci

Result of fdb-record-layer-pr-proto3 on Linux CentOS 7

  • Commit ID: 2db6220747b464004ab9ec79b79b05b59fa4434a
  • Duration 0:17:42
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Oct 14 '22 14:10 foundationdb-ci

Result of fdb-record-layer-pr-proto2 on Linux CentOS 7

  • Commit ID: 52200bffce9b9f1b28b3f4426e1c2535a1e35c5b
  • Duration 0:15:44
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Oct 14 '22 14:10 foundationdb-ci

Result of fdb-record-layer-pr-proto3 on Linux CentOS 7

  • Commit ID: 52200bffce9b9f1b28b3f4426e1c2535a1e35c5b
  • Duration 0:16:07
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Oct 14 '22 14:10 foundationdb-ci

Result of fdb-record-layer-pr-proto2 on Linux CentOS 7

  • Commit ID: 08497043364fafb86771b7aac86e20f2c59db99d
  • Duration 0:15:40
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Oct 14 '22 14:10 foundationdb-ci

Result of fdb-record-layer-pr-proto3 on Linux CentOS 7

  • Commit ID: 08497043364fafb86771b7aac86e20f2c59db99d
  • Duration 0:16:12
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Oct 14 '22 14:10 foundationdb-ci

Result of fdb-record-layer-pr-proto2 on Linux CentOS 7

  • Commit ID: d055154f8dd500cfb30ddebd62c8c1a2256c1522
  • Duration 0:10:49
  • Result: :x: FAILED
  • Error: Error while executing command: ./gradlew --no-daemon --console=plain -b ./build.gradle build destructiveTest sonarqube -PcoreNotStrict -PreleaseBuild=false -PpublishBuild=false -PspotbugsEnableHtmlReport. Reason: exit status 1
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Oct 14 '22 14:10 foundationdb-ci

Result of fdb-record-layer-pr-proto3 on Linux CentOS 7

  • Commit ID: d055154f8dd500cfb30ddebd62c8c1a2256c1522
  • Duration 0:11:29
  • Result: :x: FAILED
  • Error: Error while executing command: ./gradlew --no-daemon --console=plain -b ./build.gradle build destructiveTest sonarqube -PcoreNotStrict -PreleaseBuild=false -PpublishBuild=false -PspotbugsEnableHtmlReport. Reason: exit status 1
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Oct 14 '22 14:10 foundationdb-ci

Result of fdb-record-layer-pr-proto3 on Linux CentOS 7

  • Commit ID: 5645e64da426e9305603f3cc1bc5ad43c0bc72f3
  • Duration 0:07:42
  • Result: :x: FAILED
  • Error: Error while executing command: ./gradlew --no-daemon --console=plain -b ./build.gradle build destructiveTest sonarqube -PcoreNotStrict -PreleaseBuild=false -PpublishBuild=false -PspotbugsEnableHtmlReport. Reason: exit status 1
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Oct 14 '22 15:10 foundationdb-ci

Result of fdb-record-layer-pr-proto2 on Linux CentOS 7

  • Commit ID: 5645e64da426e9305603f3cc1bc5ad43c0bc72f3
  • Duration 0:15:35
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Oct 14 '22 15:10 foundationdb-ci

Result of fdb-record-layer-pr-proto2 on Linux CentOS 7

  • Commit ID: 50a6ab32113e28e7d90cc8950df9ed41a2fabdb9
  • Duration 0:15:37
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Oct 14 '22 15:10 foundationdb-ci

Alec gives a more compartmentalized way of thinking about it.

If the desired join is

tl.cl = func(tr.cr)

we sometimes need the equivalent

func ⁻¹(tl.cl) = tr.cr

The two new key expressions are the function and its inverse.

Alternatively, if there were some way for a single key expression to give its inverse or a registry of them, there could just be one in the persisted form.

MMcM avatar Oct 14 '22 15:10 MMcM

One is if there's an index on func(rec1.some_field), then the condition can be satisfied by scanning that index.

You would expect there to be an index on rec1.some_field, so this function index is no worse than that, unless the simpler one is also useful for other queries.

MMcM avatar Oct 14 '22 15:10 MMcM

Yeah, that's what I was thinking, regarding using function key expressions. Given that the planner currently doesn't support matching a function key expression, the way I was thinking of doing it in version 0 is to always assume the query needs to be tr.cr = func^-1(tl.cl), which means that an explicit index on func(tr.cr) wouldn't be usable, but further iterations on the join planner and query planner would allow us to relax that constraint.

alecgrieser avatar Oct 17 '22 08:10 alecgrieser

Given that the planner currently doesn't support matching a function key expression ...

Note that the planner does support matching a QueryableKeyExpression, provided that is what is stored in the index. So the case where the function is applied to match the primary key, say, and that primary key matches a stored version of the same function application for the inverse direction, should already work and we should try not to break it. The case under discussion here is where the available index does not include any function, so that its inverse needs to be applied to the record in hand.

MMcM avatar Oct 17 '22 14:10 MMcM

Ah, right, I guess I overlooked QueryableKeyExpressions. I think it should be possible to add support for those, too.

I've filed #1872 about this, and I suspect whichever PR we take, we'll want to associate that issue. I've got a draft PR, #1873, which is designed to resolve it. It got somewhat more complicated with IN comparisons due to how the InExtractor works, so the PR ballooned a bit more than I expected. I have some local test failures (hence it being in draft), but I am working on resolving those

alecgrieser avatar Oct 17 '22 15:10 alecgrieser

Should this be closed now that #1872 is closed?

alecgrieser avatar Oct 31 '22 11:10 alecgrieser

Closing because it was superseded by InvertibleFunctionKeyExpressions

scottfines avatar Nov 01 '22 18:11 scottfines