foundationdb icon indicating copy to clipboard operation
foundationdb copied to clipboard

Add getMappedRangeV2 API with byte array in request/reply

Open flowguru opened this issue 2 years ago • 67 comments

Two more features are added in V2:

  • optionally return index
  • indicate whether a primary record is served by this SS taking prefetch index query

A new API is needed because the struct of req/rep have changed and is not backward compatible.

To achieve this, Java takes parameters to indicate how to return index and whether to do a server to server call in the case of a local miss, then encode them in a byte array. C API takes a byte array and knows how to parse it. The reason to have byte array is for extendability -- new fields can be encoded in the byte array without a signature/protocol change.

It is easy to add new fields to request or reply of getMappedRangeV2 API: instead of creating another API, we only need to bump the API-defined protocol version, as long as the client and server both know how to parse the byte array in req/rep.

QUICK_GET_KEY_VALUES_FALLBACK is removed, old API will always try to do server to server call to fetch KV from other SS. While new API can specify this parameter.

g++: 20230127-165936-haofu-8d06b832dc7f0c58, 20230128-001904-haofu-625beece285c9dbb, 20230128-013639-haofu-e84f2a61583422a4

clang: 20230127-170644-haofu-18da7da534f23610, 20230128-000811-haofu-327dac4b735093f8 , 20230128-014224-haofu-f275ec6abc4e5617

20230201-231332-haofu-3b845ffbe31d8522

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

  • [ ] The PR has a description, explaining both the problem and the solution.
  • [ ] The description mentions which forms of testing were done and the testing seems reasonable.
  • [ ] Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

  • [ ] This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
  • [ ] There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

flowguru avatar Jan 27 '23 03:01 flowguru

Result of foundationdb-pr-clang-ide on Linux CentOS 7

  • Commit ID: e5f646e630d93cf1524e1b326c0bbe5229a41c91
  • Duration 0:17:34
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Jan 27 '23 03:01 foundationdb-ci

Result of foundationdb-pr-macos-m1 on macOS Monterey 12.x

  • Commit ID: e5f646e630d93cf1524e1b326c0bbe5229a41c91
  • Duration 0:32:23
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Jan 27 '23 03:01 foundationdb-ci

Result of foundationdb-pr-clang on Linux CentOS 7

  • Commit ID: e5f646e630d93cf1524e1b326c0bbe5229a41c91
  • Duration 0:44:26
  • Result: :x: FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Jan 27 '23 03:01 foundationdb-ci

Result of foundationdb-pr-macos on macOS Monterey 12.x

  • Commit ID: e5f646e630d93cf1524e1b326c0bbe5229a41c91
  • Duration 0:44:42
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Jan 27 '23 03:01 foundationdb-ci

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: e5f646e630d93cf1524e1b326c0bbe5229a41c91
  • Duration 0:56:55
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Jan 27 '23 04:01 foundationdb-ci

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: e5f646e630d93cf1524e1b326c0bbe5229a41c91
  • Duration 1:14:23
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci avatar Jan 27 '23 04:01 foundationdb-ci

Result of foundationdb-pr-clang-ide on Linux CentOS 7

  • Commit ID: e5f646e630d93cf1524e1b326c0bbe5229a41c91
  • Duration 0:21:25
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Jan 27 '23 05:01 foundationdb-ci

Result of foundationdb-pr-clang on Linux CentOS 7

  • Commit ID: e5f646e630d93cf1524e1b326c0bbe5229a41c91
  • Duration 0:38:17
  • Result: :x: FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Jan 27 '23 05:01 foundationdb-ci

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: e5f646e630d93cf1524e1b326c0bbe5229a41c91
  • Duration 0:54:50
  • Result: :x: FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Jan 27 '23 05:01 foundationdb-ci

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: e5f646e630d93cf1524e1b326c0bbe5229a41c91
  • Duration 1:14:13
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci avatar Jan 27 '23 06:01 foundationdb-ci

Result of foundationdb-pr-clang-ide on Linux CentOS 7

  • Commit ID: 5284f3cd47b68c5ad05b1cc56070fa06c4236757
  • Duration 0:17:07
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Jan 27 '23 07:01 foundationdb-ci

Result of foundationdb-pr-macos-m1 on macOS Monterey 12.x

  • Commit ID: 5284f3cd47b68c5ad05b1cc56070fa06c4236757
  • Duration 0:32:52
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Jan 27 '23 07:01 foundationdb-ci

Result of foundationdb-pr-macos on macOS Monterey 12.x

  • Commit ID: 5284f3cd47b68c5ad05b1cc56070fa06c4236757
  • Duration 0:44:35
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Jan 27 '23 07:01 foundationdb-ci

Result of foundationdb-pr-clang on Linux CentOS 7

  • Commit ID: 5284f3cd47b68c5ad05b1cc56070fa06c4236757
  • Duration 1:04:16
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Jan 27 '23 07:01 foundationdb-ci

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 5284f3cd47b68c5ad05b1cc56070fa06c4236757
  • Duration 1:10:52
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Jan 27 '23 07:01 foundationdb-ci

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 5284f3cd47b68c5ad05b1cc56070fa06c4236757
  • Duration 1:34:54
  • Result: :x: FAILED
  • Error: Error while executing command: if $(grep -q -- "--- FAIL:" ${CODEBUILD_SRC_DIR}/fdb-kubernetes-tests/logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" ${CODEBUILD_SRC_DIR}/fdb-kubernetes-tests/logs/*.log; exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci avatar Jan 27 '23 08:01 foundationdb-ci

Doxense CI Report for Windows 10

  • Commit ID: 5284f3cd47b68c5ad05b1cc56070fa06c4236757
  • Result: :x: FAILED
  • Build Logs (available for 30 days)

fdb-windows-ci avatar Jan 27 '23 14:01 fdb-windows-ci

Result of foundationdb-pr-clang-ide on Linux CentOS 7

  • Commit ID: 3debeec026d93d77034884208b030e18f5371426
  • Duration 0:23:23
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Jan 27 '23 17:01 foundationdb-ci

Doxense CI Report for Windows 10

  • Commit ID: 3debeec026d93d77034884208b030e18f5371426
  • Result: :x: FAILED
  • Build Logs (available for 30 days)

fdb-windows-ci avatar Jan 27 '23 17:01 fdb-windows-ci

Result of foundationdb-pr-clang on Linux CentOS 7

  • Commit ID: 3debeec026d93d77034884208b030e18f5371426
  • Duration 1:06:36
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Jan 27 '23 18:01 foundationdb-ci

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 3debeec026d93d77034884208b030e18f5371426
  • Duration 1:11:20
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Jan 27 '23 18:01 foundationdb-ci

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 3debeec026d93d77034884208b030e18f5371426
  • Duration 1:17:35
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci avatar Jan 27 '23 18:01 foundationdb-ci

Doxense CI Report for Windows 10

  • Commit ID: 43102450e9bc8dac4978819c7a6a4a1c0564408b
  • Result: :x: FAILED
  • Build Logs (available for 30 days)

fdb-windows-ci avatar Jan 28 '23 00:01 fdb-windows-ci

Result of foundationdb-pr-clang-ide on Linux CentOS 7

  • Commit ID: 43102450e9bc8dac4978819c7a6a4a1c0564408b
  • Duration 0:17:33
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Jan 28 '23 00:01 foundationdb-ci

Result of foundationdb-pr-macos-m1 on macOS Monterey 12.x

  • Commit ID: 43102450e9bc8dac4978819c7a6a4a1c0564408b
  • Duration 0:32:54
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Jan 28 '23 00:01 foundationdb-ci

Result of foundationdb-pr-clang on Linux CentOS 7

  • Commit ID: 43102450e9bc8dac4978819c7a6a4a1c0564408b
  • Duration 0:37:06
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Jan 28 '23 00:01 foundationdb-ci

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 43102450e9bc8dac4978819c7a6a4a1c0564408b
  • Duration 0:38:54
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Jan 28 '23 01:01 foundationdb-ci

Result of foundationdb-pr-macos on macOS Monterey 12.x

  • Commit ID: 43102450e9bc8dac4978819c7a6a4a1c0564408b
  • Duration 0:44:00
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci avatar Jan 28 '23 01:01 foundationdb-ci

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 43102450e9bc8dac4978819c7a6a4a1c0564408b
  • Duration 1:04:52
  • Result: :x: FAILED
  • Error: Error while executing command: if $(grep -q -- "--- FAIL:" ${CODEBUILD_SRC_DIR}/fdb-kubernetes-tests/logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" ${CODEBUILD_SRC_DIR}/fdb-kubernetes-tests/logs/*.log; exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci avatar Jan 28 '23 01:01 foundationdb-ci

Doxense CI Report for Windows 10

  • Commit ID: f5ccd2b6724ae34dfd1c3048d42b5c93010eaeb7
  • Result: :x: FAILED
  • Build Logs (available for 30 days)

fdb-windows-ci avatar Jan 28 '23 01:01 fdb-windows-ci