foundationdb icon indicating copy to clipboard operation
foundationdb copied to clipboard
verified publisher icon apple

Snapshot Backup Encryption

Open sfc-gh-nwijetunga opened this issue 3 years ago • 46 comments

This PR adds the following functionality

  • [ ] Break backup file blocks based on tenant boundaries
  • [ ] Encrypt and decrypt the file blocks based on the correct encryption domain
  • [ ] Connect with the EKP to fetch proper encryption keys during encryption/decryption
  • [ ] Utilize TenantEntryCache to validate tenant prefixes so backup encryption can be done when tenant mode is disabled

Correctness tested using the following:

  • [ ] 100k runs of all Backup workloads
  • [ ] New workload that bulk loads several KV pairs across 3 tenants and proceeds with backup and restore

TODOs:

  • [ ] Improve simulation testing (perhaps create a random number of tenants to load data onto)
  • [ ] Currently we cannot use the TenantEntryCache when running regular Backup workloads since the cache will attempt to refresh on every miss (expensive as many backup tests don't use tenants) causing timeouts

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

  • [ ] The PR has a description, explaining both the problem and the solution.
  • [ ] The description mentions which forms of testing were done and the testing seems reasonable.
  • [ ] Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

  • [ ] This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
  • [ ] There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

sfc-gh-nwijetunga avatar Sep 03 '22 04:09 sfc-gh-nwijetunga

Doxense CI Report for Windows 10

  • Commit ID: c25e52daf153dd050b0fde7c98ed3dc20ab02bcf
  • Result: :heavy_check_mark: SUCCEEDED
  • Build Logs (available for 30 days)

fdb-windows-ci avatar Sep 03 '22 04:09 fdb-windows-ci

Result of foundationdb-pr-macos on macOS BigSur 11.5.2

  • Commit ID: c25e52daf153dd050b0fde7c98ed3dc20ab02bcf
  • Duration 0:45:45
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Sep 03 '22 04:09 foundationdb-ci

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: c25e52daf153dd050b0fde7c98ed3dc20ab02bcf
  • Duration 1:09:02
  • Result: :x: FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Sep 03 '22 05:09 foundationdb-ci

Doxense CI Report for Windows 10

  • Commit ID: 3db499d2f55e8d44722c07f4bc59ead632c80ff9
  • Result: :heavy_check_mark: SUCCEEDED
  • Build Logs (available for 30 days)

fdb-windows-ci avatar Sep 03 '22 06:09 fdb-windows-ci

Result of foundationdb-pr-macos on macOS BigSur 11.5.2

  • Commit ID: 3db499d2f55e8d44722c07f4bc59ead632c80ff9
  • Duration 0:46:07
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Sep 03 '22 06:09 foundationdb-ci

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 3db499d2f55e8d44722c07f4bc59ead632c80ff9
  • Duration 1:13:06
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Sep 03 '22 07:09 foundationdb-ci

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: c25e52daf153dd050b0fde7c98ed3dc20ab02bcf
  • Duration 3:56:30
  • Result: :x: FAILED
  • Error: Error while executing command: if $(grep -q -- "--- FAIL:" ${CODEBUILD_SRC_DIR}/fdb-kubernetes-tests/logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" ${CODEBUILD_SRC_DIR}/fdb-kubernetes-tests/logs/*.log; exit 1; fi. Reason: exit status 1
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Sep 03 '22 08:09 foundationdb-ci

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 3db499d2f55e8d44722c07f4bc59ead632c80ff9
  • Duration 4:16:30
  • Result: :x: FAILED
  • Error: Build has timed out.
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Sep 03 '22 10:09 foundationdb-ci

Doxense CI Report for Windows 10

  • Commit ID: 574e306fc8555056956af0386ce18805b335a390
  • Result: :heavy_check_mark: SUCCEEDED
  • Build Logs (available for 30 days)

fdb-windows-ci avatar Sep 06 '22 18:09 fdb-windows-ci

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 574e306fc8555056956af0386ce18805b335a390
  • Duration 1:09:06
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Sep 06 '22 19:09 foundationdb-ci

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 574e306fc8555056956af0386ce18805b335a390
  • Duration 4:16:56
  • Result: :x: FAILED
  • Error: Build has timed out.
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Sep 06 '22 22:09 foundationdb-ci

Doxense CI Report for Windows 10

  • Commit ID: 27a22706f2c424a45d60fae070f23b3184041794
  • Result: :heavy_check_mark: SUCCEEDED
  • Build Logs (available for 30 days)

fdb-windows-ci avatar Sep 06 '22 23:09 fdb-windows-ci

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 27a22706f2c424a45d60fae070f23b3184041794
  • Duration 1:07:39
  • Result: :x: FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Sep 06 '22 23:09 foundationdb-ci

Doxense CI Report for Windows 10

  • Commit ID: 8f5f4459e8fe3b1ed214b0dd29ff65b7c6125830
  • Result: :heavy_check_mark: SUCCEEDED
  • Build Logs (available for 30 days)

fdb-windows-ci avatar Sep 07 '22 00:09 fdb-windows-ci

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 8f5f4459e8fe3b1ed214b0dd29ff65b7c6125830
  • Duration 1:13:45
  • Result: :x: FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Sep 07 '22 01:09 foundationdb-ci

Doxense CI Report for Windows 10

  • Commit ID: 92c2083a84bf71c9e928f4c96caeb5604b9ce699
  • Result: :heavy_check_mark: SUCCEEDED
  • Build Logs (available for 30 days)

fdb-windows-ci avatar Sep 07 '22 01:09 fdb-windows-ci

Result of foundationdb-pr-macos on macOS BigSur 11.5.2

  • Commit ID: 92c2083a84bf71c9e928f4c96caeb5604b9ce699
  • Duration 0:46:25
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Sep 07 '22 02:09 foundationdb-ci

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 92c2083a84bf71c9e928f4c96caeb5604b9ce699
  • Duration 1:08:39
  • Result: :x: FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Sep 07 '22 02:09 foundationdb-ci

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 27a22706f2c424a45d60fae070f23b3184041794
  • Duration 4:16:35
  • Result: :x: FAILED
  • Error: Build has timed out.
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Sep 07 '22 03:09 foundationdb-ci

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 8f5f4459e8fe3b1ed214b0dd29ff65b7c6125830
  • Duration 4:16:41
  • Result: :x: FAILED
  • Error: Error while executing command: if $(grep -q -- "--- FAIL:" ${CODEBUILD_SRC_DIR}/fdb-kubernetes-tests/logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" ${CODEBUILD_SRC_DIR}/fdb-kubernetes-tests/logs/*.log; exit 1; fi. Reason: exit status 1
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Sep 07 '22 04:09 foundationdb-ci

Result of foundationdb-pr-macos on macOS BigSur 11.5.2

  • Commit ID: 202b93578562217592299e17416d5d6262e65765
  • Duration 0:46:59
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Sep 07 '22 05:09 foundationdb-ci

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 92c2083a84bf71c9e928f4c96caeb5604b9ce699
  • Duration 3:44:13
  • Result: :x: FAILED
  • Error: Error while executing command: if $(grep -q -- "--- FAIL:" ${CODEBUILD_SRC_DIR}/fdb-kubernetes-tests/logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" ${CODEBUILD_SRC_DIR}/fdb-kubernetes-tests/logs/*.log; exit 1; fi. Reason: exit status 1
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Sep 07 '22 05:09 foundationdb-ci

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 202b93578562217592299e17416d5d6262e65765
  • Duration 1:16:32
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Sep 07 '22 05:09 foundationdb-ci

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 202b93578562217592299e17416d5d6262e65765
  • Duration 2:45:43
  • Result: :x: FAILED
  • Error: Error while executing command: if $(grep -q -- "--- FAIL:" ${CODEBUILD_SRC_DIR}/fdb-kubernetes-tests/logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" ${CODEBUILD_SRC_DIR}/fdb-kubernetes-tests/logs/*.log; exit 1; fi. Reason: exit status 1
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Sep 07 '22 07:09 foundationdb-ci

Doxense CI Report for Windows 10

  • Commit ID: 202b93578562217592299e17416d5d6262e65765
  • Result: :heavy_check_mark: SUCCEEDED
  • Build Logs (available for 30 days)

fdb-windows-ci avatar Sep 07 '22 07:09 fdb-windows-ci

Result of foundationdb-pr-macos on macOS BigSur 11.5.2

  • Commit ID: de0554cf91733a55f9a28482ed3c90a63ade4cb6
  • Duration 0:50:15
  • Result: :white_check_mark: SUCCEEDED
  • Error: N/A
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Sep 15 '22 08:09 foundationdb-ci

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: de0554cf91733a55f9a28482ed3c90a63ade4cb6
  • Duration 1:03:11
  • Result: :x: FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Sep 15 '22 08:09 foundationdb-ci

Doxense CI Report for Windows 10

  • Commit ID: a817794897f9ba746aa5e61a3a1c9f875a31ed8b
  • Result: :heavy_check_mark: SUCCEEDED
  • Build Logs (available for 30 days)

fdb-windows-ci avatar Sep 15 '22 08:09 fdb-windows-ci

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: a817794897f9ba746aa5e61a3a1c9f875a31ed8b
  • Duration 0:57:36
  • Result: :x: FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Sep 15 '22 08:09 foundationdb-ci

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: de0554cf91733a55f9a28482ed3c90a63ade4cb6
  • Duration 2:45:39
  • Result: :x: FAILED
  • Error: Error while executing command: if $(grep -q -- "--- FAIL:" ${CODEBUILD_SRC_DIR}/fdb-kubernetes-tests/logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" ${CODEBUILD_SRC_DIR}/fdb-kubernetes-tests/logs/*.log; exit 1; fi. Reason: exit status 1
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

foundationdb-ci avatar Sep 15 '22 10:09 foundationdb-ci