Test S3 Client in Simulation

[kakaiu]

Add two simulation tests: S3Client.toml and BulkDumpS3.toml. S3Client is a 'fast' test. BulkDumpS3 is a 'slow' test. Both require a seaweed instance running.

* fdbclient/include/fdbclient/S3BlobStore.h
* fdbclient/S3BlobStore.actor.cpp
 Add bypass_simulation querystring flag.
 When set, we'll make external connection to blobstore rather
 than connect in-simulation-network.

 Change the tryReadJSONFile so reads in 4k blocks -- simulation
 wants this.

 Add application/xml content-type when adding tags.

* fdbclient/S3Client.actor.cpp
 Change flags opening file and read in 4k blocks. Simulation
 insists.

* fdbserver/workloads/BulkDumping.actor.cpp
 Make transport and rootdir configurable.

* fdbserver/workloads/S3ClientWorkload.actor.cpp
 New workload to test s3client.

* tests/fast/BulkDumping.toml
 Set new config options.

* tests/fast/BulkDumpingS3.toml
 Test that uses s3 transport.
 Mostly same as BulkDumping.toml.

* tests/fast/S3Client.toml
 Test for s3client.

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

• Commit ID: 48cb7493b974d8a855f65dd004ab83114ae7717f
• Duration 0:15:46
• Result: :x: FAILED
• Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /opt/homebrew/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS Ventura 13.x

• Commit ID: 48cb7493b974d8a855f65dd004ab83114ae7717f
• Duration 0:20:58
• Result: :x: FAILED
• Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login -c ./build_pr_macos.sh. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-ide on Linux CentOS 7

• Commit ID: 48cb7493b974d8a855f65dd004ab83114ae7717f
• Duration 0:22:18
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

• Commit ID: 48cb7493b974d8a855f65dd004ab83114ae7717f
• Duration 0:54:27
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-arm on Linux CentOS 7

• Commit ID: 48cb7493b974d8a855f65dd004ab83114ae7717f
• Duration 0:55:00
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux CentOS 7

• Commit ID: 48cb7493b974d8a855f65dd004ab83114ae7717f
• Duration 1:02:39
• Result: :x: FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux CentOS 7

• Commit ID: 48cb7493b974d8a855f65dd004ab83114ae7717f
• Duration 1:06:15
• Result: :x: FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[saintstack]

I tried to run w/ this PR. Calling out to seaweedfs while in simulation fails:

bProgram terminated with signal SIGSEGV, Segmentation fault.
t#0  0x0000000004affdf8 in (anonymous namespace)::Connect_implActorState<(anonymous namespace)::Connect_implActor>::a_body1cont1 (this=0x7ff39a927cb0, loopDepth=0) at /root/src/foundationdb/fdbclient/S3BlobStore.actor.cpp:785
785             wait(conn->connectHandshake());
[Current thread is 1 (Thread 0x7ff3a0ed3ac0 (LWP 837573))]
Missing separate debuginfos, use: debuginfo-install glibc-2.17-326.el7_9.3.x86_64
(gdb) bt
[5:30](https://a1391190.slack.com/archives/D04MX1FUVNG/p1733275824888139)
The conn variable seems to be 0
[5:30](https://a1391190.slack.com/archives/D04MX1FUVNG/p1733275827433539)
(gdb) print conn
$1 = {ptr = 0x0}

For now I've extracted the refactor this PR started as #11813.

Progress on the simulation implementation is at https://github.com/saintstack/foundationdb/tree/simulation

Resolve this PR as for later?

[foundationdb-ci]

Result of foundationdb-pr-clang-ide on Linux CentOS 7

• Commit ID: 98a6d2a294eb172d37b4fa4c8bac4dd831aa647e
• Duration 0:23:27
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux CentOS 7

• Commit ID: 98a6d2a294eb172d37b4fa4c8bac4dd831aa647e
• Duration 0:36:20
• Result: :x: FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

• Commit ID: 98a6d2a294eb172d37b4fa4c8bac4dd831aa647e
• Duration 0:41:32
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux CentOS 7

• Commit ID: 98a6d2a294eb172d37b4fa4c8bac4dd831aa647e
• Duration 0:45:32
• Result: :x: FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-arm on Linux CentOS 7

• Commit ID: 98a6d2a294eb172d37b4fa4c8bac4dd831aa647e
• Duration 0:50:00
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

• Commit ID: 98a6d2a294eb172d37b4fa4c8bac4dd831aa647e
• Duration 0:59:57
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS Ventura 13.x

• Commit ID: 98a6d2a294eb172d37b4fa4c8bac4dd831aa647e
• Duration 1:08:32
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux CentOS 7

• Commit ID: c396d4f802c789b5348ad51bb1ccbc16a156ee91
• Duration 0:04:35
• Result: :x: FAILED
• Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-ide on Linux CentOS 7

• Commit ID: c396d4f802c789b5348ad51bb1ccbc16a156ee91
• Duration 0:04:38
• Result: :x: FAILED
• Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

• Commit ID: c396d4f802c789b5348ad51bb1ccbc16a156ee91
• Duration 0:04:48
• Result: :x: FAILED
• Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux CentOS 7

• Commit ID: c396d4f802c789b5348ad51bb1ccbc16a156ee91
• Duration 0:05:04
• Result: :x: FAILED
• Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

• Commit ID: c396d4f802c789b5348ad51bb1ccbc16a156ee91
• Duration 0:07:35
• Result: :x: FAILED
• Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-arm on Linux CentOS 7

• Commit ID: c396d4f802c789b5348ad51bb1ccbc16a156ee91
• Duration 0:48:36
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux CentOS 7

• Commit ID: 1b4e8f32bbd2dac0a19faee640fe0b00420e31e4
• Duration 0:04:35
• Result: :x: FAILED
• Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-ide on Linux CentOS 7

• Commit ID: 1b4e8f32bbd2dac0a19faee640fe0b00420e31e4
• Duration 0:04:32
• Result: :x: FAILED
• Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux CentOS 7

• Commit ID: 1b4e8f32bbd2dac0a19faee640fe0b00420e31e4
• Duration 0:04:38
• Result: :x: FAILED
• Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

• Commit ID: 1b4e8f32bbd2dac0a19faee640fe0b00420e31e4
• Duration 0:04:41
• Result: :x: FAILED
• Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-arm on Linux CentOS 7

• Commit ID: 1b4e8f32bbd2dac0a19faee640fe0b00420e31e4
• Duration 0:05:07
• Result: :x: FAILED
• Error: Error while executing command: cmake -S . -B build_output -D USE_CCACHE=ON -D USE_WERROR=ON -D RUN_JUNIT_TESTS=ON -D RUN_JAVA_INTEGRATION_TESTS=ON -D BUILD_DOCUMENTATION=ON -D BUILD_AWS_BACKUP=OFF -G Ninja. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

• Commit ID: 1b4e8f32bbd2dac0a19faee640fe0b00420e31e4
• Duration 0:08:33
• Result: :x: FAILED
• Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS Ventura 13.x

• Commit ID: 1b4e8f32bbd2dac0a19faee640fe0b00420e31e4
• Duration 0:08:44
• Result: :x: FAILED
• Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-arm on Linux CentOS 7

• Commit ID: 00a92e4b173249335fff97fe7b6e68262f6e2dea
• Duration 0:04:48
• Result: :x: FAILED
• Error: Error while executing command: cmake -S . -B build_output -D USE_CCACHE=ON -D USE_WERROR=ON -D RUN_JUNIT_TESTS=ON -D RUN_JAVA_INTEGRATION_TESTS=ON -D BUILD_DOCUMENTATION=ON -D BUILD_AWS_BACKUP=OFF -G Ninja. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux CentOS 7

• Commit ID: 00a92e4b173249335fff97fe7b6e68262f6e2dea
• Duration 0:04:45
• Result: :x: FAILED
• Error: Error while executing command: cmake -S . -B build_output -D USE_CCACHE=ON -D USE_WERROR=ON -D RUN_JUNIT_TESTS=ON -D RUN_JAVA_INTEGRATION_TESTS=ON -D BUILD_DOCUMENTATION=ON -D BUILD_AWS_BACKUP=ON -G Ninja. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)