HDDS-5779. Create s3g bucket creation performance test

[Zeddling]

What changes were proposed in this pull request?

Add test class for freon. It generates random buckets for the s3 gateway.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-5779

How was this patch tested?

This patch was tested manually using a docker based secure cluster and using the ozone external terminal under hadoop-ozone/dist/target/ozone-X.X.X/bin/ .

Commands to reproduce the test.

Running docker-compose and get aws key

cd hadoop-ozone/dist/target/ozone-1.3.0-SNAPSHOT/compose/ozonesecure
docker-compose up -d --scale datanode=3
docker-compose exec scm bash

bash-4.2$ kinit -kt /etc/security/keytabs/testuser.keytab testuser/[email protected]
bash-4.2$ ozone s3 getsecret -e
export AWS_ACCESS_KEY_ID='testuser/[email protected]'
export AWS_SECRET_ACCESS_KEY='...'
bash-4.2$ exit

Copying the awsSecret key, exporting it to freon test environment and then running the Freon bucket generator test

docker exec -it ozonesecure_s3g_1 bash
export AWS_ACCESS_KEY=testuser/[email protected] AWS_SECRET_KEY=<key>
ozone freon s3bg -t 100 -n 500000

Running the test from the external shell

cd hadoop-ozone/dist/target/ozone-1.3.0-SNAPSHOT/bin/
./ozone freon s3bg -t 100 -n 500000

[xBis7]

@neils-dev

[kerneltime]

cc @DaveTeng0

[kerneltime]

Nit: Is the goal only to measure performance of a single bucket? Maybe add the ability to create multiple buckets ?

[adoroszlai]

Maybe add the ability to create multiple buckets ?

It will create multiple buckets (-n parameter).

[adoroszlai]

This patch was tested manually using a docker based secure cluster and using the ozone external terminal under hadoop-ozone/dist/target/ozone-X.X.X/bin/ .

Providing exact commands helps reviewers reproduce the test.

cd hadoop-ozone/dist/target/ozone-1.3.0-SNAPSHOT/compose/ozone
OZONE_REPLICATION_FACTOR=3 ./run.sh -d
sleep 60 # wait for cluster start
docker-compose exec -T scm bash -c \
"export AWS_ACCESS_KEY_ID=any AWS_SECRET_KEY=any
ozone freon s3bg --endpoint http://s3g:9878/ -n10"

Output:

...
2022-07-27 20:33:11,634 [main] INFO freon.BaseFreonGenerator: Executing test with prefix l2nphehnly
2022-07-27 20:33:11,678 [Thread-5] INFO freon.ProgressBar: Progress: 0.00 % (0 out of 10)
2022-07-27 20:33:12,662 [Thread-5] INFO freon.ProgressBar: Progress: 100.00 % (10 out of 10)
2022-07-27 20:33:13,334 [shutdown-hook-0] INFO metrics: type=TIMER, name=bucket-create, count=10, min=181.285259, max=214.404077, mean=202.73142240000004, stddev=10.869563898219672, median=209.286641, p75=210.5612, p95=214.404077, p98=214.404077, p99=214.404077, p999=214.404077, mean_rate=9.689749933590571, m1=0.0, m5=0.0, m15=0.0, rate_unit=events/second, duration_unit=milliseconds
2022-07-27 20:33:13,335 [shutdown-hook-0] INFO freon.BaseFreonGenerator: Total execution time (sec): 2
2022-07-27 20:33:13,335 [shutdown-hook-0] INFO freon.BaseFreonGenerator: Failures: 0
2022-07-27 20:33:13,335 [shutdown-hook-0] INFO freon.BaseFreonGenerator: Successful executions: 10

[xBis7]

@adoroszlai The exact commands to reproduce the test would be

Running docker-compose and getting the aws key

cd hadoop-ozone/dist/target/ozone-1.3.0-SNAPSHOT/compose/ozonesecure
docker-compose up -d --scale datanode=3
docker-compose exec scm bash

bash-4.2$ kinit -kt /etc/security/keytabs/testuser.keytab testuser/[email protected]
bash-4.2$ ozone s3 getsecret
awsAccessKey=testuser/[email protected]
awsSecret=<key>
bash-4.2$ exit

Copying the awsSecret key, exporting it to freon test environment and then running the Freon bucket generator test

docker exec -it ozonesecure_s3g_1 bash
export AWS_ACCESS_KEY=testuser/[email protected] AWS_SECRET_KEY=<key>
ozone freon s3bg -t 100 -n 500000

And for running the test from the external shell

cd hadoop-ozone/dist/target/ozone-1.3.0-SNAPSHOT/bin/
./ozone freon s3bg -t 100 -n 500000

Maybe @Zeddling can add these steps to the PR description

[DaveTeng0]

cc @DaveTeng0

Thank you Ritesh! checking.

[adoroszlai]

Thanks @xBis7 for the steps in ozonesecure env. Note that passing the -e parameter to ozone s3 getsecret makes it print credentials in a format suitable for shell:

$ ozone s3 getsecret -h
Usage: ozone s3 getsecret [-ehV] [--om-service-id=<omServiceID>] [-u=<username>]
Returns S3 secret for a user
  -e               Print out variables together with 'export' prefix, to use it
                     from 'eval $(ozone s3 getsecret)'
...

$ ozone s3 getsecret -e
export AWS_ACCESS_KEY_ID='testuser/[email protected]'
export AWS_SECRET_ACCESS_KEY='...'

This way we can void copy-paste:

kinit -kt /etc/security/keytabs/testuser.keytab testuser/[email protected]
eval $(ozone s3 getsecret -e)
ozone freon s3bg ...

[xBis7]

Thanks @adoroszlai for the clarification. @Zeddling should add that as well in the description.

[neils-dev]

please add documentation for how to specify credentials for S3?

Thanks @kerneltime - on this in addition to the update to the PR documenting the secure cluster setup that was used to reproduce the test, it can also be run in an unsecured ozone environment with the user .aws/credentials file set or shell environment with setting the following environment variables before running: $ export AWS_ACCESS_KEY_ID=ANYID $ export AWS_SECRET_ACCESS_KEY=ANYKEY

$ bin/ozone freon s3bg

[neils-dev]

Nit: Is the goal only to measure performance of a single bucket?

The test measures the performance of s3 bucket creation. Similar to the freon s3kg test, it tests the time to create the object from the number-of-tests (objects created) specified by the --number-of-tests (-n) parameter. In other words measuring from creating multiple buckets specified by -n or, 1000 buckets by default.

[kerneltime]

Maybe add the ability to create multiple buckets ?

It will create multiple buckets (-n parameter).

Thanks! I missed the runTests framework

[kerneltime]

Please add basic sanity tests under the robot framework

[Zeddling]

Please add basic sanity tests under the robot framework

I've added the test @kerneltime

[adoroszlai]

Thanks @Zeddling for adding the test. Can you please enable the build-branch workflow in your fork? I think you'll also need to push another commit (can be empty) to your branch to trigger the checks.

[adoroszlai]

Thanks @Zeddling for enabling the workflow. There are some failures:

S3g bucket creation performance                                       | FAIL |
Keyword 'Execute' failed after retrying for 2 minutes. The last error was: 1 != 0

https://github.com/Zeddling/ozone/runs/7681050738?check_suite_focus=true#step:5:349 https://github.com/Zeddling/ozone/runs/7681050967?check_suite_focus=true#step:5:175

S3g bucket creation performance                                       | FAIL |
255 != 0

https://github.com/Zeddling/ozone/runs/7681050967?check_suite_focus=true#step:5:585

Robot and Docker logs are available in artifacts you can download from https://github.com/Zeddling/ozone/actions/runs/2799557796#artifacts Check for first failure in each log.html.

[Zeddling]

@adoroszlai just added the robot test. The changes passed all CI workflows

[Zeddling]

Added an extra change. The tests weren't picking up the ${SECURITY_ENABLED} value from the environment and thus each test would have ${SECURITY_ENABLED}=false (set in commonlib.robot) on all environments. The change gets the variable from the environment updates ${SECURITY_ENABLED} before running the other keywords in Setup v4 headers. You would have to export SECURITY_ENABLED=<true or false> in the environment before setting the credentials

[adoroszlai]

Thanks @Zeddling for the patch, @kerneltime, @neils-dev for the review.