Benchmarks for comparison to other "high-performance" S3 FS solutions

[bollig]

Tell us more about this new feature.

Please include benchmarks of mountpoint-s3 in the README so it is clear how "high-performance" is defined.

See e.g., https://github.com/kahing/goofys (benchmark script is provided here as well)

[jamesbornholt]

Thanks for the suggestion. GitHub Actions runs some benchmarks on every commit and posts the results here: https://awslabs.github.io/mountpoint-s3/dev/bench/. It's not a very pretty page right now, and I don't think it's linked or documented anywhere, so we'll see about improving that. The benchmark scripts for that live here.

[arunavamagi]

Hi,

We have been doing a POC with mountpoint-s3 for our usecases.

We wanted to do the benchmarking independently by running the fs_bench.sh in the mountpoint-s3/mountpoint-s3/scripts folder against our files in our own S3 bucket.

Our S3 Bucket (for benchmarking) has the following folder structure:

benchmark/B.mxf benchmark/C.ts

We have set the following environment variables:

export S3_BUCKET_NAME=bucket_name export S3_BUCKET_TEST_PREFIX=benchmark export S3_BUCKET_SMALL_BENCH_FILE=benchmark/C.ts export S3_BUCKET_BENCH_FILE=benchmark/B.mxf

While running the script we are getting a timeout error saying "Timeout while waiting for file system to be ready"

Not sure, but looks like findmnt -rncv -S mountpoint-s3 -T ${mount_dir} -o SOURCE,TARGET is creating some issue.

Could @jamesbornholt or @monthonk help us on this issue?

[monthonk]

Hi @arunavamagi,

Happy to help! The error might indicate that the mount process has failed during startup and the mount record cannot be found. Would you be able to provide some logs from mountpoint-s3 while running the script?

One thing I noticed is that you don't have to specify the path to bench files because you have already set it in the prefix. So your environment variables should be

export S3_BUCKET_NAME=bucket_name
export S3_BUCKET_TEST_PREFIX=benchmark
export S3_BUCKET_SMALL_BENCH_FILE=C.ts
export S3_BUCKET_BENCH_FILE=B.mxf

[arunavamagi]

Hi,

Thanks for the reply.

Have set the variables in the following way now:

export S3_BUCKET_NAME=bucket_name
export S3_BUCKET_TEST_PREFIX=benchmark
export S3_BUCKET_SMALL_BENCH_FILE=C.ts
export S3_BUCKET_BENCH_FILE=B.mxf

Is there any log folder, from where we can find the logs ?

@monthonk Interactive shell has only the following output:

[ec2-user@ip-12-0-15-141 scripts]$ sh fs_bench.sh 
Running rand_read_4t_direct
Timeout while waiting for file system to be ready

[monthonk]

The default log folder for Mountpoint is $HOME/.mountpoint-s3 and there should be a log file called bench.out in the same directory you run the script.

You can also try to mount your bucket manually by using this command cargo run $S3_BUCKET_NAME mount_dir --foreground --prefix=${S3_BUCKET_TEST_PREFIX} and the logs should be printed out to your shell.

[arunavamagi]

Hi @monthonk, we observed the following logs:

   = note: `Request` has a derived impl for the trait `Debug`, but this is intentionally ignored during dead code analysis

warning: `fuser` (lib) generated 6 warnings
    Finished release [optimized + debuginfo] target(s) in 0.30s
     Running `target/release/mount-s3 arunava-dual-cidr-test /tmp/fio-Hlij6c1zDqXn --foreground --prefix=benchmark --throughput-target-gbps=100 --thread-count=4`
thread 'main' panicked at 'prefix must be empty or end with `/`', /home/ec2-user/mountpoint-s3/mountpoint-s3/src/fs.rs:112:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

We renamed the S3_BUCKET_TEST_PREFIX to benchmark/

After that, we reran the script and the mount is succesful

[jamesbornholt]

We definitely shouldn't panic in this case, I think we're missing some input validation in main.rs. (And we should document how prefixes work better too).

[jamesbornholt]

We've published some benchmarking documentation here: https://github.com/awslabs/mountpoint-s3/blob/main/doc/BENCHMARKING.md. As mentioned above, the results of that benchmarking are published here after every commit: https://awslabs.github.io/mountpoint-s3/dev/bench/. Thanks for the suggestion!