opensearch-benchmark
opensearch-benchmark copied to clipboard
Download data from S3 location using gateway endpoint and no internet
**Is your feature request related to a problem? Please describe. We want to run the benchmark test from an Amazon EC2 instance in private subnet with no connectivity to internet but it can download files from S3 using Gateway endpoint.
A clear and concise description of what the problem is. We want to run the benchmark test from an Amazon EC2 instance in private subnet with no connectivity to internet but it can download files from S3 using Gateway endpoint.
Describe the solution you'd like We want to ability to be able to download files from S3 even without internet as connectivity to S3 is there using gateway endpoint. Currently OSB using HTTP url which can only be accessed using internet for downloads.
Describe alternatives you've considered
I cloned the OSB code and updated workload/loader.py file to use boto3 and download the files without internet
Additional context
Add any other context or screenshots about the feature request here.
@himanshu-amazon Thanks for bringing attention to this. This would certainly be useful for individuals with similar testing setups. You're more than welcome to implement this, create a PR, and tag maintainers once it's ready to be reviewed!