Set up testcontainers

[carlspring]

Task Description

We should consider setting up our tests to use testcontainers.

Tasks

The following tasks will need to be carried out:

• [x] Investigate what needs to be done to get things working (and document the findings here).
• [x] ~Investigate, if we can somehow have parameterized (JUnit 5) integration tests that can work with both testcontainers and Amazon.~ - We'll be using JUnit Tags for now (see #183)
• [x] Set up testcontainers so it starts the container and tests when -Pit-minio is used.
• [x] Enable MinIO as part of the GH Actions pipeline.
• [ ] Create a JUnit extension to start the MinIO testcontainer and use it in @MinioIntegrationTest with @ExtendWith in tests that can run against MinIO. An exampe of how to do this can be found here.
• [ ] Make integration tests use the correct URI when creating the filesystem depending on the running.it property (i.e. for S3 s3://s3.amazonaws.com/ and s3://localhost:9000/ for MinIO). Perhaps it would be a good idea to add this into the BaseIntegrationTest as a method which generates the appropriate URI for you.
• [ ] Configure integration tests to use testcontainers via @MinioIntegrationTest where this is possible. Some integration tests can be ran against S3 and MinIO since the APIs are compatible. If that is possible, then the test can be annotated with both @S3IntegrationTest and @MinioIntegrationTest.

Task Relationships

This task:

• Is required for: #103
• Relates to: #12 , #140

Help

• Our chat channel
• Points of contact:
  • @carlspring
  • @steve-todorov

[mslowiak]

I have some implementation idea to proceed with that task.

1. We can have system property when running tests such as testEnvironment.
2. We would call integration tests with mvn clean install -Pintegration-tests -DtestEnvironment={value}.
3. The value property can be MIN_IO and AMAZON_S3.
4. When you do not specify the testEnvironment property MIN_IO will be a default one.
5. Each integration test will extend the IntegrationTestBase class which will have @BeforeAll method annotated to run testcontainers if testEnvironment property is MIN_IO.

@carlspring @steve-todorov What are your thoughts about this solution?

[carlspring]

Hi @mslowiak ,

I don't think the idea with the property would be the best one. We should use Maven profiles for this instead. You might want to run just the MinIO tests locally while you're developing something, but then again, when things are executed on the CI server, both profiles would be enabled.

Also, @steve-todorov is working on #184, which would be somewhat related, so feel free to have a look at it.

[steve-todorov]

@mslowiak I've done something similar as part of #183 (PR-184). I've created two profiles it-s3 for s3 integration tests and it-minio for minio ones. If you use -Pit-s3 it can pick up S3FS_* env variables so you can run this in a container. Same goes for -Pit-minio which picks up S3FS_MINIO_* ones. Tests tagged with @MinioIntegrationTest or @S3IntegrationTest will be executed using the appropriate credentials. :)

I haven't looked into test containers that much, but maybe we could use a singleton instance which starts at the very beginning and we just expose those credentials as env variables?