Publish dedicated modules (pre-configured containers)

[HofmeisterAn]

Is your feature request related to a problem? Please describe. Testcontainers modules (pre-configured containers) don't use their hole potential. Dedicated modules can offer more functionalities like including vendor dependencies etc.

Describe the solution you'd like Move each module into its own project (inside this repository) and deploy it as a dependency.

• [ ] Add a dedicated example module (Testcontainers.Example, Testcontainers.Example.Tests)
• [ ] Add a common (shared) test structure (xUnit test attributes, logging configuration, test assets, etc.)
  • xUnit test attributes: Run tests only on specific environments
• [ ] Split tests into multiple projects (especially tests that test static classes or properties)
• [ ] Add a Cake task to deploy modules as NuGet dependencies
• [ ] Migrate CouchbaseTestcontainer
• [ ] Migrate CouchDbTestcontainer
• [ ] Migrate ElasticsearchTestcontainer
• [ ] Migrate MariaDbTestcontainer
• [ ] Migrate MongoDbTestcontainer
• [ ] Migrate MsSqlTestcontainer
• [ ] Migrate MySqlTestcontainer
• [ ] Migrate OracleTestcontainer
• [ ] Migrate PostgreSqlTestcontainer
• [ ] Migrate RedisTestcontainer
• [ ] Migrate KafkaTestcontainer
• [ ] Migrate RabbitMqTestcontainer

Describe alternatives you've considered -

Additional context

• #421
• #504
• #507
• #518

[srollinet]

Pay attention if you include vendor dependencies to MySqlTestcontainer, it is quite a mess

There are 3 variants

• MySql.Data
• MySqlConnector
  • >= 1.0.0
  • < 1.0.0 (which is still used a lot)

MySqlConnector < 1.0.0 has the same classes + namespaces as MySql.Data. So if both are installed you have ambiguous class definitions and you have to add specific code in the csproj + in the classes to remove the ambiguity

MySqlConnector >= 1.0.0 has a breaking change to fix the namespace issue

So including one of these packages is a bit risky 😞

[vlaskal]

Hi @HofmeisterAn, if you create first oplementation of

Add a dedicated example module (Testcontainers.Example, Testcontainers.Example.Tests)

then I willl be able to continue and slowly updates modules. It will take time so in the mean time of other PRs I can help with a few of them. What you think?

Or just send me some instructions and I can try it.

[HofmeisterAn]

The upcoming dedicated modules should not just move the current configurations into own projects. It should offer a more advanced design to inherit and add custom builders, configurations and instances of Docker resources. The proposal is part of feature-disabled/493-add-example-module.

The following abstract allows developers to extend and write their own builders incl. configurations specific for their use case without messing around with extension methods, etc.

https://github.com/testcontainers/testcontainers-dotnet/blob/f746adfe1edb7bd8e3296fcca4de19b2e0352ecc/src/Testcontainers.Module.Example/ExampleTestcontainersBuilder.cs#L8-L53

The design allows to inherit from builders and configurations. It supports multiple layers and takes care of forwarding a configuration object from the top of the base builder class to the bottom of the module builder class. It merges the configurations from all layers and creates a Docker resource object without reflection etc.

Before we start to add dedicated modules we need to take a quick a look at the implementation of the configuration classes to make sure that those are consistent.

https://github.com/testcontainers/testcontainers-dotnet/blob/f746adfe1edb7bd8e3296fcca4de19b2e0352ecc/src/Testcontainers.Module.Example/ExampleTestcontainersConfiguration.cs#L33-L35

/cc @kiview @eddumelendez @mdelapenya having a discussion about other designs and improvements would be great.

[KijongHan]

Hey @HofmeisterAn, thoughts on trying to simplify this method override?

https://github.com/testcontainers/testcontainers-dotnet/blob/f746adfe1edb7bd8e3296fcca4de19b2e0352ecc/src/Testcontainers.Module.Example/ExampleTestcontainersBuilder.cs#L35-L47

It looks like many module implementations of the TestcontainersBuilder abstract class will have to have similar switch statements. Having an abstract method

protected abstract TBuilderEntity Clone(ITestcontainersConfiguration testcontainersConfiguration);

defined in the TestcontainersBuilder abstract class could be easier and more transparent for module implementors.

Otherwise, I can't wait to see this change coming through soon!

[HofmeisterAn]

Hey @HofmeisterAn, thoughts on trying to simplify this method override?

Indeed, the part above needs some improvements. I will take a look at it incl. how we pass the builder configuration upwards the "tree" in the next days again. The mentioned abstract method make sense.

[HofmeisterAn]

We would like to share some more thoughts about modules (pre-configured containers and other Docker resources). We are interested in your ideas, thoughts and requirements of a Testcontainers for .NET DSL.

Note: The proposal contains a couple of breaking changes, we try to reduce them to a bar of minimum. The goal is to have no breaking changes at all and flag the old implementation obsolete (if we start to change the class names at all).

classDiagram

class ResourceConfiguration
class ContainerConfiguration
class PostgreSqlConfiguration

class AbstractBuilder_3~TBuilderEntity, TResourceEntity, TConfigurationEntity~
class ContainerBuilder_3~TBuilderEntity, TResourceEntity, TConfigurationEntity~
class ContainerBuilder
class PostgreSqlBuilder

class Container
class PostgreSqlContainer

%% Relationships
ResourceConfiguration <|-- ContainerConfiguration
ResourceConfiguration <|-- NetworkConfiguration
ResourceConfiguration <|-- VolumeConfiguration
ContainerConfiguration <|-- PostgreSqlConfiguration

AbstractBuilder_3 <|-- ContainerBuilder_3
AbstractBuilder_3 <|-- NetworkBuilder
AbstractBuilder_3 <|-- VolumeBuilder
AbstractBuilder_3 --> ResourceConfiguration : Contains

ContainerBuilder_3 <|-- ContainerBuilder
ContainerBuilder_3 <|-- PostgreSqlBuilder

Container <|-- PostgreSqlContainer

ContainerBuilder --> Container : Builds
PostgreSqlBuilder --> PostgreSqlContainer : Builds

%% Annotations
<<Abstract>>AbstractBuilder_3
<<Abstract>>ContainerBuilder_3

%% Definitions
ResourceConfiguration: + IDockerEndpointAuthenticationConfiguration DockerEndpointAuthConfig
ResourceConfiguration: + IReadOnlyDictionary~string, string~ Labels

ContainerConfiguration: + IDockerImage Image
ContainerConfiguration: + IEnumerable~string~ Entrypoint
ContainerConfiguration: + IEnumerable~string~ Command

PostgreSqlConfiguration: + string Username
PostgreSqlConfiguration: + string Password

AbstractBuilder_3: # TConfigurationEntity ResourceConfiguration
AbstractBuilder_3: + TBuilderEntity WithDockerEndpoint(string)
AbstractBuilder_3: + TBuilderEntity WithDockerEndpoint(Uri)
AbstractBuilder_3: + TBuilderEntity WithDockerEndpoint(IDockerEndpointAuthenticationConfiguration)
AbstractBuilder_3: + TBuilderEntity WithLabel(string, string)
AbstractBuilder_3: + TResourceEntity Build()*
AbstractBuilder_3: # TBuilderEntity Clone(ResourceConfiguration)*
AbstractBuilder_3: # TBuilderEntity Merge(TConfigurationEntity, TConfigurationEntity)*

ContainerBuilder_3: + TBuilderEntity WithImage(IDockerImage image)
ContainerBuilder_3: + TBuilderEntity WithEntrypoint(params string[])
ContainerBuilder_3: + TBuilderEntity WithCommand(params string[])
ContainerBuilder_3: # TBuilderEntity Clone(ContainerConfiguration)*

PostgreSqlBuilder: + TBuilderEntity WithUsername(IDockerImage image)
PostgreSqlBuilder: + TBuilderEntity WithPassword(IDockerImage image)

Container: + Task StartAsync()
Container: + Task StopAsync()
Container: + Task DisposeAsync()

Builder ⨉ Resource Configuration aka Resource Definition

Most Docker resources share common properties, like labels. Labels can be assigned to container, image, network or volume resources. Common properties like those are kept in the ResourceConfiguration object. Each Docker resource extends ResourceConfiguration and appends resource specific properties, like the ContainerConfiguration.

This applies not only to Docker resources. For example, modules (a variation of containers) follow the same approach and extend and append their specific properties to the configuration hierarchy.

Each property is immutable.

A generic instance of the ResourceConfiguration is held in AbstractBuilder`3. The builder follows a similar class hierarchy as the configuration. Each Docker resource has at least one builder implementation. The builders provide public methods to configure and create a Docker resource object, like a container:

IDockerContainer container = new PostgreSqlBuilder()
  .WithLabel("Key", "Value") // hierarchy 0
  .WithImage("postgres:15") // hierarchy 1
  .WithUsername("Username") // hierarchy 2
  .WithPassword("Password")
  .Build();

Due to immutability, it is necessary to clone (copy) and merge resource configurations, to receive an object that contains / combines all properties. The base builders are not aware of the particular resource configuration type (like PostgreSqlConfiguration). For base resource configuration types, it is necessary to create an instance of the particular resource configuration before merging it (ExampleBuilder:L36-L50):

protected override PostgreSqlBuilder Clone(IResourceConfiguration resourceConfiguration)
{
  // Receives a configuration update from the base class implementations. The configuration update only contains properties from the base class configuration types.
  // If we merge the configurations immediately we will lose any properties that are unknown by the base configuration:
  // E.g. for IResourceConfiguration ⨉ IPostgreSqlConfiguration we will lose username and password. Due to immutable data we can only merge the same types.
  return this.Merge(new PostgreSqlConfiguration(resourceConfiguration), this.ResourceConfiguration);
}

protected override PostgreSqlBuilder Clone(IContainerConfiguration resourceConfiguration)
{
  // Receives a configuration update from the base class implementations. The configuration update only contains properties from the base class configuration types.
  // If we merge the configurations immediately we will lose any properties that are unknown by the base configuration:
  // E.g. for IContainerConfiguration ⨉ IPostgreSqlConfiguration we will lose username and password. Due to immutable data we can only merge the same types.
  return this.Merge(new PostgreSqlConfiguration(resourceConfiguration), this.ResourceConfiguration);
}

The immutable resource configuration allows us to create and share common configurations for all kind of builders. This makes it easier to run similar configuration with slightly different changes:

var baseBuilder = new PostgreSqlBuilder()
  .WithUsername("Username")
  .WithPassword("Password")
  .WithLabel("Key", "Value");

var postgres15 = baseBuilder
  .WithImage("postgres:15")
  .Build();

var postgres14 = baseBuilder
  .WithImage("postgres:14")
  .Build();

By using copy constructors, resource configurations are copied and merged. The constructors take care of three different cases:

1. A developer calls a public builder method to configure a resource configuration property. Equals the resource configuration a base resource configuration, continue with 2., otherwise 3 (ExampleBuilder:L21-L29, ExampleConfiguration:L11-L17).
2. Create a particular resource configuration from the base resource configuration (ExampleBuilder:L36-L50, ExampleConfiguration:L19-L32).
3. Merge the resource configuration held by the builder with the new change to receive an object that contains / combines all properties (ExampleConfiguration:L34-L39).

Providing a default resource configuration through the abstract builder can be done like this: https://github.com/testcontainers/testcontainers-dotnet/commit/920077464069d39919da82740b4299b5d887825c (not optimized yet). This allows developers to either use and extend the default configuration or even override it and set a completely different default resource configuration.

Lifecycle

Calling Build() creates an instance of a Docker resource. The instance manages the lifecycle of the Docker resource. It starts, stops or creates and deletes the underlying Docker resource and provides access to common properties.

This approach enables to support reusable Docker resources like containers too. A hash value identifies the immutable resource configuration to reuse again.

[HofmeisterAn]

We would like to share more thoughts about the module DSL. We followed the Cognitive Dimensions Of Code Base approach and went through all dimensions to identify the ones we need to put more thought and work into: Resource Configuration aka Resource Definition.pdf. We think it helps to make design decisions with more confident instead of pure gut feeling.

[HofmeisterAn]

I will start with Microsoft SQL Server.

[HofmeisterAn]

I will start with MySQL and MariaDB.

[HofmeisterAn]

Next Redis.

[HofmeisterAn]

Next Oracle.

[HofmeisterAn]

Next PostgreSQL.

[HofmeisterAn]

Next MongoDB.

[vlaskal]

Hi @HofmeisterAn I see you started move modules to own projects. I started work on Azurite. The work should be ready in next week. Is it ok for you?

[HofmeisterAn]

OC, any help is appreciated. Thanks.

[HofmeisterAn]

Next CouchDb.

[HofmeisterAn]

Next Neo4j.

[HofmeisterAn]

Next Elasticsearch.

[HofmeisterAn]

Next Couchbase.

[HofmeisterAn]

Next Kafka.