testcontainers
[Enhancement]: Single database per test.
Problem
First linked/referenced in #1165 Shoutout to @0xced
Context: MS-SQL containers + xUnit / testing
One of the frustrating things with running DB tests and test containers is that all the DB tests are synchronous. This is via the Collection which we run all the tests under.
This raises some issues
- slowness: would be nicer to have a multiple DB tests running at the same time
- data isolation: each test should have it's own data so it doesn't conflict/fight with other tests.
- data changed: test 1 might do something to the db data, while test 2 then expects the data to be in a fresh state, but it's modified.
a quick fix to all of this is a single container per test method but that is resource expensive 😢 Resetting the Db back after each test still means we're stuck with synchronous tests.
Solution
Would be really lovely would be to have a mix of both!
- 🐳 single container
- 💾 single db per test method
(I believe this is what 🐦⬛ RavenDb does?)
In the context of MS-SQL this could be achieved via the connection string Initial Catalog key/value.
So maybe this means
- we don't use the
[Collection()]attribute (which forces all the tests to run in synchronous mode) - when the test run first starts (is this called the test app?), we create the single container
- when each test method runs, then we can set the initial catalogue to be unique.
For example - here's two classes to try and set this up. I'm not sure how close this is to #1165 PR code:
// Simple fixture which is ran once at start when the test run first runs..
public class SqlServerFixture : IAsyncLifetime
{
private readonly MsSqlContainer _msSqlContainer;
public SqlServerFixture() =>
_msSqlContainer = new MsSqlBuilder()
.WithImage("mcr.microsoft.com/mssql/server:2022-CU13-ubuntu-22.04")
//.WithReuse(true)
.Build();
public string ConnectionString => _msSqlContainer.GetConnectionString();
public async Task InitializeAsync()
{
await _msSqlContainer.StartAsync();
}
public async Task DisposeAsync()
{
if (_msSqlContainer != null)
{
await _msSqlContainer.StopAsync();
}
}
}
// Sample 'DB' base class for tests
public abstract class BaseSqlServerTest(
SqlServerFixture _sqlServerFixture,
ITestOutputHelper _testOutputHelper) : IClassFixture<SqlServerFixture>, IAsyncLifetime
{
protected string ConnectionString { get; private set; }
public async Task InitializeAsync()
{
// Generate a unique database name using the test class name and the test name
const int maxDatabaseNameLength = 100;// MSSql has a problem with long names.
var guid = Guid.NewGuid().ToString().Replace("-", "");
var testName = _testOutputHelper.TestDisplayName();
var uniqueDbName = $"{testName}_{guid}";
if (uniqueDbName.Length > maxDatabaseNameLength)
{
var truncatedText = uniqueDbName.Substring(0, maxDatabaseNameLength - Guid.NewGuid().ToString().Length);
uniqueDbName = $"{truncatedText}_{guid}";
}
// Update the connection string to use the unique database name.
// ⭐⭐ This is the magic 🪄🪄🪄
ConnectionString = new SqlConnectionStringBuilder(_sqlServerFixture.ConnectionString)
{
InitialCatalog = uniqueDbName
}.ToString();
_testOutputHelper.WriteLine($"** SQL Server is running. Connection String: {ConnectionString}");
}
public Task DisposeAsync() => Task.CompletedTask;
}
So now we can just inherit this into our own xUnit test class:
public class DoSomethingAsyncTests(SqlServerFixture _sqlServerFixture, ITestOutputHelper _testOutputHelper)
: BaseSqlServerTest(_sqlServerFixture, _testOutputHelper)
{
}
So what I left out of this solution is creating your own DbConnection or EF DbContext. Do that .. then you can Seed your data and you're good to go. You can even add that functionality to the above BaseSqlServerTest class if you want that to happen for each test ran or just set some protected properties which can be accessible in all your concrete classes which inherit from this.
Benefit
- Single Container: Container creation is expensive
- Faster DB tests: a single db and single isolated data per test. tests can run parallel now! (if xUnit allows that)
Potential downside: each test will seed the data which could be more records than needed. That's up to the developer. Potential downside: each tests has to create all the tables and other schema objects.
I'm assuming that the sum of the positive (minus the negs) will still be faster that the current 'Collection' solution.
Alternatives
Would you like to help contributing this enhancement?
Yes
Hi,
In a scenario when you have hundreds of tests and each test needs to run db migration consisting of tens (if not hundreds) of migration files the tests become slow anyway. You also do want to test a scenario as close to real life as possible meaning multiple records created, modified, deleted, etc. in parallel (hello deadlocks). 1 db per test does not appear to be such an improvement then.
We achieve great test isolation using unique user ids - guid - then each user creates its own entity, let's say shopping cart, puts items in the cart, updates, etc. The cart is then selected for assertion by a userId. Hundreds of tests are getting executed in parallel thanks to Xunit.Extensions.Ordering
@summerson1985 Great points but also different scenario's. Yep, it's true that it can be a very important test to see how systems work with load. I can't stress hard enough how much mental anquish i've suffered over the years when seeing 'deadlock' errors/situations 😭 💀
But those are a specific test condition which would generally be targeting applications with lots of multicurrent requests. I would have thought that a common starting/entry level test scenario would be to just making sure all the DB queries work. For 1 person. With -some- existing data.
I was hoping this would be considered with their .NET library, especially considering they are doing a heap of awesome work on #1165
@HofmeisterAn 👋🏻 G'day!
I'm not sure what the xUnit.net package has to do with this issue? As in ... your package impliments this concept of a unique db per test ?
@HofmeisterAn and pulling in @0xced (from #1165) into this convo to provide more answers.
do we still need this issue?
I think so.
I couldn't find any documentation on the xUnit TC nuget that is available. I did look at the sample test code to see how it's getting used and I think I understand the following:
- Each test class (which has test methods Fact/Theory in them) will inherit from a new class and it allows A SINGLE TESTCONTAINER to be linked to this test class
- Each test method will create/destroy the container
If my understanding is correct, then this is different to this issue.
I'm suggesting
- A single container is created for the entire test run (so it's like an AssemblyFixture
[assembly: AssemblyFixture(typeof(YourTestContainerFixture))] - Each test will have some 'smarts' to change the connection string / create the db, per test -method- that is ran.
- A test class could require more than 1 TestContainer. Eg. Database + Redis + Storage.
So i'm not sure if this is already in the code base and there's nothing to do or there's some changes required.
So first answer for everyone: did I summarize @0xced 's PR/changes?
I couldn't find any documentation on the xUnit TC nuget that is available.
It's here: Testing with xUnit.net
- A single container is created for the entire test run (so it's like an AssemblyFixture
[assembly: AssemblyFixture(typeof(YourTestContainerFixture))]
The DbContainerFixture is already compatible to be an assembly fixture. It's even designed to be subclassed! Getting a fresh new database for each test can be achieved with 3 lines of code.
Here's how it can be done for PostgreSQL with the Testcontainers.XunitV3 package.
using System.Data.Common;
using System.Threading;
using Npgsql;
using SampleCode.Tests;
using Testcontainers.PostgreSql;
using Testcontainers.Xunit;
using Xunit;
using Xunit.Sdk;
[assembly: AssemblyFixture(typeof(PostgreSqlFixture))]
namespace SampleCode.Tests;
public class PostgreSqlFixture(IMessageSink messageSink) : DbContainerFixture<PostgreSqlBuilder, PostgreSqlContainer>(messageSink)
{
private int _dbNumber;
public override DbProviderFactory DbProviderFactory => NpgsqlFactory.Instance;
public NpgsqlDataSource GetNewDatabase()
{
var dbName = $"testdb_{Interlocked.Increment(ref _dbNumber)}";
CreateCommand($"CREATE DATABASE {dbName}").ExecuteNonQuery();
return NpgsqlDataSource.Create(new NpgsqlConnectionStringBuilder(ConnectionString) { Database = dbName, IncludeErrorDetail = true });
}
}
Then in your test simply call the GetNewDatabase() method on the fixture to get a fresh new database. In this example, Test1 and Test2 will operate on two different databases running in the same container.
using Npgsql;
using Xunit;
namespace SampleCode.Tests;
public class UnitTest1(PostgreSqlFixture fixture)
{
[Fact]
public void Test1()
{
NpgsqlDataSource dataSource = fixture.GetNewDatabase();
// Use dataSource to test whatever needs to be tested
}
[Fact]
public void Test2()
{
NpgsqlDataSource dataSource = fixture.GetNewDatabase();
// Use dataSource to test whatever needs to be tested
}
}
Requirements for how to create a new database (and possible seed some data) can vary a lot from one project to another so this part is better left outside of the Testcontainers.XunitV3 package.
Also note how this requires a reference to the Npgsql package that Testcontainers.Xunit and Testcontainers.XunitV3 don't want to take, by design.
- Each test will have some 'smarts' to change the connection string / create the db, per test -method- that is ran.
Having an explicit method on the fixture that creates a new db is probably the most straightforward way to go as shown above.
- A test class could require more than 1 TestContainer. Eg. Database + Redis + Storage.
You can inject multiple fixtures with xUnit.net so this should not be an issue.
Ok wow @0xced - this is working very nicely now.
My tests were about 20-30 mins and now it's down to 70 secs.
- Using the above code to create an MSSql instance (via AssemblyFixture)
- Unique db created per test which returns the new unique connection string.
- note: some tests don't need any data, so I create an empty db
- note: some tests required seeded data. it was taking about 18 secs to seed all my data. Instead, I now restore a bak which is near instant.
- I am "reusing" the sql container because normal start time is always a minimum of 10 seconds :( It's so crap!
Further info:
- It would be so nice if the MSSql doesn't 'stop' after the tests but just stays running. so then i don't need to wait.
- On the Easter Weekend, I made a 'CleanupWatcher' which removes the container instance if it hasn't been "touched" after 'x' seconds.
- Each test sends a message to the watcher (which is just a simple API host) saying which container has been used and the expiry timer resets.
- later, this api will be put into a container and when nothing is left to terminate, it will end, thus self terminating.
- this is similar to the Ryuk Reaper.
@0xced this is good but now I notice that the tests run one after the other. Is it possible to parallelize them?
@rsgilbert xUnit.net parallelization is pretty well explained in the Running Tests in Parallel documentation.
If you want to parallelize tests within a test collection, you should have a look at Meziantou.Xunit.ParallelTestFramework and its introductory blog post: Parallelize test cases execution in xUnit by Gérald Barré (aka. meziantou).