dotnet
.WithDataBindMount() not working with EF and Postgres or MongoDB
Is there an existing issue for this?
- [X] I have searched the existing issues
Describe the bug
I'm unable to get .WithDataBindMount() working with EF and Postgres or MongoDB.
The included sample works great if I use .WithDataVolume() instead of .WithDataBindMount(). I was unable to find any online examples of how to use .WithDataBindMount() in this sort of context.
The sample repo showing this problem is a minimal fork of a repo described in a blog post by @NikiforovAll. The blog post and original repo describe what is basically a hello world Aspire database sample. The original version works great, the forked version crashes, and the only difference is .WithDataBindMount() vs .WithDataVolume().
The AppHost has for example:
var usersDb = builder
.AddPostgres("dbserver", pAdmin, password)
//.WithDataVolume()
.WithDataBindMount("../../volumes/PostgresUsersDb")
.WithPgAdmin(c => c.WithHostPort(5050))
.AddDatabase("users-db");
When the running code attempts a migration to populate the databases, the code throws an exception in await dbCreator.ExistsAsync(cancellationToken) that is Npgsql.NpgsqlException: Exception while reading from stream in the following database migration code:
private static async Task EnsureDatabaseAsync(
UsersDbContext context,
CancellationToken cancellationToken
)
{
var dbCreator = context.GetService<IRelationalDatabaseCreator>();
var strategy = context.Database.CreateExecutionStrategy();
await strategy.ExecuteAsync(async () =>
{
// Create the database if it does not exist.
// Do this first so there is then a database to start a transaction against.
if (!await dbCreator.ExistsAsync(cancellationToken))
{
await dbCreator.CreateAsync(cancellationToken);
}
});
}
The AppHost throws a similar exception in an .AddMongoDB() context but successfully executes in an .AddElasticsearch() context.
Again, for clarity, the sample code runs without issue if configured to use .WithDataVolume(), the issue only happens with .WithDataBindMount().
Expected Behavior
The Aspire project should successfully initialize the Postgres and MongoDB databases when launched.
Steps To Reproduce
- Clone the sample repo showing this problem
- Open the solution (I'm using Rider on apple silicon, presumably Visual Studio would work too)
- Run/Debug the Aspire project
Exceptions (if any)
When the running code attempts a migration to populate the databases, the code throws an exception in await dbCreator.ExistsAsync(cancellationToken) that is Npgsql.NpgsqlException: Exception while reading from stream
.NET Version info
.NET SDK: Version: 8.0.201 Commit: 4c2d78f037 Workload version: 8.0.200-manifests.011fccd5
Runtime Environment: OS Name: Mac OS X OS Version: 14.5 OS Platform: Darwin RID: osx-arm64 Base Path: /usr/local/share/dotnet/sdk/8.0.201/
.NET workloads installed: [aspire] Installation Source: SDK 8.0.200 Manifest Version: 8.0.1/8.0.100 Manifest Path: /usr/local/share/dotnet/sdk-manifests/8.0.100/microsoft.net.sdk.aspire/8.0.1/WorkloadManifest.json Install Type: FileBased
Host: Version: 8.0.4 Architecture: arm64 Commit: 2d7eea2529
.NET SDKs installed: 6.0.401 [/usr/local/share/dotnet/sdk] 6.0.403 [/usr/local/share/dotnet/sdk] 6.0.404 [/usr/local/share/dotnet/sdk] 6.0.405 [/usr/local/share/dotnet/sdk] 6.0.407 [/usr/local/share/dotnet/sdk] 6.0.408 [/usr/local/share/dotnet/sdk] 6.0.410 [/usr/local/share/dotnet/sdk] 6.0.413 [/usr/local/share/dotnet/sdk] 6.0.415 [/usr/local/share/dotnet/sdk] 7.0.100 [/usr/local/share/dotnet/sdk] 7.0.101 [/usr/local/share/dotnet/sdk] 7.0.102 [/usr/local/share/dotnet/sdk] 7.0.202 [/usr/local/share/dotnet/sdk] 7.0.203 [/usr/local/share/dotnet/sdk] 7.0.304 [/usr/local/share/dotnet/sdk] 7.0.307 [/usr/local/share/dotnet/sdk] 8.0.100 [/usr/local/share/dotnet/sdk] 8.0.201 [/usr/local/share/dotnet/sdk]
.NET runtimes installed: Microsoft.AspNetCore.App 6.0.9 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.11 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.12 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.13 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.15 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.16 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.18 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.21 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.23 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 7.0.0 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 7.0.1 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 7.0.2 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 7.0.4 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 7.0.5 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 7.0.7 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 7.0.10 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 8.0.0 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 8.0.2 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.NETCore.App 6.0.9 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.11 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.12 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.13 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.15 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.16 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.18 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.21 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.23 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App] Microsoft.NETCore.App 7.0.0 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App] Microsoft.NETCore.App 7.0.1 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App] Microsoft.NETCore.App 7.0.2 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App] Microsoft.NETCore.App 7.0.4 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App] Microsoft.NETCore.App 7.0.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App] Microsoft.NETCore.App 7.0.7 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App] Microsoft.NETCore.App 7.0.10 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App] Microsoft.NETCore.App 8.0.0 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App] Microsoft.NETCore.App 8.0.2 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App] Microsoft.NETCore.App 8.0.4 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Other architectures found: x64 [/usr/local/share/dotnet/x64]
Environment variables: Not set
global.json file: Not found
Anything else?
I'm using Rider on apple silicon, presumably Visual Studio will exhibit the same issues
@DamianEdwards When I run the AppHost project under OSX with Rider,
dbserver exits and reports:
2024-08-22T16:45:09.5380894 stderr chown: changing ownership of '/var/lib/postgresql/data': Permission denied
2024-08-22T16:45:09.5383047 stderr chown: changing ownership of '/var/lib/postgresql/data/.gitkeep': Permission denied
posts-mongoldb exits and reports:
2024-08-22T16:45:09.3367844 stderr chown: changing ownership of '/data/db': Permission denied
2024-08-22T16:45:09.3369921 stderr chown: changing ownership of '/data/db/.gitkeep': Permission denied
posts-mongodb-mongoexpress exits and reports:
2024-08-22T16:45:13.1870119 Waiting for host.docker.internal:51140...
2024-08-22T16:45:17.7247738 No custom config.js found, loading config.default.js
2024-08-22T16:45:17.7250681 Welcome to mongo-express 1.0.2
2024-08-22T16:45:17.7250718 ------------------------
2024-08-22T16:45:17.7250728
2024-08-22T16:45:17.7250738
2024-08-22T16:45:47.8851859 stderr Could not connect to database using connectionString: mongodb://host.docker.internal:51140/?directConnection=true"
2024-08-22T16:45:47.9152922 stderr /app/node_modules/mongodb/lib/sdam/topology.js:285
2024-08-22T16:45:47.9153080 stderr const timeoutError = new error_1.MongoServerSelectionError(`Server selection timed out after ${serverSelectionTimeoutMS} ms`, this.description);
2024-08-22T16:45:47.9153098 stderr ^
2024-08-22T16:45:47.9153110 stderr
2024-08-22T16:45:47.9153120 stderr MongoServerSelectionError: Server selection timed out after 30000 ms
2024-08-22T16:45:47.9153131 stderr at Timeout._onTimeout (/app/node_modules/mongodb/lib/sdam/topology.js:285:38)
2024-08-22T16:45:47.9153141 stderr at listOnTimeout (node:internal/timers:569:17)
2024-08-22T16:45:47.9153151 stderr at process.processTimers (node:internal/timers:512:7) {
2024-08-22T16:45:47.9153161 stderr reason: TopologyDescription {
2024-08-22T16:45:47.9153171 stderr type: 'Single',
2024-08-22T16:45:47.9153180 stderr servers: Map(1) {
2024-08-22T16:45:47.9153190 stderr 'host.docker.internal:51140' => ServerDescription {
2024-08-22T16:45:47.9153203 stderr address: 'host.docker.internal:51140',
2024-08-22T16:45:47.9153213 stderr type: 'Unknown',
2024-08-22T16:45:47.9153222 stderr hosts: [],
2024-08-22T16:45:47.9153231 stderr passives: [],
2024-08-22T16:45:47.9153241 stderr arbiters: [],
2024-08-22T16:45:47.9153250 stderr tags: {},
2024-08-22T16:45:47.9153259 stderr minWireVersion: 0,
2024-08-22T16:45:47.9153269 stderr maxWireVersion: 0,
2024-08-22T16:45:47.9153279 stderr roundTripTime: -1,
2024-08-22T16:45:47.9153288 stderr lastUpdateTime: 2199814466,
2024-08-22T16:45:47.9153297 stderr lastWriteDate: 0,
2024-08-22T16:45:47.9153307 stderr error: null,
2024-08-22T16:45:47.9153316 stderr topologyVersion: null,
2024-08-22T16:45:47.9153326 stderr setName: null,
2024-08-22T16:45:47.9153335 stderr setVersion: null,
2024-08-22T16:45:47.9153345 stderr electionId: null,
2024-08-22T16:45:47.9153354 stderr logicalSessionTimeoutMinutes: null,
2024-08-22T16:45:47.9153364 stderr primary: null,
2024-08-22T16:45:47.9153373 stderr me: null,
2024-08-22T16:45:47.9153382 stderr '$clusterTime': null
2024-08-22T16:45:47.9153476 stderr }
2024-08-22T16:45:47.9153486 stderr },
2024-08-22T16:45:47.9153494 stderr stale: false,
2024-08-22T16:45:47.9153502 stderr compatible: true,
2024-08-22T16:45:47.9153509 stderr heartbeatFrequencyMS: 10000,
2024-08-22T16:45:47.9153517 stderr localThresholdMS: 15,
2024-08-22T16:45:47.9153528 stderr setName: null,
2024-08-22T16:45:47.9153535 stderr maxElectionId: null,
2024-08-22T16:45:47.9153543 stderr maxSetVersion: null,
2024-08-22T16:45:47.9153550 stderr commonWireVersion: 0,
2024-08-22T16:45:47.9153557 stderr logicalSessionTimeoutMinutes: null
2024-08-22T16:45:47.9153566 stderr },
2024-08-22T16:45:47.9153623 stderr code: undefined,
2024-08-22T16:45:47.9153636 stderr [Symbol(errorLabels)]: Set(0) {}
2024-08-22T16:45:47.9153644 stderr }
2024-08-22T16:45:47.9228188 stderr
2024-08-22T16:45:47.9228511 stderr Node.js v18.20.3
Also, in case it's relevant, I'm using Rancher Desktop to provide the docker and k8s services
@yoDon thanks. It seems it's what I suspected, an issue adjusting file permissions on the bind mount from within the container. Can you try changing the bind mount source to a folder that's completely empty? Also it's possible there's some difference with Rancher to the container runtimes we test with (Docker Desktop and Podman).
/Cc @danegsta
@DamianEdwards running the sample with empty directories doesn't fix the problem. It removes the error about unable to chown .gitkeep but doesn't remove the error about unable to chown the data related directory.
@yoDon OK then. My guess is it's something to do with Rancher and/or macOS and how we're requesting the bind mount.
/Cc @karolz-ms
@DamianEdwards I just installed Podman, and have a non-root Podman machine.
Good news and bad news: The bad news is having a .gitkeep file present to preserve the empty folder breaks the deployment. The good news is deleting both the .gitkeep file and the target folder works because something (Podman?) is creating the missing target folder prior to mounting it. I'm pretty sure Rancher was NOT creating that folder, so there's still a potential issue.
When the .gitkeep file is present:
2024-08-23T16:28:01.0000000 The files belonging to this database system will be owned by user "postgres".
2024-08-23T16:28:01.0000000 This user must also own the server process.
2024-08-23T16:28:01.0000000
2024-08-23T16:28:01.0000000 The database cluster will be initialized with locale "en_US.utf8".
2024-08-23T16:28:01.0000000 The default database encoding has accordingly been set to "UTF8".
2024-08-23T16:28:01.0000000 The default text search configuration will be set to "english".
2024-08-23T16:28:01.0000000
2024-08-23T16:28:01.0000000 Data page checksums are disabled.
2024-08-23T16:28:01.0000000
2024-08-23T16:28:01.0000000 stderr initdb: error: directory "/var/lib/postgresql/data" exists but is not empty
2024-08-23T16:28:01.0000000 stderr initdb: detail: It contains a dot-prefixed/invisible file, perhaps due to it being a mount point.
2024-08-23T16:28:01.0000000 stderr initdb: hint: Using a mount point directly as the data directory is not recommended.
2024-08-23T16:28:01.0000000 stderr Create a subdirectory under the mount point.
@karolz-ms @danegsta I seem to recall another discussion about whether we should be ensuring the source directory of a bind mount is created before starting the container? The message here seems to suggest we also might want to revisit the details of how WithDataBindMount() works for the Postgres resource as it's suggesting not directly binding an empty directory to the data directory.
@DamianEdwards DCP explicitly creates folders for a bind mount share if one isn't present. We do this to make the behavior consistent between Docker and Podman. Docker, by default, would create a folder for a bind mount if the specified path doesn't exist, but it would also always create the folder as root. On the other hand, Podman out of the box won't create a missing bind mount path at all and will just report an error. We split the difference and create the missing folder as the current user before starting the container.
Hmm OK then. @yoDon in theory then, if you delete the directory, Aspire (DCP) should be auto-creating it for you even when using Rancher.
@DamianEdwards Rancher is still a problem, at least on OSX. It creates the folder but Postgres still complains about ownership
2024-08-26T19:53:06.7261374 stderr chown: changing ownership of '/var/lib/postgresql/data': Permission denied
In case the host OS chown values are relevant, when I cd into the newly created folder, I see
ls -la
total 0
drwx------ 2 don staff 64 Aug 26 19:53 .
drwxr-xr-x 6 don staff 192 Aug 26 19:53 ..
This is almost certainly a user permission issue between the volume mount and the container; it looks as though the default user for the postgresql image is postgres with user id and group id 999. The user you are running as on MacOS most likely doesn't have the same user id or default group id. Since containers preserve folder ownership and permissions when mounting from a *nix OS, the postgres user in the container doesn't have permission to modify the folder. I don't think this behavior would be exclusive to Rancher Desktop; I expect this can be reproduced on Mac/Linux with Docker or Podman desktop as well.
I see three possible options to get this working:
- Switch to using a named volume instead of a bind mount; if you don't need to access container data from the host, named volume mounts are almost always a simpler and less error prone option than bind mounts for data persistence and sharing.
- Add logic to your AppHost to manually create the local bind mount folder (if it doesn't already exist) and grant read+write permission on the folder to everyone before creating the container.
- Same as 2, ensure the folder is created in the AppHost prior to creating the bind mount, but instead of granting everyone access, change the group ownership of the file to the same group ID as the
postgresuser in the container (999 by default). Something likechown :999 <path to share>. This is the most involved option, but gives the most granular control over the bind mount folder permissions.
@danegsta I think you may have missed that this works fine with Podman on OSX.
I appreciate the helpful work arounds, but personally I'm already running on podman to work around this issue.
that's not a resolution, btw, at least not in my opinion. Running databases under aspire is a pretty fundamental use case, and mounts like this are a pretty important way to do it.
This is almost certainly a user permission issue between the volume mount and the container; it looks as though the default user for the postgresql image is
postgreswith user id and group id999. The user you are running as on MacOS most likely doesn't have the same user id or default group id. Since containers preserve folder ownership and permissions when mounting from a *nix OS, thepostgresuser in the container doesn't have permission to modify the folder. I don't think this behavior would be exclusive to Rancher Desktop; I expect this can be reproduced on Mac/Linux with Docker or Podman desktop as well.I see three possible options to get this working:
- Switch to using a named volume instead of a bind mount; if you don't need to access container data from the host, named volume mounts are almost always a simpler and less error prone option than bind mounts for data persistence and sharing.
- Add logic to your AppHost to manually create the local bind mount folder (if it doesn't already exist) and grant read+write permission on the folder to everyone before creating the container.
- Same as 2, ensure the folder is created in the AppHost prior to creating the bind mount, but instead of granting everyone access, change the group ownership of the file to the same group ID as the
postgresuser in the container (999 by default). Something likechown :999 <path to share>. This is the most involved option, but gives the most granular control over the bind mount folder permissions.
@danegsta is there any sample code on how to do this in the apphost? I mean changing the ownership (chown ) of the folder...
Databases and WithDataBindMount are core use cases for Aspire.
Going live in Aspire 9 for new users, this needs to just work out of the box on OSX and Linux in addition to Windows. It's not acceptable to advertise this functionality and have it not actually work on OSX (and similarly not work or be brittle on Linux)
@yoDon while I appreciate this issue is frustrating, Rancher is not one of our supported container runtimes at this time. That said, I'd like to ensure we have as much information as possible to understand this issue.
To summarize this thread so far, is it correct you've reported that when using Rancher you're seeing that the host directory for the bind mount is not created automatically for you, but it is when you using Podman? Do we have any confirmation that data bind mounts for PostgreSQL work on macOS when using Docker Desktop?
/Cc @maddymontaquila (macOS user)
@DamianEdwards I'm not able to test with Docker on OSX, but I did include a minimal repo in the initial report that provides a full repro for the issue.
Also, I wasn't expressing frustration I was trying to indicate the reverse: I've personally worked around this issue and what's important is the developer experience for new developers. My point was given the importance of databases in Aspire projects, this issue should be prioritized in the push to get v9 out.
Could the docs please be updated to say which are and are not supported, so this issue isn't the definitive source of that info?
From: David Fowler @.> Sent: Monday, November 4, 2024 12:19:23 AM To: dotnet/aspire @.> Cc: Don Alvarez @.>; Mention @.> Subject: Re: [dotnet/aspire] .WithDataBindMount() not working with EF and Postgres or MongoDB (Issue #4956)
Rancher is currently unsupported, can you use podman desktop instead?
— Reply to this email directly, view it on GitHubhttps://github.com/dotnet/aspire/issues/4956#issuecomment-2453846750, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AANN3QMHESBE46SOKU43R2LZ63YUXAVCNFSM6AAAAABLB4QUBWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINJTHA2DMNZVGA. You are receiving this because you were mentioned.Message ID: @.***>
It's documented already at: https://learn.microsoft.com/dotnet/aspire/fundamentals/setup-tooling#container-runtime
