Persist failure due to database connection errors marks persist operations as rejected

[marcotod1410]

Version Information Version of Akka.NET? Reproduced on both 1.4.49 and 1.5.26 Which Akka.NET Modules? Akka.Persistence.Sql.Common (Akka.Persistence.SqlServer and Akka.Persistence.Sqlite)

Describe the bug If a Persist operation of a message fails due to a connection error, the persist operation is marked as rejected and not failed. Hence, the actor is not stopped leading to further errors. We're reproduced the bug with SQL Server persistence, but we were able to narrow it down to all implementations of SqlJournal inside Akka.Persistence.Sql.Common.

To Reproduce Execute the following code. It uses the Sqlite plugin as an example. It simulates the shutting down of the database by basically making the CreateConnection method fail programmatically.

internal class Program
{

    public static readonly Config Config = @$"
                        akka.persistence.journal.plugin = ""akka.persistence.journal.sql""
                        akka.persistence.journal.sql.class = ""{typeof(CustomSqlJournal).AssemblyQualifiedName}""            
                        akka.persistence.journal.sql.connection-string = ""Data Source=database.db""
                        akka.persistence.journal.sql.auto-initialize = true  # just for db initialization, not necessary after first run
";

    static async Task Main(string[] args)
    {
        var system = ActorSystem.Create("MySystem", Config);
        var props = Props.Create(() => new TestActor("test"));

        var child = system.ActorOf(props);

        child.Tell(new SaveName(1));

        Console.WriteLine("Press any key to shut down SQL persistence...");
        Console.ReadKey();

        CustomSqlJournal.Enabled = false;

        child.Tell(new SaveName(2));

        Console.ReadKey();

        await system.WhenTerminated;
    }

    public record SaveName(int Counter);

    public class TestActor : ReceivePersistentActor
    {
        public override string PersistenceId { get; }

        public TestActor(string actorName)
        {

            PersistenceId = typeof(TestActor).AssemblyQualifiedName + "_" + actorName;

            Recover<SaveName>(_ =>
            {
                Console.WriteLine($"Recovered SaveName message #{_.Counter}");
            });

            Recover<RecoveryCompleted>(_ =>
            {
                Console.WriteLine("Recovery completed");
            });

            Command<SaveName>(x =>
            {
                Persist(x, _ => Console.WriteLine($"Persisted SaveName #{x.Counter}"));
                Console.WriteLine($"Received save name message #{x.Counter}");
            });

        }

        protected override void PostStop()
        {
            Console.WriteLine("Actor is stopped");
            base.PostStop();
        }

        protected override void PreStart()
        {
            base.PreStart();
        }
    }

    public class CustomSqlJournal : SqliteJournal
    {

        public static bool Enabled = true;

        public CustomSqlJournal(Config journalConfig) : base(journalConfig)
        {
        }

        protected override DbConnection CreateDbConnection(string connectionString)
        {
            if(!Enabled)
            {
                throw new Exception("Unable to create a SQL connection");
            }

            return base.CreateDbConnection(connectionString);
        }
    }
}

Expected behavior At first, messages are recovered and written without issues. After pressing a key to make the SqlJournal fail, the console must print a persist failure message, with the exception thrown in CustomSqlJournal, and the message Actor is stopped.

Actual behavior After pressing the key, a Persist rejected error is printed to the console and no message Actor is stopped is printed, inferring that the actor is still alive.

Actual console output:

Press any key to shut down SQL persistence...
Recovered SaveName message #1
Recovered SaveName message #1
Recovery completed
Received save name message #1
Persisted SaveName #1
Received save name message #2
[ERROR][07/24/2024 08:58:49.357Z][Thread 0009][akka://MySystem/user/$a] Rejected to persist event type [TestAkkaPersistenceSqlServer.Program+SaveName] with sequence number [4] for persistenceId [TestAkkaPersistenceSqlServer.Program+TestActor, TestAkkaPersistenceSqlServer, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null_test] due to [Unable to create a SQL connection].
Cause: System.Exception: Unable to create a SQL connection
   at TestAkkaPersistenceSqlServer.Program.CustomSqlJournal.CreateDbConnection(String connectionString) in C:\Users\marco.todisco\source\repos\TestAkkaBackoff\TestAkkaPersistenceSqlServer\Program.cs:line 94
   at Akka.Persistence.Sql.Common.Journal.SqlJournal.<WriteMessagesAsync>b__17_0(AtomicWrite message)

Environment This was reproduced on a Windows console application on .NET 6.

Additional context According to AsyncWriteJournal.WriteMessagesAsync method documentation, which is extended by SqlJournal: Data store connection problems must not be signaled as rejections..

This is the main reason why I've interpreted the rejection of the message as a bug.