System.Net.Sockets.SocketException (0x80004005)

[tiziano-morgia]

trafficstars

Hi, I have a problem with connection and I did not found a solution searching in old issues. I think that is some "lost/wrong parameter" in my connection string or in some incorrect using of connection/reader. I hope that someone has had the same problem in the past, can help me. My software need to read a mysql db and send data using an HttpClient/POST/JSON. So all system is reduced to (for every "group of data"): open connection, perform a select, loop on resultset and for each line (or a group of lines with an break key verify) copy data in an object, json.serialize, send and wait for response (in v2 I need this to flag "no send" data and so on).

My connection is in a disposable class as suggested (open when create and close on dispose). I have tried with Async and not async method (open, openasync, etc...)

All works but after a consistent set of reading I receive this message:

System.Net.Sockets.SocketException (0x80004005): Connessione in corso interrotta forzatamente dall'host remoto
   in MySqlConnector.Protocol.Serialization.SocketByteHandler.DoReadBytesSync(Memory`1 buffer)
--- Fine traccia dello stack da posizione precedente dove è stata generata l'eccezione ---
   in System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   in MySqlConnector.Protocol.Serialization.BufferedByteReader.<ReadBytesAsync>d__2.MoveNext()
--- Fine traccia dello stack da posizione precedente dove è stata generata l'eccezione ---
   in System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   in System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   in MySqlConnector.Protocol.Serialization.ProtocolUtility.ReadPacketAfterHeader(ReadOnlySpan`1 headerBytes, BufferedByteReader bufferedByteReader, IByteHandler byteHandler, Func`1 getNextSequenceNumber, ProtocolErrorBehavior protocolErrorBehavior, IOBehavior ioBehavior)
   in MySqlConnector.Protocol.Serialization.ProtocolUtility.ReadPacketAsync(BufferedByteReader bufferedByteReader, IByteHandler byteHandler, Func`1 getNextSequenceNumber, ProtocolErrorBehavior protocolErrorBehavior, IOBehavior ioBehavior)
   in MySqlConnector.Protocol.Serialization.ProtocolUtility.DoReadPayloadAsync(BufferedByteReader bufferedByteReader, IByteHandler byteHandler, Func`1 getNextSequenceNumber, ArraySegmentHolder`1 previousPayloads, ProtocolErrorBehavior protocolErrorBehavior, IOBehavior ioBehavior)
   in MySqlConnector.Protocol.Serialization.StandardPayloadHandler.ReadPayloadAsync(ArraySegmentHolder`1 cache, ProtocolErrorBehavior protocolErrorBehavior, IOBehavior ioBehavior)
   in MySqlConnector.Core.ServerSession.ReceiveReplyAsync(IOBehavior ioBehavior, CancellationToken cancellationToken)
--- Fine traccia dello stack da posizione precedente dove è stata generata l'eccezione ---
   in System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   in System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   in MySqlConnector.Core.ServerSession.<ReceiveReplyAsyncAwaited>d__81.MoveNext()
--- Fine traccia dello stack da posizione precedente dove è stata generata l'eccezione ---
   in System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   in System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   in MySqlConnector.Core.ResultSet.<<ScanRowAsync>g__ScanRowAsyncAwaited|9_0>d.MoveNext()
--- Fine traccia dello stack da posizione precedente dove è stata generata l'eccezione ---
   in System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   in System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   in MySqlConnector.Core.ResultSet.<ReadAsync>d__7.MoveNext()
--- Fine traccia dello stack da posizione precedente dove è stata generata l'eccezione ---
   in System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   in System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   in MySqlConnector.Core.ResultSet.Read()
   in [MY SOURCE]

[MY SOURCE]

using(var mySqlConn = new MyDbConnector(dbConfig).mySqlConn)
{
  using(var command = new MySqlCommand(SQL_SEL, mySqlConn))
  {
    command.CommandTimeout = 180; //with or without, same error
    using(var reader = await command.ExecuteReaderAsync()) //async or not, same error
    {
      int readRows = 0;
      while(await reader.ReadAsync())
      {
        readRows++;
        try
        {
          if(art.FillFromRawData(reader))
          {
            if((readRows % 500) == 0) logger.Info("[ReadAndSend] Count Record: " + readRows.ToString("N0"));
            jsdata = art.asJson();
            await midConn.PostDataToMiddleware("/article/sync", jsdata); //http client
          }
          else
          {
            logger.Error("[ReadAndSend] Data read error");
          }
        }
        catch(Exception ex)
        {
          logger.Error(ex);
        }
      }
      reader.Close(); //with or without same error
    }
  }
}

[bgrainger]

I'm assuming "Connessione in corso interrotta forzatamente dall'host remoto" is "An existing connection was forcibly closed by the remote host".

That makes me think the cause is on the remote end. I suggest looking at the MySQL general_log to make sure no errors are logged, and checking that wait_timeout is set to a high-enough value that the server won't close an existing connection for inactivity.

What MySQL Server are you using? Do you have any kind of proxy or load balancer?

[tiziano-morgia]

Hi, thanks for your answer. For translation from italian (sorry I forgot to do it!), yes it's correct. I don't know if it is exactly that, but it "say" same thing.

For MySql I have same error in two env: in production is a MySQL 5.6 (I must check exact version), in test local server (my pc) I have a Maria DB 10.1. No proxy on boht env. I have tried with this combinations cliente/server:

• Client On my PC -> VPN -> Server Production (MySQL)
• Client On PC into same Network of Server Prod -> Server Production (MySQL)
• Client On Server Production -> Server Production (MySQL)
• Client On my PC -> Server Test on my PC (MariaDB)

I think is something on my connection string (below) or something similar:

public class MyDbConnector : IDisposable
{
  public readonly MySqlConnection mySqlConn;

  public MyDbConnector(MyDbConfig dbConfig)
  {
    string connString = String.Format(
      "host={0};port={1};user id={2};password={3};database={4};Pooling={5};ConnectionLifeTime={6};Keepalive={7}", 
      dbConfig.HostName, dbConfig.PortNumb, dbConfig.DBUser, dbConfig.DBPswd, dbConfig.DBName,
      dbConfig.Pooling, dbConfig.ConnectionLifeTime, dbConfig.Keepalive);
    mySqlConn = new MySqlConnection(connString);
    mySqlConn.Open();
  }

  public void Dispose()
  {
    mySqlConn.Close();
  }

  public bool isOpen() {
    return mySqlConn != null && mySqlConn.State == System.Data.ConnectionState.Open;
  }
}

How I can check general_log and set wait_timeout? I can try to check and change these...

[tiziano-morgia]

ConnectionLifeTime = "0", Keepalive = "5", Pooling = "false" (I have tried some values without success)

[bgrainger]

You can SELECT @@wait_timeout; from any MySQL client (e.g., MySQL workbench).

Reviewing the general log may require direct access to the MySQL Server (if it's stored in a file), if it's enabled at all.

[tiziano-morgia]

I checked and: wait_timeout production => 28800 wait_timeout my pc=> 28800

For general log, into my pc I have found into [mysql data path] this files (all have last modified on 2016): MySQLLog_Error.log MySQLLog_General.log MySQLLog_SlowQuery.log (how can I enable it? I think that logging is disabled)

Into server prod I have found only a [server name].err but nothing inside in a time near my error.

[tiziano-morgia]

Today error (I have enabled log error on test env, but I cannot do it on production):

[...]
2021-02-22;10:21:54.7305;INFO;[ReadAndSendRecord] Count Record: 520.500
2021-02-22;10:22:04.0693;ERROR;System.IO.EndOfStreamException: Expected to read 71 payload bytes but only received 23.
   in System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   in MySqlConnector.Protocol.Serialization.ProtocolUtility.DoReadPayloadAsync(BufferedByteReader bufferedByteReader, IByteHandler byteHandler, Func`1 getNextSequenceNumber, ArraySegmentHolder`1 previousPayloads, ProtocolErrorBehavior protocolErrorBehavior, IOBehavior ioBehavior)
   in MySqlConnector.Protocol.Serialization.StandardPayloadHandler.ReadPayloadAsync(ArraySegmentHolder`1 cache, ProtocolErrorBehavior protocolErrorBehavior, IOBehavior ioBehavior)
   in MySqlConnector.Core.ServerSession.ReceiveReplyAsync(IOBehavior ioBehavior, CancellationToken cancellationToken)
--- end of stack trace from previous location where exception was thrown ---
   in System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   in System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   in MySqlConnector.Core.ServerSession.<ReceiveReplyAsyncAwaited>d__81.MoveNext()
--- end of stack trace from previous location where exception was thrown ---
   in System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   in System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   in MySqlConnector.Core.ResultSet.<<ScanRowAsync>g__ScanRowAsyncAwaited|9_0>d.MoveNext()
--- end of stack trace from previous location where exception was thrown ---
   in System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   in System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   in MySqlConnector.Core.ResultSet.<ReadAsync>d__7.MoveNext()
--- end of stack trace from previous location where exception was thrown---
   in System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   in System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   in MySqlConnector.Core.ResultSet.Read()
   in rdDao2Mid.Data2Middleware.<ReadAndSendRecord>d__6.MoveNext() in [...]\Data2Middleware.cs:riga 144
2021-02-22;10:22:04.0693;INFO;[ReadAndSendRecord] TotRecord: 1226634 Loop: 13 FROM record 650000 FOR 50000
[...]

I must loop into 1,226,634 record so I divided it into 50,000 records loops (as I wrote above). Another change I had do is into http caller:

      try
      {
        using(var httpCli = new HttpClient())
        {
          httpCli.DefaultRequestHeaders.Host = srvUri.Host;
          StringContent content = new StringContent(jsonData, Encoding.UTF8, "application/json");
          HttpResponseMessage response = await httpCli.PostAsync(srvUri, content);
          //response.EnsureSuccessStatusCode(); <<<<<<<<<<<<<<<<<< REMOVED
          //resMsg = await response.Content.ReadAsStringAsync();   <<<<<<<<<<<<<<<<<< REMOVED
        }
      }
      catch(Exception ex)
      {
        logger.Error(ex);
      }

So I "post and forget" and I can not check for middleware result. Today I will try to reproduce first error type "System.Net.Sockets.SocketException (0x80004005)" on test-env to check logs...

[bgrainger]

Closing old issue without a consistent repro.