Issue with some methods from Microsoft.Spark.Sql while using Docker for debug from VS

[paul-pl]

Hello, I'm getting an error during debug from Visual Studio. Docker container 3rdman/dotnet-spark:latest is up and running with port mapping -p 5567:5567.

Below sample code is failing at last First() method. Show() and Count() works fine, but First() is failing due to connection error to 127.0.0.1:40687 Similar issue is with another methods.

Any idea what I've missed? Has anyone faced similar problem?

Sample dotnet code:

          dataFrameTest.Sort().Show();
          dataFrameTest.Count();
          var firstRowFromDataFrameTest = dataFrameTest.First();

Log from dotnet app in VS:

[Info] [ConfigurationService] Using port 5567 for connection.
[Info] [JvmBridge] JvMBridge port is 5567

System.Net.Internals.SocketExceptionFactory.ExtendedSocketException: 'No connection could be made because the target machine actively refused it. 127.0.0.1:40687'

Log from Docker:

INFO DotnetRunner: Starting DotnetBackend with .
INFO DotnetBackend: The number of DotnetBackend threads is set to 10.
INFO DotnetRunner: Port number used by DotnetBackend is 5050
INFO SparkContext: Running Spark version 3.0.1
...
* .NET Backend running debug mode. Press enter to exit *

...
INFO DAGScheduler: Job 0 finished: count at NativeMethodAccessorImpl.java:0, took 1.889003 s
Exception in thread "serve-DataFrame" java.net.SocketTimeoutException: Accept timed out
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
	at java.net.ServerSocket.implAccept(ServerSocket.java:560)
	at java.net.ServerSocket.accept(ServerSocket.java:528)
	at org.apache.spark.security.SocketAuthServer$$anon$1.run(SocketAuthServer.scala:58)

It seems it's failing within Microsoft.Spark.Sql.DataFrame class where socket.Connect is being called with parameters coming from GetConnectionInfo

        private IEnumerable<Row> GetRows(string funcName, params object[] args)
        {
            (int port, string secret, _) = GetConnectionInfo(funcName, args);
            using ISocketWrapper socket = SocketFactory.CreateSocket();
            socket.Connect(IPAddress.Loopback, port, secret);
            foreach (Row row in new RowCollector().Collect(socket))
            {
                yield return row;
            }
        }