[BUG]: Error connecting to Azure SQL database using SqlConnection when within a private VNET in Azure

[RichardDalton]

When trying to connect to an Azure SQL database from an Azure Databricks instance that is within a private VNET it fails with the following error: Unhandled exception. Microsoft.Data.SqlClient.SqlException (0x80131904): A connection was successfully established with the server, but then an error occurred during the login process. (provider: TCP Provider, error: 35 - An internal exception was caught)

Same code works on a Databricks cluster that is not in the VNET when accessing the same database. Loading data from the database via SparkSession.Read method also works, so this issue seems isolated to using the SqlClient.

To Reproduce

Databricks is configured to run .NET code as per this guide: [https://docs.microsoft.com/en-us/dotnet/spark/tutorials/databricks-deployment Runtime version: 7.3 LTS (includes Apache Spark 3.0.1, Scala 2.12)

Simplest code that fails:

public void Run()
{
     var _sqlStaging = "connection string";
     using var conn = new SqlConnection(_sqlStaging);
     conn.Open(); // Fails here
     var param = new DynamicParameters(new { metricId = Metrics.Total });
 
     var records = conn.Query("StoredProcedureName", param, commandType: CommandType.StoredProcedure)
           .ToList();
 
     Console.WriteLine(records.Count);
}

Expected behavior The database connection succeeds and returns data.