spark icon indicating copy to clipboard operation
spark copied to clipboard

Accessing azure databricks table using spark from azure function

Open KarpagamGurumurthy opened this issue 3 years ago • 1 comments

I am trying to access azure databricks table. I have created a time trigger azure function which should connect to the databricks. I have used the jdbc url, personal access token available in ADB

when I run the azure function, I get an error System.Private.CoreLib: Exception while executing function: GetMetaRegistryData. System.Net.Sockets: No connection could be made because the target machine actively refused it. <<IP>>:5567.

SparkSession spark = SparkSession .Builder() .AppName("Accessing ADB table from Azure fn") .GetOrCreate();

         DataFrame jdbcDf = spark.Read()
            .Format("jdbc")
            .Options(
                new Dictionary<string, string>
                {
                    {"url", "jdbc:spark://adb-instanceURL/databasename;transportMode=http;ssl=1;httpPath=sql/protocolv1/o/;UID=token;PWD=peronal acess token"},
                    {"dbtable", "mytablename"}
                })
            .Load();

        jdbcDf.Show();
        spark.Stop();

KarpagamGurumurthy avatar Feb 04 '22 20:02 KarpagamGurumurthy

@KarpagamGurumurthy how are you running your .NET Spark application in Azure Functions? Are you attempting to run the manually run the application binary? ie, <app name>.exe or are you using spark-submit ?

suhsteve avatar Mar 10 '22 18:03 suhsteve