spark
spark copied to clipboard
Accessing azure databricks table using spark from azure function
I am trying to access azure databricks table. I have created a time trigger azure function which should connect to the databricks. I have used the jdbc url, personal access token available in ADB
when I run the azure function, I get an error System.Private.CoreLib: Exception while executing function: GetMetaRegistryData. System.Net.Sockets: No connection could be made because the target machine actively refused it. <<IP>>:5567.
SparkSession spark = SparkSession .Builder() .AppName("Accessing ADB table from Azure fn") .GetOrCreate();
DataFrame jdbcDf = spark.Read()
.Format("jdbc")
.Options(
new Dictionary<string, string>
{
{"url", "jdbc:spark://adb-instanceURL/databasename;transportMode=http;ssl=1;httpPath=sql/protocolv1/o/;UID=token;PWD=peronal acess token"},
{"dbtable", "mytablename"}
})
.Load();
jdbcDf.Show();
spark.Stop();
@KarpagamGurumurthy how are you running your .NET Spark application in Azure Functions? Are you attempting to run the manually run the application binary? ie, <app name>.exe
or are you using spark-submit
?