Add-DatabricksCluster error when piping additional configurations/attributes.

[saldroubi]

Trying to use splatting so I have $clusterParams is read from file and is hashtable and looks like this:

Name Value

---

DriverNodeTypeId Standard_E16ds_v4 MaxWorkers 10 SparkConf {spark.sql.session.timeZone, spark.sql.adaptive.enabled, spark.databricks.io.cache.enabled, spark.databricks.repl.allowedLanguages…} MinWorkers 4 AutoterminationMinutes 45 ClusterName spark SparkVersion 10.4.x-scala2.12 ClusterMode HighConcurrency NodeTypeId Standard_E16ds_v4

It works when I do this call: Add-Databricks @clusterParams

But I also want to pass Azure attributes. So I created a PS custom object $obj for that and piped it to call but I am getting an error. Here is the object

$obj | ConvertTo-Json

{ "azure_attributes": { "first_on_demand": 1, "availability": "SPOT_WITH_FALLBACK_AZURE", "spot_bid_max_price": -1 } }

I tried $obj | ConvertTo-Json | Add-DatabricksCluster $clusterParams and $obj Add-DatabricksCluster $clusterParams

Both produce this error:

Add-DatabricksCluster: Cannot bind positional parameters because no names were given.

What is the proper way to call this function when you trying to pass additional configurations not in parameter list?

I notice that there is a parameter called AwsAttributes. In azure if you do a GET-DatabricksCluster you get this line in output

azure_attributes : @{first_on_demand=1; availability=SPOT_WITH_FALLBACK_AZURE; spot_bid_max_price=-1}

Which I am trying to set when Adding (creating) a cluster. That is the jest of the problem I am having. So that is why I was trying to pipe an object to the function with these attributes.

I am somewhat confused why this isn't working because you have a parameter called $clusterObject which is ValueFromPipeline.

Thank you.

[gbrueckl]

so this should actually work: $obj | Add-DatabricksCluster @clusterParams with the latest version I will also add aliases for the parameters of Add-DatabricksCluster so the values in @clusterParams also support the same names as in the actual cluster definition (ClusterName vs cluster_name)

the only thing you need to make sure is that the @clusterParams value only contains values that are native parameters of Add-DatabricksCluster which as it seems would not work for you

alternatively, you can merge the two objects ($clusterParams and $obj) first and then pipe the new object into Add-DatabricksCluster

so here is some code that works for me:

$existingCluster = gdbrc -ClusterID  12345-103331-j2ajjq25

$newAttributes = @{
	"autotermination_minutes" = 61
	"cluster_name" = "my new cluster"
}

$existingCluster | Add-DatabricksCluster @newAttributes

[gbrueckl]

is this still an issue or can we close this?

[saldroubi]

Sorry again for the delay in testing. Yes, I just tested this and it is working fine. I really appreciate how fast you fix these issues. Sometimes I find a workaround and forget to go back and test the fixes you made, hence the delay. But I will try to get better at this.

Sincerely, Sam Al-Droubi