Concurrency in tests and audits

[siddharth-sc1903]

https://tobiko-data.slack.com/archives/C044BRE5W4S/p1758688241595459

Hi,

Opening an issue as discussed in the slack thread above. My sqlmesh project has py spark models and uses databricks connect to be able to plan using the cli. We arent able to use duck db for local tests as the tests for these python based models cannot execute on duck, and arent able to use a local spark cluster as we use databricks connect. We hence provision a small cluster in databricks for tests to run here and use sychronized deployments with the CI/CD bot.

I need help with two issues

1. Each time the bot is run, our tests take more than 5 mins to succeed as they are executed one after the other. Same case with audits run from local. This doesnt improve if i provision a medium cluster or any other for that matter.
2. I also would like to understand , if its possible to switch to a local spark cluster(as its more closer to the databricks dialect) for testing while we plan against a prod databricks, as it saves us cost. Would this be the right usage of disable-connect flag? I havent tried this option yet

TIA, Sid

[eakmanrq]

I also would like to understand , if its possible to switch to a local spark cluster(as its more closer to the databricks dialect) for testing while we plan against a prod databricks, as it saves us cost. Would this be the right usage of disable-connect flag? I havent tried this option yet

No that is not the purpose of that flag. As long as databricks-connect is installed within your Python environment then you can't run things against a local Spark cluster.

[siddharth-sc1903]

Thanks for clarifying @eakmanrq .. please lmk if you need more information on 1

[VaggelisD]

Hey @siddharth-sc1903,

Support for unit test concurrency was added a while ago. Try setting concurrent_tasks in your test_connection, e.g:

  databricks:
    connection:
      type: databricks
      server_hostname: ...
      http_path: ...
      access_token: ...
      catalog: ...
    test_connection:
      type: databricks
      server_hostname: ...
      http_path: ...
      access_token: ...
      concurrent_tasks: 8

I believe audits would be safe to run concurrently as well, will forward it internally for us to discuss.

[siddharth-sc1903]

this is great @VaggelisD .. i see test runs already taking only 30% time compared to their previous runs! will await audits next

[blecourt-private]

Thanks for creating this issue @siddharth-sc1903!

In my team we would really like to have concurrency for audits (we have more than 500 audits to run by now...), so we would very much appreciate if this were to be implemented in sqlmesh. @VaggelisD

[blecourt-private]

Any news on running audits concurrently @VaggelisD, @eakmanrq?

[VaggelisD]

Hey @blecourt-private, apologies for the late reply, I don't have an exact timeline on when we'll pick up this issue.

We run audits in a tight loop so making that concurrent should be straightforward; The issue however is not the implementation itself but arguing that concurrent audits won't have side effects, adding sufficient testing to prove so etc, as is the case with anything sequential turned concurrent.