delta icon indicating copy to clipboard operation
delta copied to clipboard
verified publisher icon delta-io

[Infra] Separate out delta-spark python tests

Open nastra opened this issue 1 year ago • 3 comments

Which Delta project/connector is this regarding?

  • [ ] Spark
  • [ ] Standalone
  • [ ] Flink
  • [ ] Kernel
  • [X] Other (Infra)

Description

Separate out delta-spark python tests to their own GitHub CI

How was this patch tested?

CI tests

Does this PR introduce any user-facing changes?

No

nastra avatar Aug 13 '24 15:08 nastra

Before this PR:

  • delta-spark against Spark 3.5.2 -- Scala 2.12 :: 84, 115, 95 -- AVG: 98.00
  • delta-spark against Spark 3.5.2 -- Scala 2.13 :: 65, 93, 78 -- AVG: 78.67

With this PR:

  • delta-spark against Spark 3.5.2 -- Scala 2.12 :: 77, 108, 88 -- AVG: 91
  • delta-spark against Spark 3.5.2 -- Scala 2.13 :: 71, 93, 84 -- AVG: 82.6

scottsand-db avatar Aug 14 '24 18:08 scottsand-db

Hi, could you also consider this PR? It may help a bit: https://github.com/delta-io/delta/pull/2690

felipepessoto avatar Aug 15 '24 18:08 felipepessoto

we also need to split out the same for the spark master isn it?

tdas avatar Aug 21 '24 21:08 tdas

@tdas << we also need to split out the same for the spark master isn it? >> we don't run python tests for spark master at the moment

scottsand-db avatar Aug 28 '24 20:08 scottsand-db