mack Unit test over multiple python versions

I started with the currently supported python versions and any minor versions of delta lake > 1.0 - I'm happy to adjust these as needed. I thought it also might be worthwhile to test over different versions of pyspark, so I included the minor versions > 3 as well:

pyspark-version: ["3.1", "3.2", "3.3", "3.4"]

Jul 25 '23 02:07 ianwesleyarmstrong

I've been running the tests on my fork and it appears that python 3.11 may cause some problems - I'll remove that for the time being.

https://github.com/ianwesleyarmstrong/mack/actions/runs/5652132175

Jul 25 '23 03:07 ianwesleyarmstrong

How does this deal with the Delta Lake / PySpark versions that are incompatible?

I don't think mack supports Delta 1.0.

Here are the version combinations I think we should run (Delta Lake version | PySpark version):

2.4.x | 3.4.x 2.3.x | 3.3.x 2.2.x | 3.3.x 2.1.x | 3.3.x 2.0.x | 3.2.x

This comes from the docs.

Oct 01 '23 04:10 MrPowers

How does this deal with the Delta Lake / PySpark versions that are incompatible?

I modified it to only test those pairs listed in the docs using Python 3.10 like you suggested

Successful run

Happy to make any other suggestion improvements you find! @MrPowers

Nov 01 '23 01:11 ianwesleyarmstrong

Hi @MrPowers, Is there any updates on this branch? I'm looking forward to using mack on pyspark 3.4.1 and delta-spark 2.4.0.

Nov 24 '23 08:11 andreale28