Start using spark4-preview versions

[mathbunnyru]

Describe your changes

Issue ticket if applicable

Checklist (especially for first-time contributors)

• [ ] I have performed a self-review of my code
• [ ] If it is a core feature, I have added thorough tests
• [ ] I will try not to use force-push to make the review process easier for reviewers
• [ ] I have updated the documentation for significant changes

[mathbunnyru]

I checked the build logs, and the expected version of spark is installed:

make build/pyspark-notebook

 => => # INFO:__main__:Latest version: 4.0.0-preview2
 => => # INFO:__main__:Downloading and unpacking Spark
 => => # INFO:__main__:Spark directory name: spark-4.0.0-preview2-bin-hadoop3

Then I checked the image itself, and the directories are named properly:

docker run -it --rm quay.io/jupyter/pyspark-notebook bash

(base) jovyan@b6f0e3c1463d:~$ ll /usr/local | grep spark
lrwxrwxrwx  1 root root   43 Oct 20 12:51 spark -> /usr/local/spark-4.0.0-preview2-bin-hadoop3/
drwxr-xr-x 14 root root 4096 Sep 16 04:02 spark-4.0.0-preview2-bin-hadoop3/

And finally, the image tag:

make hook/pyspark-notebook

INFO:__main__:Calculated tag, tagger_name: SparkVersionTagger tag_value: spark-4.0.0-preview2

docker image ls | grep aarch64-spark
quay.io/jupyter/pyspark-notebook   aarch64-spark-4.0.0-preview2   116a1ce7d803   2 minutes ago   4.51GB

So, everything works exactly as expected

[mathbunnyru]

Unfortunately, sparklyr doesn't seem to support spark v4 yet.

When I run spark_available_versions(), it only gives me versions up to 3.5.

I created an upstream issue: https://github.com/sparklyr/sparklyr/issues/3468

[mathbunnyru]

Unfortunately, sparklyr doesn't seem to support spark v4 yet.

When I run spark_available_versions(), it only gives me versions up to 3.5.

I created an upstream issue: sparklyr/sparklyr#3468

It finally has been fixed and I was able to add the test back: https://github.com/jupyter/docker-stacks/commit/2ce2c06a2266c5cf192a2e16af74958741c1b917