Facing vulnerability issue with pyspark and ray version using bigdl-spark3==2.4.0

[SjeYinTeoIntel]

Below result is from trivy scan where pyspark==3.1.3 and ray==2.6.3 having vulnerability issue.

I'm trying to upgrade pyspark to 3.2.2 version but it will incompatible with bigdl-dllib-spark3 2.4.0.

when trying to upgrade ray to latest version, bigdl-nano 2.4.0 requires protobuf==3.19.5 where only compatible to ray==2.6.3.

Is there a new release that using updated pyspark & ray library?

[sgwhat]

Hi @SjeYinTeoIntel, we do not have a release for pyspark 3.2.2 and ray 2.9.0, but you could run BigDL applications on pyspark 3.2.2. Installing BigDL will automatically install the required corresponding versions of dependencies.