rasterframes Spark 3.2

Pushing for Spark 3.2, very draft

Jun 30 '22 05:06 echeipesh

I'm going to bump the min Spark version in GT as well https://github.com/locationtech/geotrellis/pull/3389 Thogh it should not affect this PR much.

Jul 10 '22 20:07 pomadchin

@pomadchin It'd be nice to get 3.6.3 out with spark 3.2.0 and then 3.7.0 with CES3 and spark 3.2.1. That'd allow this to sync with frameless 0.11 and later with 0.12+.

Jul 12 '22 03:07 echeipesh

@echeipesh 🚀 https://github.com/locationtech/geotrellis/pull/3471 + I'm dropping Spark 3.0 over there https://github.com/typelevel/frameless/pull/637 (frameless will be published for 3.1.x, 3.2.x, and 3.3.x)

Jul 12 '22 13:07 pomadchin

CI is dead? https://app.circleci.com/pipelines/github/locationtech/rasterframes?utm_campaign=workflow-failed&utm_medium=email&utm_source=notification

Jan 03 '23 15:01 pomadchin

I've been looking at the python side of the project enough to form strong enough opinions of it. I think it's actually going to need some work and it doesn't make sense to do it in this PR.

make the build PEP 517 compatible
ditch use of conda for poetry (we're not building a conda package here anyway)
separate the the pweave deps from the release deps
decide if shipping jars with .whl still makes sense
- it was a nice attempt at nicety but unless you start the process from ipython instead of pyspark it is just dead weight.
pyrasterframes python files are in the sbt project structure
- Kind of the only benefit of this is to couple the production of assembly jar with the building of the .whl
- Not sure the weirdness of this justifies the setup

So that's a lot of moving stuff around potentially and it would be much cleaner to make that it's own PR.

Jan 19 '23 21:01 echeipesh