spark-deep-learning
spark-deep-learning copied to clipboard
[WIP - do not merge!] Move sparkdl utilities for conversion between numpy arrays and image schema to ImageSchema
[WIP] Preparation for moving stuff to Spark.
Moved utilities for image schema <=> numpy array conversion to (copy pasted from spark 2.3) Image schema code.
- Extended ImageSchema scala code with support/information for all OpenCv modes
- python toNDArray and toImage utilities extended to work with all supported data types.
- [minor] sparkdl toImage function included batch size stripping - had to make a separate call for that
Codecov Report
Merging #90 into master will increase coverage by
1.43%
. The diff coverage is77.46%
.
@@ Coverage Diff @@
## master #90 +/- ##
==========================================
+ Coverage 82.49% 83.92% +1.43%
==========================================
Files 33 33
Lines 1879 1866 -13
Branches 35 39 +4
==========================================
+ Hits 1550 1566 +16
+ Misses 329 300 -29
Impacted Files | Coverage Δ | |
---|---|---|
python/sparkdl/udf/keras_image_model.py | 75.6% <0%> (+1.8%) |
:arrow_up: |
...main/scala/com/databricks/sparkdl/ImageUtils.scala | 90.9% <100%> (ø) |
:arrow_up: |
...n/sparkdl/estimators/keras_image_file_estimator.py | 74.35% <100%> (ø) |
:arrow_up: |
python/sparkdl/transformers/tf_image.py | 94.06% <33.33%> (-0.05%) |
:arrow_down: |
python/sparkdl/param/image_params.py | 81.81% <50%> (+6.14%) |
:arrow_up: |
.../scala/org/apache/spark/ml/image/ImageSchema.scala | 77.94% <75%> (-1.1%) |
:arrow_down: |
python/sparkdl/image/imageIO.py | 73.33% <81.25%> (-4.77%) |
:arrow_down: |
python/sparkdl/image/image.py | 78.82% <82.35%> (+40.58%) |
:arrow_up: |
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact)
,ø = not affected
,? = missing data
Powered by Codecov. Last update aeff9c9...ce17c43. Read the comment docs.
@sueann Yes I agree, I would merge spark version first and merge this one only after spark 2.4 is released. I made the PR here mostly because that's what we need the changes for, so it can be reviewed in context, also to run tests.
I'll mark it WIP.
ah ok got it. thanks!