spark-deep-learning icon indicating copy to clipboard operation
spark-deep-learning copied to clipboard

[WIP - do not merge!] Move sparkdl utilities for conversion between numpy arrays and image schema to ImageSchema

Open tomasatdatabricks opened this issue 7 years ago • 3 comments

[WIP] Preparation for moving stuff to Spark.

Moved utilities for image schema <=> numpy array conversion to (copy pasted from spark 2.3) Image schema code.

  1. Extended ImageSchema scala code with support/information for all OpenCv modes
  2. python toNDArray and toImage utilities extended to work with all supported data types.
  3. [minor] sparkdl toImage function included batch size stripping - had to make a separate call for that

tomasatdatabricks avatar Dec 28 '17 23:12 tomasatdatabricks

Codecov Report

Merging #90 into master will increase coverage by 1.43%. The diff coverage is 77.46%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #90      +/-   ##
==========================================
+ Coverage   82.49%   83.92%   +1.43%     
==========================================
  Files          33       33              
  Lines        1879     1866      -13     
  Branches       35       39       +4     
==========================================
+ Hits         1550     1566      +16     
+ Misses        329      300      -29
Impacted Files Coverage Δ
python/sparkdl/udf/keras_image_model.py 75.6% <0%> (+1.8%) :arrow_up:
...main/scala/com/databricks/sparkdl/ImageUtils.scala 90.9% <100%> (ø) :arrow_up:
...n/sparkdl/estimators/keras_image_file_estimator.py 74.35% <100%> (ø) :arrow_up:
python/sparkdl/transformers/tf_image.py 94.06% <33.33%> (-0.05%) :arrow_down:
python/sparkdl/param/image_params.py 81.81% <50%> (+6.14%) :arrow_up:
.../scala/org/apache/spark/ml/image/ImageSchema.scala 77.94% <75%> (-1.1%) :arrow_down:
python/sparkdl/image/imageIO.py 73.33% <81.25%> (-4.77%) :arrow_down:
python/sparkdl/image/image.py 78.82% <82.35%> (+40.58%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update aeff9c9...ce17c43. Read the comment docs.

codecov-io avatar Dec 29 '17 19:12 codecov-io

@sueann Yes I agree, I would merge spark version first and merge this one only after spark 2.4 is released. I made the PR here mostly because that's what we need the changes for, so it can be reviewed in context, also to run tests.

I'll mark it WIP.

tomasatdatabricks avatar Jan 09 '18 21:01 tomasatdatabricks

ah ok got it. thanks!

sueann avatar Jan 09 '18 21:01 sueann