feat: add support for numpy scalar types

[techytushar]

Signed-off-by: Tushar Mittal [email protected]

TL;DR

• Add support for numpy scaler types
• Update type matching in the type engine
• Add tests

Type

• [x] Bug Fix
• [x] Feature
• [ ] Plugin

Are all requirements met?

• [x] Code completed
• [x] Smoke tested
• [x] Unit tests added
• [ ] Code documentation added
• [ ] Any pending items have an associated Issue

Complete description

How did you fix the bug, make the feature etc. Link to any design docs etc

• Add support for numpy scaler types
• Update type matching in the type engine
• Add tests

Tracking Issue

Fixes: https://github.com/flyteorg/flyte/issues/2753

Follow-up issue

NA

[techytushar]

@samhita-alla Have created a new PR with tests

[codecov[bot]]

Codecov Report

Merging #1258 (f0ed0df) into master (5608f80) will decrease coverage by 0.03%. The diff coverage is 65.21%.

@@            Coverage Diff             @@
##           master    #1258      +/-   ##
==========================================
- Coverage   68.68%   68.64%   -0.04%     
==========================================
  Files         288      290       +2     
  Lines       26337    26418      +81     
  Branches     2119     2490     +371     
==========================================
+ Hits        18089    18135      +46     
- Misses       7770     7802      +32     
- Partials      478      481       +3     

Impacted Files	Coverage Δ
flytekit/types/numpy/__init__.py	0.00% <0.00%> (ø)
flytekit/types/schema/types.py	37.55% <ø> (-0.16%)	:arrow_down:
flytekit/types/numpy/scalar.py	37.83% <37.83%> (ø)
flytekit/core/type_engine.py	58.89% <100.00%> (ø)
tests/flytekit/unit/types/numpy/test_scalar.py	100.00% <100.00%> (ø)
flytekit/interfaces/random.py	20.00% <0.00%> (-5.00%)	:arrow_down:
flytekit/configuration/internal.py	16.43% <0.00%> (-2.03%)	:arrow_down:
flytekit/types/directory/types.py	54.16% <0.00%> (-0.46%)	:arrow_down:
flytekit/types/file/file.py	59.72% <0.00%> (-0.42%)	:arrow_down:
flytekit/models/security.py	12.82% <0.00%> (-0.34%)	:arrow_down:
... and 4 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[samhita-alla]

I'm not sure if we should opt for a targeted approach or ensure all the NumPy scalars are serializable. The latter case is handled by this PR which uses joblib for serialization. As for the targeted approach, we can use Primitive to build the Literal but it might not be able to handle all the NumPy scalars and every scalar type (at least the sub-groups) need to be handled separately.

[pingsutw]

yup, It's better to convert numpy scalars to the primitive type, just like we do in structured dataset

[samhita-alla]

@techytushar, can you modify your code to return primitive types rather than serializing data directly? This might take some time, so if you aren't able to get that done before Oct. 31st, I can still add the "hacktoberfest-accepted" label and it should count towards your hacktoberfest PRs.

[techytushar]

@samhita-alla sure, I can look into it, until then please add the label