feature-store-api
feature-store-api copied to clipboard
[FSTORE-1285] Model Dependent Transformation Functions
This PR implements model dependent transformation function as per the design log : https://hopsworks.atlassian.net/wiki/spaces/FST/pages/388464653/Model+Dependent+Transformation+Proposal+-+version+2
Changes Performed
-
Added decorator called
hopsworks_udfthat should be used to defined transformation functions. -
Added possibility to obtain statistics within transformation functions.
-
Added possibility for one to many and many to many transformation functions.
-
Transformation functions executed as
pandas_udf'sin spark engine and regular pandas functions in python engine. -
Added ability to attach to a feature view transformation functions that are not explicitly saved to the backend using the transformations API.
-
Removed attaching transformation function to training dataset features and attached transformation functions to feature view directly. Hence removed the DTO class
transformation_functions_attached.
JIRA Issue: https://hopsworks.atlassian.net/browse/FSTORE-1285?atlOrigin=eyJpIjoiYjYwZDAyM2ZjZjNkNDU5MWEyNmRjN2MxZjNiZmNhZmEiLCJwIjoiaiJ9
Related PRs:
- https://github.com/logicalclocks/hopsworks-ee/pull/1775
- https://github.com/logicalclocks/hopsworks-chef/pull/1140
- https://github.com/logicalclocks/loadtest/pull/370
How Has This Been Tested?
- [x] Unit Tests
- [x] Integration Tests
- [x] Manual Tests on VM
Checklist For The Assigned Reviewer:
- [ ] Checked if merge conflicts with master exist
- [ ] Checked if stylechecks for Java and Python pass
- [ ] Checked if all docstrings were added and/or updated appropriately
- [ ] Ran spellcheck on docstring
- [ ] Checked if guides & concepts need to be updated
- [ ] Checked if naming conventions for parameters and variables were followed
- [ ] Checked if private methods are properly declared and used
- [ ] Checked if hard-to-understand areas of code are commented
- [ ] Checked if tests are effective
- [ ] Built and deployed changes on dev VM and tested manually
- [x] (Checked if all type annotations were added and/or updated appropriately)