feature-store-api icon indicating copy to clipboard operation
feature-store-api copied to clipboard

[FSTORE-1285] Model Dependent Transformation Functions

Open manu-sj opened this issue 1 year ago • 0 comments

This PR implements model dependent transformation function as per the design log : https://hopsworks.atlassian.net/wiki/spaces/FST/pages/388464653/Model+Dependent+Transformation+Proposal+-+version+2

Changes Performed

  • Added decorator called hopsworks_udf that should be used to defined transformation functions.

  • Added possibility to obtain statistics within transformation functions.

  • Added possibility for one to many and many to many transformation functions.

  • Transformation functions executed as pandas_udf's in spark engine and regular pandas functions in python engine.

  • Added ability to attach to a feature view transformation functions that are not explicitly saved to the backend using the transformations API.

  • Removed attaching transformation function to training dataset features and attached transformation functions to feature view directly. Hence removed the DTO class transformation_functions_attached.

JIRA Issue: https://hopsworks.atlassian.net/browse/FSTORE-1285?atlOrigin=eyJpIjoiYjYwZDAyM2ZjZjNkNDU5MWEyNmRjN2MxZjNiZmNhZmEiLCJwIjoiaiJ9

Related PRs:

  • https://github.com/logicalclocks/hopsworks-ee/pull/1775
  • https://github.com/logicalclocks/hopsworks-chef/pull/1140
  • https://github.com/logicalclocks/loadtest/pull/370

How Has This Been Tested?

  • [x] Unit Tests
  • [x] Integration Tests
  • [x] Manual Tests on VM

Checklist For The Assigned Reviewer:

- [ ] Checked if merge conflicts with master exist
- [ ] Checked if stylechecks for Java and Python pass
- [ ] Checked if all docstrings were added and/or updated appropriately
- [ ] Ran spellcheck on docstring
- [ ] Checked if guides & concepts need to be updated
- [ ] Checked if naming conventions for parameters and variables were followed
- [ ] Checked if private methods are properly declared and used
- [ ] Checked if hard-to-understand areas of code are commented
- [ ] Checked if tests are effective
- [ ] Built and deployed changes on dev VM and tested manually
- [x] (Checked if all type annotations were added and/or updated appropriately)

manu-sj avatar Apr 29 '24 14:04 manu-sj