Add plugin operators
This is a proposal for a new behaviour that allows to change card operators from the final command such that:
load_dataset("card=cards.wikitq,table_serializer=serializers.table.markdown")
will be loading the wikitq with different table serializer, markdown serializer in this case.
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:
Comparison is base (
0e2d06e) 87.54% compared to head (05fbded) 87.66%. Report is 17 commits behind head on main.
Additional details and impacted files
@@ Coverage Diff @@
## main #574 +/- ##
==========================================
+ Coverage 87.54% 87.66% +0.12%
==========================================
Files 83 84 +1
Lines 7104 7143 +39
==========================================
+ Hits 6219 6262 +43
+ Misses 885 881 -4
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
If I understand there are two changes here.
- The ability to add constant fields to the datasets via the recipe (what happens If it overwrites an existing field?)
- An operator PlugIn operators that by defaults apply an operator to the stream, but allows changes the operator.
The Plugin operator is a bit tricky to understand (even the example is).
A naïve question.. : Since the card is a json file, a simpler, and perhaps better understood way (addressing @yoavkatz concern) would be to first load that json file into a json object, then change that json object, and use the edited version as the input to the recipe? Or the card must be instantiated via fetch_artifact? can that ad-hoc edited card be added to the catalog via add_card?