lux
lux copied to clipboard
[WIP] No subclasses refactor v2
The idea of this PR is to:
- Remove the classes
LuxDataFrame
,LuxSeries
, etc - Entirely avoid strong modifications to the pandas library such as overriding
pandas.DataFrame
class (and all other internal references to it) with their Lux counterpart e.g.:
pandas.DataFrame = LuxDataFrame
Current behaviour can be surprising to the user and it makes supporting new versions harder as you might have to deal with more edge cases because the impact of the modifications is so large. Instead the idea is to some lightweight monkey patching of fiew core methods (which mostly do bookkeeping) and move the rest to out of the DataFrame class to avoid possible conflicts. A lot of discussion and possible iteration is needed. These are the current efforts:
Ongoing changes
-
LuxDataFrame
as a class is removed, instead key methods of interest formpandas.DataFrame
are patched. - The type
LuxDataFrame
is now aProtocol
that is there only for possible validation and type inference purposes. - Core methods are patched using a new
@patch(cls)
decorator, this replaces the method on the class but the original method is still available as._super_{method_name}
, e.g:
@patch(DataFrame)
def _set_axis(self, axis, labels):
self._super_set_axis(axis, labels)
...
- To reduce the surface area in which Lux impacts Pandas, a single
.lux
property that includes all lux extension methods is created. Its very similar to the existing.str
and.dt
functionality already present in pandas, here is an example usage:
df.lux.save_as_html(...)
Hey @dorisjlee, I created this new PR which aims at a cleaner refactor in that it does as few modifications possible to the existing structure e.g. doesn't mess with imports. It has the following differences with the previous:
-
LuxDataFrameMethods
andLuxSeriesMethods
now inherit fromLuxMethods
which contains common fields and a utility class method calledfrom_lux_object
that facilitates copying fields. - Patching is performed in the
lux/patch/{frame, series}.py
scripts.
This time I am starting bottom up in that I created a new test file called tests/test_basic.py
that I am using as a playground to tryout very simple stuff. Currently it only has 1 test that sets an intent to a dataframe and checks that its propagated to a series. Feel free to play around with this I think the basic structure is there.
Next Steps
- Change all
{dataframe}.{lux_attribute}
calls to{dataframe}.lux.{lux_attribute}
, specifically code that has the formldf.{lux_attribute}
can be replaced in mass toldf.lux.{lux_attribute}
for all known attributes. - For all
LuxMethods
code, change{lux_methods}.{dataframe_attribute}
to{lux_methods}.df.{dataframe_attribute}
.
Thanks @cgarciae! This makes sense, we can slowly add the tests back in to test a subset of the functionality one at a time.
Hey @dorisjlee! I've been fixing tests and solving bugs.
For tomorrow I'd like to discuss is the implementation of groupby
, logic has to be adjusted so feedback would be appreciated.
@dorisjlee In this test, after df.lux.intent = ["Attrition"]
it results in df.current_vis
being non-empty, new_df
copies current_vis
and on the last assertion its still not empty. At which point should current_vis
be emptied?
def test_metadata_propogate_invalid_intent():
df = pd.read_csv(
"https://raw.githubusercontent.com/lux-org/lux-datasets/master/data/employee.csv")
df.lux.intent = ["Attrition"]
new_df = df.groupby("BusinessTravel").mean()
assert new_df.lux.intent[0].attribute == "Attrition", "User-specified intent is retained"
assert new_df.lux._inferred_intent == [], "Invalid inferred intent is cleared"
new_df._ipython_display_()
assert new_df.lux.current_vis == [] # <<--- not being emptied
Just as a recap, I ran all the tests an dcurrently these are the results:
125 failed, 103 passed
@dorisjlee In this test, after
df.lux.intent = ["Attrition"]
it results indf.current_vis
being non-empty,new_df
copiescurrent_vis
and on the last assertion its still not empty. At which point shouldcurrent_vis
be emptied?def test_metadata_propogate_invalid_intent(): df = pd.read_csv( "https://raw.githubusercontent.com/lux-org/lux-datasets/master/data/employee.csv") df.lux.intent = ["Attrition"] new_df = df.groupby("BusinessTravel").mean() assert new_df.lux.intent[0].attribute == "Attrition", "User-specified intent is retained" assert new_df.lux._inferred_intent == [], "Invalid inferred intent is cleared" new_df._ipython_display_() assert new_df.lux.current_vis == [] # <<--- not being emptied
Hey @cgarciae, The current_vis is cleared when we call clear_intent, which calls set_intent. Inside the set_intent, when we do _parse_validate_compile_intent
the current_vis gets recomputed based on the now empty intent, and so becomes empty.