Jeff Reback
Jeff Reback
are you folks aware of: http://pandas.pydata.org/pandas-docs/stable/development/extending.html#extension-types these are in-pandas support for arbitrary data types defined (internally or externally). These in fact were developed to allow for nested data types. Using...
@HyukjinKwon from your comments: https://github.com/databricks/koalas/issues/420#issuecomment-498561529 pandas provides full fledged Extension Dtypes (which are used in for example Datetime w/tz, Intervals, Categorical, nullable Ints for example), that simply are 1-d array...
pyarrow is soon going to be able to fully replicate a round-trip of a DataFrame with extension types (and serialize to parquet), see: https://issues.apache.org/jira/browse/ARROW-2428, https://issues.apache.org/jira/browse/ARROW-5271, https://issues.apache.org/jira/browse/ARROW-3829
this prob should be fine as null key grouping is something we don't typically see
would for sure be +1 here maybe could even do this w/o breaking the world (eg back compatible)
@westurner blaze, DyND, odo, datashape are long abandonware :-
@cpcloud this is a really aggressive change and will likey blow up the world. Can you explain the rationale
i read the text above and it doesn't answer the question of why here in any event please don't merge these u til we have a discussion
well see @icexelloss yhis seems like a huge amount of churn for a very small gain so am -1 at this point
https://github.com/zbrookle/sql_to_ibis this already exists and supports most of the syntax