cognite-sdk-python icon indicating copy to clipboard operation
cognite-sdk-python copied to clipboard

Make pandas conversion distribute over iterables

Open tommyod opened this issue 5 years ago • 3 comments

Is your feature request related to a problem? Please describe.

I was a bit surprised that

pd.concat((asset.to_pandas().T for asset in assets)) != assets.to_pandas()

since I would expect that the conversion of a collection should equal the collection of converted elements. For instance, I would expect that the following property would hold for all pandas.Series:

ser = pd.Series([1, 2, 4, 5])
ser.map(str) == pd.Series(map(str, ser))

The inconsistency is due to one of them including metadata, while the other one does not.

assets = client.assets.list(limit=99999)
        
assets[0:1].to_pandas() # Single row, no metadata
assets[0].to_pandas().T # Single row, with metadata

Describe the solution you'd like

  1. Be consistent.
  2. Perhaps add an expand_metadata argument to the to_pandas method allowing users to control behavior.

tommyod avatar Aug 29 '19 06:08 tommyod

CogniteResource.to_pandas() has an expand parameter, which takes a list of keys. Given that the corresponding values are dicts, it will expand the key-value pairs of each dict into their own rows. expand defaults to ["metadata"], but can be set to an empty list if you want to keep the metadata column.

erlendvollset avatar Aug 29 '19 09:08 erlendvollset

Didn't see that!

I would still argue that it should distribute. By the function signature there's no indication (1) what the argument means precisely, (2) how it can be used, or (3) what the default is. The default could for instance be expand = ("metadata",) to avoid mutable parameters, or expand=None with an explanation in the docstring.

help(assets[0].to_pandas)
Help on method to_pandas in module cognite.client.data_classes._base:

to_pandas(expand:List[str]=None, ignore:List[str]=None) method of cognite.client.data_classes.assets.Asset instance
    Convert the instance into a pandas DataFrame.
    
    Returns:
        pandas.DataFrame: The dataframe.

tommyod avatar Aug 29 '19 10:08 tommyod

Describe the solution you'd like

  1. Perhaps add an expand_metadata argument to the to_pandas method allowing users to control behavior.

Coming in the next major version: #1453

haakonvt avatar Oct 30 '23 11:10 haakonvt