Parse data from datetime-args and list of values

[danielhuppmann]

The https://github.com/OpenEnergyPlatform/oedatamodel uses a timeseries data format that has "time" identified as datetime-parseable args (start, end, period length) in one column (as list) and the values in another column (also as list).

See a stylized example below. I made some experiments on how to parse that timeseries data format and expand it to a pyam-compatible pandas.DataFrame.

Copied here for future reference.

import pandas as pd
import pyam

index_cols = ["variable", "unit"]

# the model/scenario/region columns are moved to the IamDataFrame-init to highlight the relevant parts
dt_dataframe = pd.DataFrame(
    [
        ["Primary Energy", "EJ/yr", ("2011-01-01", "2011-01-02", "D"), [1, 6.0]],
        ["Primary Energy|Coal", "EJ/yr", ("2011-01-01", "2011-01-02", "D"), [0.5, 3]],
    ],
    columns=index_cols + ["datetime", "series"],
)

def init_dataframe(row):
    return pd.DataFrame(data=row.series, columns=["value"], index=parse_daterange(row.datetime))

def parse_daterange(x):
    return pd.date_range(x[0], x[1], freq=x[2], name="time")

df = pd.concat([
    pd.concat([init_dataframe(row)], keys=[tuple(row[index_cols])], names=index_cols)
    for i, row in dt_dataframe.iterrows()
])

pyam.IamDataFrame(df, model="model_a", scenario="scen_a", region="World")

[danielhuppmann]

fyi @jh-RLI @christian-rli @henhuy