Support autogeneration of RES data/diagram

[gidden]

Request

In many contexts, understanding the underlying reference energy system (RES) structure of a given scenario is needed.

Given the RES structure in a dataframe, we can then use this directly and generate visuals to support debugging and other concerns.

Example

Generally, this can be done by intelligently interrogating input and output parameters for a scenario.

Below is an example of two functions that could help

def find_inp_link(s, tec, filters):
    filters = filters.copy()
    filters['technology'] = tec
    df = s.par('input', filters=filters)
    df = df[['commodity', 'level']].drop_duplicates()
    return df

def find_outp_link(s, inp_link, filters):
    filters = filters.copy()
    filters = dict(**filters, **inp_link)
    df = s.par('output', filters=filters)
    df = df[['technology']].drop_duplicates()
    return df

Consider the example of LNG_exp in current versions of the global model

filters = {'node_loc': 'R15_NAM', 'year_act': 2020}
tec = 'LNG_exp'
inp_link = find_inp_link(s, tec, filters).iloc[0].to_dict()
>>> {'commodity': 'LNG', 'level': 'primary'}

then

find_outp_link(s1, inp_link, filters).iloc[0].to_dict()
>>> {'technology': 'LNG_prod'}

and so on

[khaeru]

Graphviz and NetworkX (including via graphviz) provide useful features. The Python graphviz package is what's used under the hood by dask (https://docs.dask.org/en/latest/graphviz.html#visualize-task-graphs), thus also by genno and {ixmp,message_ix}.reporting.

To produce the kind of visualization that we often use, the code would need to, among other things:

• Receive configuration setting the order of items in the level set. (The ixmp backend does not make strong promises about the ordering of set item; I don't think we can count on the items being in order in our scenarios; and in any case the notion of order is ambiguous once we have some levels pertaining to energy, and others to water, materials, etc.)
• Group the (commodity, level) nodes into multiple sets based on (level).
• Group the (technology) nodes into multiple sets based on which input/output levels they use. E.g. (technology) nodes that all input from (commodity, "primary") and output to (commodity, "secondary") would need to be grouped.
• Use something like splines=ortho to make sure lines between (commodity, level) -> (technology) -> (commodity, level) are xy-aligned.

[behnam-zakeri]

Thanks @gidden and @khaeru for the thoughts. I believe what one of the collaborators developed under https://github.com/iiasa/message_data/pull/164 can inform this discussion or even be transferred to message_ix. Using this feature, one can generate the Sankey diagram of a scenario, using arrows with different sizes indicating the amount of commodity flow as well. This has been useful in debugging.

[khaeru]

Discussion in a Slack thread with a particular request for this feature from one participant in this week's MESSAGEix training workshop.

CC also @SiddharthJoshi-Git