x-axis in plotting.lines too short

[steipatr]

Problem Description

When using plotting.lines to visualize experiment outcomes, the range of the plot x-axis is the duration of the first model run. If model runs don't have the same length (e.g. NetLogo model with endogenous stopping condition), any runs longer than the first will not be plotted beyond the first run's last time step. This is unexpected, I would expect the entirety of every run to be plotted.

Relevant Lines of Code

The axis limits are set in plotting.py/plot_lines_with_envelopes:

ax.set_xlim(left=time[0], right=time[-1])

Possible Solutions

I see two avenues for ameliorating this issue:

A) assign the correct x-axis limits directly in the referenced plotting function:

ax.set_xlim(left = outcomes['TIME'].min(), right = outcomes['TIME'].max())

B) change plotting_utils.py/determine_time_dimension, which currently checks the first run to infer the time horizon of all model runs:

time = outcomes['TIME']
time = time[0, :]

It could instead find the run with the longest duration and reference that instead:

time = outcomes['TIME']
longest = np.where(outcomes['TIME'] == outcomes['TIME'].max()) #get index of longest run
time = time[longest, :]

Hope this description makes sense, otherwise I am happy to put together an example.

[quaquel]

The problem is a bit more involved than just the lines plotting function. A central assumption throughout the workbench is that all experiments are equal in length (see e.g. how results are stored). Yes, we can fix this for one plotting function, but that is a bit of a hack in my view.

[steipatr]

I can see that. Perhaps it is more principled to not edit the experimental results internally, but to make it an explicit postprocessing step (like averaging out replications):

#postprocessing
#flatten replications
outcomes_2D = {key:np.mean(outcomes[key],axis=1) for key in outcomes.keys()}

#overwrite time output for plotting
outcomes_2D['TIME'] = np.tile(range(outcomes_2D['TIME'].shape[1] + 1),(outcomes_2D['TIME'].shape[0], 1))

results_2D = (experiments.copy(), outcomes_2D)

[quaquel]

are you generating your results using the workbench?

[steipatr]

Yes, why? PDFs of Jupyter Lab books attached.

Data Generation.pdf Data Import.pdf

It's curious that you ask, because I am trying to write a function that imports NetLogo results files into the workbench. But I don't think that should have any bearing on this issue, unless I am getting things mixed up somewhere.

[quaquel]

the reason I am asking is that I would expect an error message somewhere during the experimentation if the length of the result is different from the first one. In particular, if a subsequent result is longer than the first result, I would expect an error. It seems that you don't get this error.

To be clear, this is separate from using the workbench for analysing/visualizing data generated in some other manner.

as an aside, have you looked at format strings in python? Your save_results can be quite a bit cleaner in this way:

save_results(results, f'./{start_time}-eperiments{nr_experiments}x{model_replications}.tar.gz'

[steipatr]

The outcome arrays have the shape (experiments, replications, longest_run + 1), shorter runs are padded to fill the third array dimension. So it looks something like:

#2 experiments, 2 replications, 10 steps in longest run
array([[[0, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2], #model ran for two time steps
[0, 1, 2, 3, 4, 4, 4, 4, 4, 4, 4]], #model ran for four time steps
[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 9], #model ran for nine time steps
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]] #model ran for ten time steps
])

Is this padding intentional?

Re: string formatting, yes I know it's a thing and that string concatenation is not clean. I copy-paste together most of my EMA code, which doesn't help with staying on top of proper coding practices. Maybe I should make template notebooks for common EMA tasks. Will keep it in mind, thanks!

[quaquel]

so this padding happens within the workbench?

[steipatr]

Did some testing,I think the padding comes from run_experiment in the NetLogo connector, which simply runs the go command for the number of steps passed from model.run_length:

c_start = "repeat {} [".format(self.run_length)
c_close = "go ]"
c_middle = " ".join(commands)
command = " ".join((c_start, c_middle, c_close))
_logger.debug(command)
self.netlogo.command(command)

So even if the model run reaches its endogenous stopping condition in less than run_length steps, further time steps are attempted. Since the stopping condition is already fulfilled, the output and time step variables don't change, but are recorded again in the results. That creates the "padding", I believe. Conversely, if model.run_length is set very low, then all model runs are cut off before reaching their endogenous stopping condition.

For completeness: if model.run_length is not explicitly set, the workbench returns an error:

EMAError: exception in run_model
Caused by: NetLogoException: Nothing named NONE has been defined.

[quaquel]

That makes sense. If you want to use the stopping condition, you probably have to implement a variant of the NetLogoModel class which either implements the stopping condition on the python side, or in some other way checks or receives a signal when the stopping conditions has been reached.

of course, this then will create an error in the callback because of varying lengths of returns.

[steipatr]

I think the current functionality is perfectly sufficient from an exploratory modelling standpoint. I imagine most systems being studied with the workbench are non-terminating, so using model.run_length to control that is reasonable. I didn't consider that it might matter whether my test model was terminating or not.

I've also now realized that an easy solution to this issue is to delete the 'TIME' output, which forces the determine_time_dimension function to infer what the model run length might be, based on the shape of outcomes. That seems to generate the expected lines plots.

In summary: the output time series of NetLogo models with endogenous stopping conditions are padded to a length set by model.run_length, if the model self-terminates before the run_length is reached. This includes the ticks output used to record model time. This may in turn cause issues when setting the x-axis limits for plotting.lines. To plot the entirety of all model runs, some post-processing of the results may be required, either by modifying or deleting the time output.

[quaquel]

which raises the issue that is very high on my wish list: proper handling of time series data. Basically, have the possibility of storing results with a meaningful index (rather than storing the index separately using TIME as name).

[steipatr]

Would that enable new functionality? Or just be a cleaner implementation of the existing functionality?

[quaquel]

primarily cleaner but also might enable additional functionality down the line

[steipatr]

It's certainly intriguing. Pandas seems to have extensive time series capability. Might also simplify clustering and pattern analysis in outputs.

As for the originally raised issue, I would be fine with closing it. It's been a very informative discussion, thank you!

[quaquel]

I am indeed thinking of using pandas for this. Basically, time series outcomes should become a series with a meaningful index. Probably requires a keyword argument on TimeSeriesOutcome to specify what to use as index. Either a string,refering to a variable from the model, or something that can be used by Pandas as index.