Tooling Nengo GUI as a Research Tool

[Seanny123]

The Nengo GUI is already a pretty good prototyping tool. However, for it to be useful for research past prototyping and moving into analysis/publication, it needs a few new features:

• Run for a specific amount of time
• Run multiple times with different parameters On the data processing side:
• Save non-visualised data to a file
• Temporarily hide plots (because animating plots is expensive)

This brings us to the big question of whether probes should be connected to the Nengo GUI and if so, how. Assuming all the updates to the Probe API have been completed (being able to define a file/output, define how much data is saved) should they be placed at the end of the code? Should they be in their own file?

[studywolf]

hmmm so when I need to do these things I typically just take the code outside of nengo gui and into straight nengo...i could imagine that there's reasons it would be useful to be able to do this inside the gui still but i'm having trouble coming up with any. I basically use the GUI for prototyping and then once i'm doing things like multiple runs i'm out of the prototyping stage and just run it in nengo.

[tcstewar]

Yeah, I also tend to go outside the gui for things like that... I'd say the first step is to have example code of doing the various things you're listing there.

I'm a bit worried about bringing stuff like that into the gui. Mostly because there are so many different options of what people want to do with the running and analysis.

I've been putting together some code that sorta supports this in https://github.com/ctn-waterloo/ctn_benchmarks It's got an interesting combination of being able to run things from the command line, pop up nengo_gui if needed, storing raw data and plots from individual runs, and plotting aggregate data. I'm also using doit http://pydoit.org/ for some of the automation.

[celiasmith]

The reason this came up in discussion is because of a difference between how new users will encounter nengo, and how it's grown up in the lab. In the lab, nengo was first; but I suspect for most new users, nengo_gui will be first. We were thinking that the gui would be more appealing if people could do all the steps involved in, e.g. writing a paper, within the gui itself... otherwise they have to learn two different things (ok, not that different, but you know what i mean), for what is probably a main use case. So we were thinking that to keep the barrier to deployment (i.e. use in everyday workflow for people) low, it might be good to have most things in the gui.

Of course, if it's not technically possible, that might be a deal breaker. However, that wasn't obvious in our brief discussions, so we thought it would be worth discussing further.

[studywolf]

hm my concern would be the gui becoming bloated, these kinds of operations are natural when you move over to coding just by adding for loops around the script you built in the gui. adding a couple of examples about how to do this, or even some helper functions? might be a more cleaner solution

[tcstewar]

Definitely worth discussing further, and I strongly believe that we want to have some sort of natural way of supporting/assisting that transition from prototyping (which nengo_gui is currently great at) to running your model multiple times with different parameters and gathering data. But I think we first need to have helper tools for doing this sort of thing outside of nengo_gui, and then as we find particular things that are extremely common that might fight naturally with nengo_gui, we can look at merging them in.

[celiasmith]

Sure, we don't necessarily want the GUI to be used for everything, but it would be nice, as to @studywolf's suggestion, to be using as much the same model as possible when you put it in a for loop or something. Similarly for @tcstewar's comment that we want to move in a natural way from prototyping to multiple runs. This is why we were talking about how it might be nice if probes could potentially be used in the gui directly (ie merging the two ways of collecting data). This definitely needs a lot more discussion, but the starting 'vision' is something like:

1. have a section of code that has all the probes
2. have probes be both displayable and deletable from the gui
3. let probes write to files (for later processing); as well as use up some memory if they're being displayed

If users get used to probes and the probe system in the gui, then I think the move to pure scripting would be more painless (and for simple things they might not have to make the move, if resistant).

[tbekolay]

We discussed this at the dev meeting today; some things that were brought up:

1. It would be difficult to cover all possible experimental use cases, yet it's sort of necessary to cover almost all of them if we want people to primarily run experiments through the GUI. If 10% of the time you need to resort to scripts or notebooks, then we might as well start teaching people about that transition sooner rather than trying to keep them in the GUI slightly longer.
2. One good thing to do would be to write up some explicit tutorials guiding people through the transition from prototyping networks in the GUI to running simple experiments in the notebook to running legit experiments through something like pytry.
3. There are some things we could that aren't all the use cases, but would still be useful. @tcstewar brought up the idea of running n copies of the same network and having the plots either show all n values overlaid, or with confidence intervals, or something else cool like that.
4. I brought up the idea of adding a "record" button so that you could run a simulation for some time and spit the data out to a file for later analysis. This would only save you the step of figuring out what probes you need and saving the data from sim.data to a file, but that is a kind of tedious step so eliminating it could be beneficial even though people will still need to write their own analysis scripts.