Efficient passthrough nodes

[hunse]

On some models (including some Nengo tests), our current code to remove passthrough nodes is failing. This is often because removing the passthrough node entirely turns a factored (m, 1) x (1, n) connection into a full (m, n) connection, and the latter takes much more memory on the chip.

I proposed replacing passthrough nodes with interneurons, since that would allow the connection to stay factored in such places. Here's the discussion from Slack:

@hunse

A point that Aaron raised before, and I think is a good one, is that we're not very explicit or transparent about our mapping from a Nengo network to the network we put (with interneurons) on the chip.

Passthrough nodes are often intentionally used at bottlenecks, where we have many ensembles feeding in (e.g. to different dimensions, like in an EnsembleArray) and we want to group them together.

What if rather than removing passthrough nodes, we replaced them with interneuron groups. Then we can keep the connections factored, and don't have the problem of an e.g. (m, 1) x (1, m) connection turning into (m, m) weights, as Dan pointed out above. It would mean adding a few more features so e.g. we have more control over the weights between interneurons and the post ensemble (right now, they're always the encoders). But I think in the long run, being more explicit about interneurons would help clean up our builder, make things clearer and more customizable for users, and help fix this problem.

@tcstewar

I definitely like that idea. There might be some intelligent thing about figuring out what passthrough nodes to keep (there are times when we just use them for organization, rather than for bottlenecks, such as in the circular convolution network), which will be an interesting challenge, but something in that direction seems to me like it'd be needed...

@hunse

Any thoughts on how to do this? One option would be to make it part of the splitter, and basically have one or more rules that look for situations where we need interneurons and puts in Ensembles to represent those interneurons. The advantage to this is that the output is also a Nengo network, so you could simulate it with the reference backend to see what the effects of interneurons are. I think this would help considerably with debugging, since right now we have issues (e.g. with integrator accuracy, with two inputs not adding like we expect) and we're not sure how much of them is interneurons versus other aspects of mapping to Loihi (like quantization).

I know that there has been some resistance to the splitter as it currently stands. I agree that it makes sense to have it more integrated into the build process, so that it's easier to deal with seeds, etc. However, I think there is a lot to be said for a build process that provides useful output at many stages, rather than just doing tons of manipulations with no way to tease them apart or test them individually. I think that the more individual stages we have, the easier it will be to write precise unit tests that test a particular part of the build process (within reason, of course. I'm not picturing a 20 stage process here.)

What I'm thinking about now is whether we could do the model fixing (i.e. turning all the distributions/random parameters, as well as setting seeds) in a similar way. Take in a Nengo network, and spit out one where all the seeds have been set on objects and all the distributions are replaced with arrays. Then this Nengo model can be handed off to any backend, and it will standardize that model fixing across all backends, and make it easier to write them.

So, thoughts?

(For more context, and details about the failing tests, see the merged PR that added the current passthrough node optimization: #64)

[tcstewar]

This feels to me to be the right way forward. I'm not quite sure the best way to do it, but I like the idea of having the splitter and the builder separate so that we can explore the effects of various ways of doing interneurons in normal nengo. So I think that'd require a) modifying the splitter so that it can produce interneurons, and b) modifying the builder such that if it notices interneurons it will put them on Loihi. I don't know how much that would require reorganizing the builder, but I think that'd be about right.

What I'm thinking about now is whether we could do the model fixing (i.e. turning all the distributions/random parameters, as well as setting seeds) in a similar way. Take in a Nengo network, and spit out one where all the seeds have been set on objects and all the distributions are replaced with arrays. Then this Nengo model can be handed off to any backend, and it will standardize that model fixing across all backends, and make it easier to write them.

Something like that feels right to me too. I think the biggest challenge with it is that we cannot do it by setting the seeds on objects and replacing the distributions, since we require the build process to not modify the original nengo.Network. But we could produce a new nengo.Network with new objects that have their seeds set and their distributions replaced, and a mapping dictionary that indicates which original object maps to which new object (so that the sim.data dictionary can behave as expected). That doesn't feel too awkward to me, and might be a good basic utility to offer as part of core nengo eventually....