NIR
NIR copied to clipboard
neuromorphs
Documentation of Primative inputs / outputs
It isn't always clear from the documentation of Primatives what the expected inputs / outputs are. Examples:
- Convolutions state the function is f (star) g but then list the parameters as not including either f or g, though I believe with external knowledge of convolutions that one of f and g are made up from the parameters (i.e. one of them is the convolution matrix) and one is the input. The output is then presumably the result of the operation?
- Models like LI / L / I / CubaLI which don't output spikes presumably output voltage or even just dimensionless values? Even if dimensionless, it would be good to document the variable that is output.
- Models which take "current input" like the Cuba models presumably expect Spike inputs where the LI and LIF models expect... possibly current inputs, or is it just dimensionless inputs? Again, even if dimensionless, it would be good to document the variables assumed to be inputs.
- Do models that end with a Threshold (Spike) element output Spikes only, or can they also output Voltage i.e. can a model output multiple things and is that something that has to be explicit somewhere?
You bring an excellent point. I think it's fair to say that we haven't done a good job of linking the continuous-time formulation (such as $f \star g$ for the convolution) to the implementation (such as the weight parameter in the convolution). To address your points one by one:
- The output is indeed the result of the operation and the formalization closely follows the parameters for the PyTorch convolutions.
- The parameters for the neurons have clear meanings (and SI units) for them. You can probably suss out some of them in this table over NIR primitives.
- Same can be said for the Cuba models, it should not be dimensionless.
- Models with a threshold can actually be considered as "higher-order nodes" that are composed by multiple smaller nodes. Below is an example for the (L)IF neuron. If you would like to read out the voltage, you can draw an arrow from the integrator node to the graph output. This is poorly documented, but it's possible. I'm happy to elaborate on this here, while also working on the docs to explain this a little better.
The output is indeed the result of the operation and the formalization closely follows the parameters for the PyTorch convolutions.
Ah OK, good to know about the PyTorch documents - those are actually very clear for convolutions. You could probably directly reference that I guess if it makes the docs easier to do.
The parameters for the neurons have clear meanings (and SI units) for them. You can probably suss out some of them in this table over NIR primitives.
I know what the parameter types are generally for the neurons in terms of units, though the exact unit can vary between implementations in general. For example, I know that PyNN uses nA (nano) for current, but I believe that NEST uses pA (pico) for the same parameter, which has resulted in confusion in the past! It is not critical which of these is chosen, but choosing one would be good if these are going to be more fully documented.
Models with a threshold can actually be considered as "higher-order nodes" that are composed by multiple smaller nodes. Below is an example for the (L)IF neuron
Yes, I saw that composition. A good example of my confusion is in the image I found in the discord chat:
This raises several questions related to the above:
- Is the Input here sending Spikes into the network? Are all Input nodes expected to send Spike input? For example, in the example network in the documents that does Input -> LIF -> Output, is the LIF then expected to receive Spikes or current?
- If the input is spikes, does the Affine do something with them to turn them into current? Does that mean an Affine always takes spikes and turns it into current?
- When CubaLIF outputs to the two Affine nodes, are these both receiving Spike output? If so, is it the same output, or would the CubaLIF be expected to output two different signals?
- Is the Affine that feeds back to the CubaLIF converting spikes to currents?
I don't need answers to all of these here; they are more examples of what I am unsure of, but I am happy to wait for the documentation to be updated first. Hopefully this gives an idea of what might be useful to include though.
The output is indeed the result of the operation and the formalization closely follows the parameters for the PyTorch convolutions.
Ah OK, good to know about the PyTorch documents - those are actually very clear for convolutions. You could probably directly reference that I guess if it makes the docs easier to do.
The parameters for the neurons have clear meanings (and SI units) for them. You can probably suss out some of them in this table over NIR primitives.
I know what the parameter types are generally for the neurons in terms of units, though the exact unit can vary between implementations in general. For example, I know that PyNN uses nA (nano) for current, but I believe that NEST uses pA (pico) for the same parameter, which has resulted in confusion in the past! It is not critical which of these is chosen, but choosing one would be good if these are going to be more fully documented.
Thank you for raising this, it's an excellent point! It will both help the understanding of the networks and the implementation for anyone working with NIR. Is something like this what you had in mind? https://github.com/neuromorphs/NIR/pull/166
This raises several questions related to the above:
* Is the Input here sending Spikes into the network? Are all Input nodes expected to send Spike input? For example, in the example network in the documents that does Input -> LIF -> Output, is the LIF then expected to receive Spikes or current?
In the NIR paper, we used a spiking input source, but NIR doesn't constrain the inputs. We have no strong opinions about what input types to use. That said, a major goal for NIR is to disambiguate the computation, so this is clearly not an optimal solution. We're actually working on adding some constraints in a separate data format. If you're interested to join that effort, please let me know. Your experience would be valuable.
* If the input is spikes, does the Affine do something with them to turn them into current? Does that mean an Affine always takes spikes and turns it into current?
Yet another point that hasn't been specified, I'm afraid. Do you have opinions how we can go about this? Right now, Affine can be coupled with any primitive, irrespective of the units.
* When CubaLIF outputs to the two Affine nodes, are these both receiving Spike output? If so, is it the same output, or would the CubaLIF be expected to output two different signals?
Yes, CubaLIF outputs spikes. And, by definition, every output from a primitive is the same, so both Affine nodes would receive the same spike-train input.
Does that address your questions?
Does that address your questions?
Yes generally thanks! Having done some work on the code, I am getting more of an idea how it all works.
Thank you for raising this, it's an excellent point! It will both help the understanding of the networks and the implementation for anyone working with NIR. Is something like this what you had in mind? https://github.com/neuromorphs/NIR/pull/166
Yes, that is useful for the parameters, but it is also useful to think about the inputs and outputs at each component and what values are actually flowing through the graph (see below).
In the NIR paper, we used a spiking input source, but NIR doesn't constrain the inputs. We have no strong opinions about what input types to use. That said, a major goal for NIR is to disambiguate the computation, so this is clearly not an optimal solution. We're actually working on adding some constraints in a separate data format. If you're interested to join that effort, please let me know. Your experience would be valuable.
In the SpiNNaker implementation so far I have different node implementations depending on whether the input is "spikes" or "matrix" data. In our case I am assuming spikes means the values are all 0 or 1 and in fact it then expects only the 1s to appear - so the input is essentially a list of positions in a matrix of values where the value is 1. I then assume matrix data means a value for every position which might be any real value (practically this is limited by the lack of floating point on SpiNNaker to the range of 32-bit signed S1615 fixed-point values, but the idea is the same).
From what I have seen so far, most components need to be able to accept both spike and matrix inputs depending on the output of the component before them, but equally most component produce only spikes or matrix outputs, not both. This means then that you likely only need to know the type of data at the Input node, after which the rest is probably determinable from knowing the output type of the component before it.
I would be happy to join in the effort to specify the data formats.
Yet another point that hasn't been specified, I'm afraid. Do you have opinions how we can go about this? Right now, Affine can be coupled with any primitive, irrespective of the units.
In some sense, the units of the output will simply depend on the units of the input in that case. In terms of the input and output data types though, I think an Affine always outputs a matrix rather than spikes. It can accept either spikes or matrix input though, since spikes are just values of 1 (assuming this is accepted as a definition). If you want spiking output, you would then couple it with a spike component; this might simply have a threshold of "just above 0" to mean that anything non-zero produces a spike (or you might pick a different value).
Yes, CubaLIF outputs spikes. And, by definition, every output from a primitive is the same, so both Affine nodes would receive the same spike-train input.
OK, this makes sense now I think, and especially if you add in the above regarding data types. In the diagram in this case, the CubaLIF receives two different Affine inputs, which is fine since the output of the Affines will have the same types.
From what I have seen so far, most components need to be able to accept both spike and matrix inputs depending on the output of the component before them, but equally most component produce only spikes or matrix outputs, not both. This means then that you likely only need to know the type of data at the Input node, after which the rest is probably determinable from knowing the output type of the component before it.
Yes, this sounds correct. There is a hypothetical possibility that one node receives input from both, say an Affine and LIF node. But I haven't seen that before (I'm not sure how much sense it would make) and maybe it's fine to ignore that for now.
graph LR;
Affine --> LIF2;
LIF1 --> LIF2;
Yet another point that hasn't been specified, I'm afraid. Do you have opinions how we can go about this? Right now, Affine can be coupled with any primitive, irrespective of the units.
In some sense, the units of the output will simply depend on the units of the input in that case. In terms of the input and output data types though, I think an Affine always outputs a matrix rather than spikes. It can accept either spikes or matrix input though, since spikes are just values of 1 (assuming this is accepted as a definition). If you want spiking output, you would then couple it with a spike component; this might simply have a threshold of "just above 0" to mean that anything non-zero produces a spike (or you might pick a different value).
Yes, exactly, spikes are interpreted as 1s. And coupling it with a threshold primitive (which should be called Spike as per #141) is a perfect way to go from continuous to spiking values, I think. Nice.
... this makes sense now I think, and especially if you add in the above regarding data types. In the diagram in this case, the CubaLIF receives two different Affine inputs, which is fine since the output of the Affines will have the same types.
Happy to clear things up a bit. Is there anything else that could/should be documented better? I'm afraid I sometimes can't see the forest for trees...
Yes, this sounds correct. There is a hypothetical possibility that one node receives input from both, say an Affine and LIF node. But I haven't seen that before (I'm not sure how much sense it would make) and maybe it's fine to ignore that for now.
Yes interesting; I hadn't really considered it, but I suppose it works in that if it is spikes, it is 1 values where there are spikes and 0 everywhere else from a NIR perspective. In the current SpiNNaker implementation, this wouldn't currently work obviously, though I could just say the spiking input isn't really spiking or something, and convert it dynamically to a matrix since the component will need to go through everything anyway. I'm sure I can work that out...
I think it is important still to identify that this is the input, if only to disambiguate what you are doing and avoiding the check for all 0s/1s in the input. A possible idea would be to make a SpikeInput separately to make it really clear; a normal Input would still let you specify all 1s / 0s but you wouldn't get any efficiency advantage from it basically. I understand if this is considered an implementation detail though!
Happy to clear things up a bit. Is there anything else that could/should be documented better? I'm afraid I sometimes can't see the forest for trees...
This is possibly a different issue, but I noticed that CubaLI(F) is documented as being LI -> Linear -> LI(F), however in the NIR source code I can see that it expects the input and output to have the same shape, both being the shape of the parameters essentially. However Linear in the NIR source says that the weights have to be 2D and then it is doing a matrix multiplication between input and weights to get outputs, which suggests that the outputs do not necessarily have the same shape as the inputs unless the weight matrix is specifically sized.
I am then wondering if this really is LI -> Linear -> LI(F) or really just LI -> LI(F). This would then take the input values and add them to a decaying input buffer (the current shaping) and then the output of the current shaping would be passed to the neuron as its input each time step (in discrete mode ;). This might then line up better if you wanted two different synapse types (often split into excitatory and inhibitory with two different decay factors) with something like two separate LIs feeding into a single LIF.
I can see that Linear would be useful to receive from a set of neurons which doesn't have the same shape as the output (i.e. different population sizes), and this then provides the weights to apply on each incoming spike, and which neurons in the output are targeted (where 0 weight would imply not targeted - this could be made into a sparse matrix in implementation to avoid the 0s). Then you get a NIR graph of Spike (which might be output of a previous LIF) -> Linear -> LI -> LIF to give what might be considered a "standard" spiking neuron model.
That seems logical to me, but I could easily be missing what you meant here...