sPyNNaker
sPyNNaker copied to clipboard
SpiNNakerManchester
Too much delay causes delay packets to be be dropped.
Script: import spynnaker8 as sim
n_neurons = 76 # 75 works 76 fails sim.setup(timestep=1.0)
pop_1 = sim.Population(n_neurons, sim.IF_curr_exp(), label="pop_1") input1 = sim.Population( n_neurons, sim.SpikeSourceArray(spike_times=range(0, 1000, 100)), label="input")
input2 = sim.Population( n_neurons, sim.SpikeSourceArray(spike_times=range(0, 1000, 100)), label="input") input3 = sim.Population( n_neurons, sim.SpikeSourceArray(spike_times=range(0, 1000, 100)), label="input") input4 = sim.Population( n_neurons, sim.SpikeSourceArray(spike_times=range(0, 1000, 100)), label="input") input5 = sim.Population( n_neurons, sim.SpikeSourceArray(spike_times=range(0, 1000, 100)), label="input") sim.Projection( input1, pop_1, sim.OneToOneConnector(), synapse_type=sim.StaticSynapse(weight=5, delay=1)) sim.Projection( input1, pop_1, sim.OneToOneConnector(), synapse_type=sim.StaticSynapse(weight=5, delay=20)) sim.Projection( input1, pop_1, sim.OneToOneConnector(), synapse_type=sim.StaticSynapse(weight=5, delay=40)) sim.Projection( input1, pop_1, sim.OneToOneConnector(), synapse_type=sim.StaticSynapse(weight=5, delay=60)) sim.Projection( input1, pop_1, sim.OneToOneConnector(), synapse_type=sim.StaticSynapse(weight=5, delay=80)) pop_1.record(["spikes"]) sim.run(1200)
spikes = pop_1.spinnaker_get_data("spikes") sim.end() print(len(spikes)) print(10 * 5 * n_neurons)
Found when moving https://github.com/SpiNNakerManchester/sPyNNaker8/pull/429
If too many packets arrive at the core at the same time, and they can't all be processed within a single timestep, it will throw those away. This is by design. Having different delays will help a bit because the packets are then captured in delay extensions which spread these packets out more.
The issue is NOT the just the number of packets that arrive at the core at the same time.
Changing it so that each projection comes from a different but identically firing SpikeSourceArray avoid the issue.
So it is an overload on the one DelayVertex that is causing the packets to be lost.
Do you have an example of the error being generated? This might indicate which bit is losing the packets to make it more clear...
It reports:
20 packets from pop_1:0:75 on 1, 1, 4 were dropped from the input buffer, because they arrived too late to be processed in a given time step. Try increasing the time_scale_factor located within the .spynnaker.cfg file or in the pynn.setup() method.
A maximum of 2 background tasks were queued on pop_1:0:75 on 1, 1, 4. Try increasing the time_scale_factor located within the .spynnaker.cfg file or in the pynn.setup() method.
Then it reports only 3769 spikes instead of the expected 3800
Changing it so that each project comes from a different but identical input population and instead it reports:
The input buffer for pop_1:0:75 on 1, 1, 8 lost packets on 759 occasions. This is often a sign that the system is running too quickly for the number of neurons per core. Please increase the timer_tic or time_scale_factor or decrease the number of neurons per core.
BUt it has all 3800 spikes.
The first case where it loses packets is expected if the receiver core is overloaded, and from the messages, it is. It doesn't seem related to delay extension overload here either.
The second case where it loses packets and then creates all spikes is quite odd! That may need more investigation...
The first case where it loses packets is expected if the receiver core is overloaded, and from the messages, it is. It doesn't seem related to delay extension overload here either.
The second case where it loses packets and then creates all spikes is quite odd! That may need more investigation...
After some discussion on Skype this morning I believe we have concluded that both these cases are behaving as we would expect them to with the current code, and that the drops in the input buffer in the case where there are separate input populations are of packets that don't matter with regard to the correct functioning of the code.
It's worth noting here that in this particular example (with separate input populations), increasing the default value of incoming_spike_buffer_size (from 256 to 512) in the config file stops the warnings / overflows etc.
Assigning to @rowleya as may be an interesting test script for his no longer dropping packets idea.