Optimize the processing to make it more reliable and easier to test

[samsmithnz]

• [ ] Split the nightly processing into two steps
  • [ ] Add the processing into a queue on a schedule
  • [ ] For each item in the queue, spawn a function and process the data