buttery-eel icon indicating copy to clipboard operation
buttery-eel copied to clipboard

Data feeding restructure

Open Psy-Fer opened this issue 10 months ago • 1 comments

with the changes in how dorado-server processes batches, it is causing performance issues.

I need to change the current static pipeline of

1. get batch
2. mutate batch into compatible structure
3. send that to basecaller
4. wait for it to be basecalled
5. get the reads back from the basecaller and process them
6. send to the writter queue.

to something like

1. get batch
2. mutate to compatible structure
3. send to basecaller
4. check if reads are ready, if so, process them, otherwise repeat 1-3
5. if a batch comes back, process it and send to writter
6. have a max number of batches "in flight" limit for 1-3

This should fix the batch waiting shinanigans caused by the new "optimisations" caused by ONT.

Psy-Fer avatar Jan 29 '25 03:01 Psy-Fer

I found a better way to do this, and it's implemented into the dev branch atm.

Psy-Fer avatar Aug 07 '25 04:08 Psy-Fer