jetson_nanopore_sequencing
jetson_nanopore_sequencing copied to clipboard
sirselim
Sequencing run stopped halfway, after first fastq in data folder
Hello sirselim,
Thank you for the wonderful repo. I tried to do a sequencing run, details as follows:
- Lib kits SQK-LSK109, EXP-NBD104
- Flow cell R10.3
- Run on Xavier AGX
The setup was smooth according to setup-guide-mk1c.txt. Device checks was fine. Flow cell QC was also fine.
When I tried to do the run, it started fine. Then after the first fastq and fast5 files were generated, it Stopped with error. Pictures below, and log files from the last run attached:
drift_correction_FAS49565_9620821b.csv final_summary_FAS49565_9620821b.txt mux_scan_data_FAS49565_9620821b.csv sequencing_summary_FAS49565_9620821b.txt
I don't think anyone reported this so far. Please help me. Thank you.
Have you installed an extra ssd and changed the data path? MinKNOW stops the sequencing run if it has less than 50gb free. On a desktop I’ve seen similar behaviour when it goes below 50gb free. It wasn’t easy to troubleshoot either, but was in the notifications in MinKNOW.
MinKNOW will let you know if there is not enough storage present. This to me looks like an error with GPU Guppy not being correctly set up. If you take a look through the logs you should be able to quickly see if there are issues with communication between minknow and guppy.
Hmm, I will try to find the logs later. But just to let you know, I ended up doing the run on another comp. But I did the basecalling on the Xavier AGX using Guppy through MinKNOW. It works well though, no error. Doesn't that mean that MinKNOW was able to find the GPU Guppy?
Interesting, sorry I had missed the fact that you'd been running for 20+ mins. Yes if you can basecall through the MinKNOW GUI then it usually means that MinKNOW and Guppy are communicating.
I would go through the logs and look for anything suspicious. @mcrone mentioned storage space, I personally haven't been able to start a run if the amount of space is deemed as too low, so I would be surprised if you've run out of disk space - but always worth checking.
@sirselim I've personally had a run in MinKNOW where it ran for an hour and then stopped, then when I restarted it it did the same thing and then I looked at the notifications and there was a warning that it had stopped because I had less than 50gb of disk space left. Deleted old data and it then worked fine.
@mcrone - good to know. What type of set up are you using? The reason I ask is that by default MinKNOW won't let you start a run if it determines that there is what it believe's to be too little space. A the very minimum there is a warning stating that XGB are left on the storage drive before you start a run.
@sirselim it did have the warning, but I usually ignore it. This was just a windows machine that I use to just get the raw data. Trying to move over to the Orin, but it has been difficult to install either guppy or bonito (fast-ctc-decode seems tricky to install).
@mcrone - the joys of being at the bleeding edge of hardware and software. When I started out with the Jetson boards I hit all the same road blocks. Things will evolve quickly though, especially now that Apple has adopted ARM as their architecture moving forwards.
@ZarulHanifah - did you figure out if you had run out of drive space as @mcrone suggested?
I am also getting this issue. It fails the hardware check on collecting data for 5 seconds with the following error in the bream.txt file:
[__main__: 214] - 2022-05-16 13:12:33,117 - INFO - Starting hardware check [bream4.device_interfaces.devices.base_device: 422] - 2022-05-16 13:12:34,986 - INFO - Set the device to test current [bream4.device_interfaces.devices.base_device: 225] - 2022-05-16 13:12:34,990 - INFO - Set the device to test current 0 [bream4.toolkit.procedure_components.data_extraction.collect_raw_data: 24] - 2022-05-16 13:12:35,007 - INFO - Capturing 1.0 seconds of raw data, current sample number: 2496 [bream4.device_interfaces.devices.base_device: 422] - 2022-05-16 13:12:36,209 - INFO - Set the device to test current [bream4.device_interfaces.devices.base_device: 225] - 2022-05-16 13:12:36,214 - INFO - Set the device to test current 100 [bream4.toolkit.procedure_components.data_extraction.collect_raw_data: 24] - 2022-05-16 13:12:36,231 - INFO - Capturing 1.0 seconds of raw data, current sample number: 8640 [bream4.device_interfaces.devices.base_device: 225] - 2022-05-16 13:12:37,401 - INFO - Set the device to test current 0 [bream4.toolkit.calibration.calibrate_device: 118] - 2022-05-16 13:12:39,660 - INFO - Using calibration 1/1 [bream4.toolkit.procedure_components.data_extraction.collect_raw_data: 24] - 2022-05-16 13:13:07,692 - INFO - Capturing 5 seconds of raw data, current sample number: 67904 [__main__: 82] - 2022-05-16 13:13:15,134 - INFO - CTC script failed with total failed 124 [__main__: 214] - 2022-05-16 13:13:15,135 - INFO - Hardware check failed. Please unplug and re-plug the CTC or Flongle adapter and try again.
It does the initial demux fine as a part of the sequencing run, but then stops quite early on with the error that @ZarulHanifah has identified (and this in the system messages "There was an error receiving data from the device. Try re-plugging the device if possible, or rebooting the system. If the issue persists, please contact Support.").
I'm trying to just do the data collection, no basecalling.
Sorry for the late reply, and thank you for the comments.
Well, I am pretty sure I still have plenty of space. A slight difference in my setup is that I am using a 2 Tb SSD drive,
but I think I have setup MinKNOW to the proper data path (This is from a CTC check, works fine. See the message Disk space remaining)
From the failures in the run mentioned above, there were no error messages.
There were a few WARNINGs in the logs, but I am not sure if they are related to my case; I saw something like a mk manager exit asap? It is also a bit tricky to know which log reports are relevant because for some reason my AGX setup cannot update date and time properly; it shows some symbols instead. I tried looking for solutions, but haven't had success yet. For now, I'll just upload a bunch of log files if you want to have a look at them - sorry.
usb_firmware.toml.txt mk_manager_svc_log-10.txt MN39858.zip