Bad fast5 warning . How to I fix ? code I run and output attached ( r10 chemistry)

[senukobla]

f5c call-methylation -t 8 -r BC9to16.fastq -b BC9to16decomfast5.sorted.bam -g cdiffstr22457.fasta > BC9to16b.tsv [meth_main::INFO] Default methylation tsv output format is changed from f5c v0.7 onwards to match latest nanopolish output. Set --meth-out-version=1 to fall back to the old format. [init_core::INFO] builtin DNA R9 nucleotide model loaded [init_core::INFO] builtin DNA R9 cpg model loaded [handle_bad_fast5::WARNING] Fast5 file [] for read [4aa8eac1-aed1-5d2e-8e28-b75c17d56812] is unreadable and will be skipped [meth_main::1.4180.71] 512 Entries (1.8M bases) loaded [handle_bad_fast5::WARNING] Fast5 file [] for read [27b43cf4-aa90-5353-8b46-877f6ea301c0] is unreadable and will be skipped [handle_bad_fast5::WARNING] Fast5 file [] for read [31f9cd71-73c5-5cbb-8bd6-eeb3e53defce] is unreadable and will be skipped [pthread_processor::2.6374.19] 512 Entries (1.8M bases) processed [meth_main::2.9123.87] 512 Entries (1.6M bases) loaded [slow_fast5_warn::INFO] Fast5 reading took more time than processing. Try increasing --iop. See http://bit.ly/f5cperf [handle_bad_fast5::WARNING] Fast5 file [] for read [e512156e-595e-5433-9930-0a0198572ed3] is unreadable and will be skipped [pthread_processor::3.9545.08] 512 Entries (1.6M bases) processed [handle_bad_fast5::WARNING] Fast5 file [] for read [e497de0a-6ac2-54f2-861a-85f41c2e2a79] is unreadable and will be skipped [meth_main::4.3124.72] 512 Entries (1.6M bases) loaded [handle_bad_fast5::WARNING] Fast5 file [] for read [54ed3e60-cbc9-50bf-98c1-b4a363d7d8fc] is unreadable and will be skipped [pthread_processor::5.3625.45] 512 Entries (1.6M bases) processed [meth_main::5.6805.19] 512 Entries (1.8M bases) loaded [handle_bad_fast5::WARNING] Fast5 file [] for read [859e33f6-8798-5546-a477-fcb7c4d4107f] is unreadable and will be skipped [handle_bad_fast5::WARNING] Fast5 file [] for read [926f934f-7008-54c2-a08c-92e89903fe9c] is unreadable and will be skipped [handle_bad_fast5::WARNING] Fast5 file [] for read [1ce6a3bf-8c3f-5175-950b-1a3b5268ec42] is unreadable and will be skipped [handle_bad_fast5::WARNING] Fast5 file [] for read [71775be5-93ac-5ae1-b1f8-9ec6a0954b11] is unreadable and will be skipped [handle_bad_fast5::WARNING] Fast5 file [] for read [280d67ea-1915-5f46-b08a-a64e3ce48a1d] is unreadable and will be skipped [pthread_processor::6.8155.74] 512 Entries (1.8M bases) processed [meth_main::7.033*5.59] 512 Entries (1.7M bases) loaded

[hasindu2008]

Hello

Thanks for the information. This is likely because those read IDs do not exist in the FAST5 files. How large is the FAST5 dataset?

Would you mind converting the FAST5 dataset to a single BLOW5 file using slow5tools and then see if these reads exit (it is much easier once converted to BLOW5)?

# convert each fast5 to a blow5
slow5tools f2s fast5_dir -d slow5_dir/
# merge all blow5 into a single file
slow5tools merge slow5_dir/ -o merged.blow5
# get the list of readIDs
slow5tools skim --rid merged.blow5 > reads.list
# see if those reads exist using grep
grep "280d67ea-1915-5f46-b08a-a64e3ce48a1d" reads.list

This way we can narrow down if the problem is due to a bug or if somehow they are actually not present.

[CountBorgula]

I keep getting this error, which I am also getting with nanopolish, anyone know how to fix it?

HDF5-DIAG: Error detected in HDF5 (1.10.4) thread 0: #000: H5G.c line 460 in H5Gopen2(): not a location major: Invalid arguments to routine minor: Inappropriate type #001: H5Gloc.c line 246 in H5G_loc(): invalid object ID major: Invalid arguments to routine minor: Bad value HDF5-DIAG: Error detected in HDF5 (1.10.4) thread 0: #000: H5G.c line 682 in H5Gclose(): not a group major: Invalid arguments to routine minor: Inappropriate type

[hasindu2008]

Do you get that when you use slow5tools to convert fast5 to slow5 too?

[CountBorgula]

The problem I am having seems to be due to an error whilst installing the hdf5 libraries.

I haven't ever tried using slow5tools so I am just trying to do that now...

[CountBorgula]

I get similar errors whilst trying to compile nanopolish.

These warnings are arising whilst compiling slow5tools

In file included from ../../../test/h5test.h:29, from zip_perf.c:23: ../../../src/H5Eprivate.h:33: note: this is the location of the previous definition 33 | #define HERROR(maj_id, min_id, ...) H5E_printf_stack(NULL, FILE, FUNC, LINE, H5E_ERR_CLS_g, maj_id, min_id, VA_ARGS) | In file included from ../../../tools/lib/h5tools.h:24, from zip_perf.c:24: ../../../tools/lib/h5tools_error.h:106: warning: "H5E_THROW" redefined 106 | #define H5E_THROW(fail_value, min_id, ...) {
| In file included from ../../../test/h5test.h:29, from zip_perf.c:23: ../../../src/H5Eprivate.h:169: note: this is the location of the previous definition 169 | #define H5E_THROW(...) {

[CountBorgula]

I get very similar warnings when compiling nanopolish, and then when I try to use it I get the erros I originally posted. So, it does seem to have something to do with the HDF5 libraries...

[CountBorgula]

Could it be the version of HDF5 I am using?

h5cc -showconfig | grep "HDF5 Version"

returns:

HDF5 Version: 1.10.4

[CountBorgula]

Nanopolish says in dependencies - libhdf5-1.8.14 compiled with multi-threading support --enable-threadsafe

[hasindu2008]

This error you have mentioned "HDF5-DIAG: Error detected in HDF5 (1.10.4) thread 0:", if I remember right (it was a long time ago) had something to do with the thread-safe version of the HDF5. The slow5tools does not require the thread-safe version of HDF5, so I suggested checking it. The commands I use to compile HDF5 for slow5tools are available at https://github.com/hasindu2008/slow5tools/blob/master/scripts/install-hdf5.sh. But for the sake of quick testing, you can try the statically linked compiled binary that comes with HDF5 built-in - these compiled binaries work on most x86_64 Linux systems: https://github.com/hasindu2008/slow5tools/releases/download/v1.1.0/slow5tools-v1.1.0-x86_64-linux-binaries.tar.gz

Since when S/BLOW5 was available, I simply convert any FAST5 to BLOW5 and simply use f5c/nanopolish on them directly as that is much easier.

[CountBorgula]

I resolved it by making sure I was using HDF5 Version: 1.8.14. Now I have run into the issue that nanopolish doesn't support R10 flow-cells yet. I am moving over to f5c as advised. My issue now is that I get no alignments of my data when running the call-methylation tool. I'll start a new issue for that if this is ok?

[hasindu2008]

Yes sure, you may open a new issue :D