incomprehensible issue

[ChloeAmbroset]

Goodmorning,

I run longqc with ONT data with the following command : srun longQC.py sampleqc -x ont-rapid -s ${sample} -p 30 -o ${folder_sampleqc}/${sample}_500X_rapid ${long_read} and some of my samples succeeded but some other seemed to crash and I don't understand why (the issue is uncomprehensible for me, I'm biologist...-> see below for the issue) I've questioned the cluster manager @lecorguille and he thinks it's rather a tool dependant issue than an installation one. Could you help us to resolve it ?

thans a lot regards

Chloé lq_coverage:2021-03-16 18:57:46,714:374:INFO:Coordinates of coverage analysis were parsed. Traceback (most recent call last): File "/opt/LongQC/longQC.py", line 933, in main(args) File "/opt/LongQC/longQC.py", line 63, in main args.handler(args) File "/opt/LongQC/longQC.py", line 598, in command_sample lc.plot_length_vs_coverage(fig_path_cl) File "/opt/LongQC/lq_coverage.py", line 461, in plot_length_vs_coverage self.__check_outlier_coverage(interval) File "/opt/LongQC/lq_coverage.py", line 482, in __check_outlier_coverage meds = stats['median'][np.where(stats['size']>=LqCoverage.LENGTH_BIN_THRESHOLD)[0]] File "/opt/conda/lib/python3.8/site-packages/pandas/core/series.py", line 908, in getitem return self._get_with(key) File "/opt/conda/lib/python3.8/site-packages/pandas/core/series.py", line 943, in _get_with return self.loc[key] File "/opt/conda/lib/python3.8/site-packages/pandas/core/indexing.py", line 879, in getitem return self._getitem_axis(maybe_callable, axis=axis) File "/opt/conda/lib/python3.8/site-packages/pandas/core/indexing.py", line 1099, in _getitem_axis return self._getitem_iterable(key, axis=axis) File "/opt/conda/lib/python3.8/site-packages/pandas/core/indexing.py", line 1037, in _getitem_iterable keyarr, indexer = self._get_listlike_indexer(key, axis, raise_missing=False) File "/opt/conda/lib/python3.8/site-packages/pandas/core/indexing.py", line 1254, in _get_listlike_indexer self._validate_read_indexer(keyarr, indexer, axis, raise_missing=raise_missing) File "/opt/conda/lib/python3.8/site-packages/pandas/core/indexing.py", line 1315, in _validate_read_indexer raise KeyError( KeyError: "Passing list-likes to .loc or [] with any missing labels is no longer supported. The following labels were missing: Int64Index([2, 3], dtype='int64', name='Binned read length'). See https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#deprecate-loc-reindex-listlike" srun: error: cpu-node-18: task 0: Exited with exit code 1

[yfukasawa]

Hello @ChloeAmbroset,

Thank you for your interest in LongQC. As you can run the tool for some datasets, some unexpected output could be generated by some of your datasets.

If it is possible, could you please share 2 files (coverage_err.txt and coverage_out.txt) for a dataset faced this issue? They should be located in a folder ${folder_sampleqc}/${sample}_500X_rapid/analysis/minimap2/. I would be able to spot the issue from these files. For now, I think lengths of reads may have caused this issue (either much shorter/longer than our expectations).

Technically these files should not contain sensitive information, but you can edit (sample name, path, etc.) when you need.

Yoshinori

[ChloeAmbroset]

Hello Yoshinori,

Thank's a lot for your reply. As request, I share you the following files : coverage_err_Mbv-L15762C1.txt coverage_out_Mbv-L15762C1.txt

hope you will be abble to read it. otherwize I'll copy-paste them again.

Thank's a lot in advance

Best regards

Chloé

[yfukasawa]

Hello @ChloeAmbroset,

Thank you for sharing the files. They're very helpful. Let me check at my side. In the meantime, upgrading the version by cluster manager (@lecorguille) might resolve the issue. Your output was generated by a bit older version (<version 1.2), and may have hit some old code issue.

Will update once I pinpoint the issue. thanks.

Best, Yoshinori

[yfukasawa]

Hello @ChloeAmbroset,

Thank you for your patience and sharing. I pushed revised codes, and could you please ask upgrading LongQC?

Version 1.2.0b should be able to handle your datasets faced some issues. $python longQC.py --version The command should return "LongQC 1.2.0b".

Yoshinori

[lecorguille]

Hi @yfukasawa , can you tag the 1.2.0b version. We provide longqc as a Singularity image that pull this github repository

Thanks!

[yfukasawa]

Hi @lecorguille,

Sure, I just added 1.2.0b in release. Could you please check it?

We provide longqc as a Singularity image that pull this github repository

that's cool. never tested on Singularity, but our cluster also uses it. it's good to know, thanks!

Y.

[ChloeAmbroset]

Hi @yfukasawa ,

@lecorguille installed and updated the new version of longqc/1.2 and longqc crashed again ... I had this error message after this following command: srun longQC.py sampleqc -x ont-rapid -s ${sample} -p 30 -o ${folder_sampleqc}/${sample}_rawreads-rapid ${long_read}

Traceback (most recent call last): File "/opt/LongQC/longQC.py", line 957, in main(args) File "/opt/LongQC/longQC.py", line 63, in main args.handler(args) File "/opt/LongQC/longQC.py", line 370, in command_sample df_mask = pd.read_table(lm.get_outfile_path(), sep='\t', header=None) File "/opt/conda/lib/python3.8/site-packages/pandas/io/parsers.py", line 689, in read_table return _read(filepath_or_buffer, kwds) File "/opt/conda/lib/python3.8/site-packages/pandas/io/parsers.py", line 462, in _read parser = TextFileReader(filepath_or_buffer, **kwds) File "/opt/conda/lib/python3.8/site-packages/pandas/io/parsers.py", line 819, in init self._engine = self._make_engine(self.engine) File "/opt/conda/lib/python3.8/site-packages/pandas/io/parsers.py", line 1050, in _make_engine return mapping[engine](self.f, **self.options) # type: ignore[call-arg] File "/opt/conda/lib/python3.8/site-packages/pandas/io/parsers.py", line 1898, in init self._reader = parsers.TextReader(self.handles.handle, **kwds) File "pandas/_libs/parsers.pyx", line 521, in pandas._libs.parsers.TextReader.cinit pandas.errors.EmptyDataError: No columns to parse from file srun: error: cpu-node-39: task 0: Exited with exit code 1

thanks again for your help

Chloé

[yfukasawa]

Hi @ChloeAmbroset,

Thank you for letting me know. The error looks unrelated to the issue revised recently, unfortunately, so let me ask a below question to trouble shoot.

1. do you get some errors when you execute below? (from the command, I understood LongQC was installed under /opt) /opt/LongQC/minimap2-coverage/sdust

the expected message is below:

Usage: sdust [-w 64] [-t 20] <in.fa>.

If you don't get the above message, a program 'sdust' may haven't run in your environment due to some reasons. This may not be the case, but let me check this point first.

Yoshinori

[lecorguille]

Indeed, my bad! I had a misspelling in the minimap2-coverage path (minimap2_coverage) and so sdust wasn't compiled