BrokenPipeError: [Errno 32] Broken pipe near the end of anvi-pan-genome

[sttongjai]

Hello,

I have encountered a BrokenPipeError [Errno 32] near the end of "anvi-pan-genome". I have been using a docker version of ANVIO (pulled the latest version) on MacPro 2019 8-Core Intel Xeon W 3.5 GHz. I am wondering if you could help fixing this issue. The following is what I got after running the following command line.

Thank you very much,

Siripong

ERROR

anvi-pan-genome -g Staphylococcus_aureus-GENOMES.db --project-name "USANEE_20221212" --output-dir StaphAU__USANEE_20221212 --num-threads 14 --minbit 0.5 --mcl-inflation 10 --use-ncbi-blast

WARNING

If you publish results from this workflow, please do not forget to cite DIAMOND (doi:10.1038/nmeth.3176), unless you use it with --use-ncbi-blast flag, and MCL (http://micans.org/mcl/ and doi:10.1007/978-1-61779-361-5_15)

Functions found ..............................:
Genomes storage ..............................: Initialized (storage hash: hashc8e252cd)
Num genomes in storage .......................: 41 Num genomes will be used .....................: 41 Pan database .................................: A new database,
/Users/siripongtongjai/ST_Bioinformatics/ST_ANVIO_Work/TEST_20221212_PanGenomics/StaphAU__USANEE_20221212/USANEE_20221212-PAN.db, has been created. Exclude partial gene calls ...................: False

AA sequences FASTA ...........................: /Users/siripongtongjai/ST_Bioinformatics/ST_ANVIO_Work/TEST_20221212_PanGenomics/StaphAU__USANEE_20221212/combined-aas.fa

Num AA sequences reported ....................: 110,924 Num excluded gene calls ......................: 0 Unique AA sequences FASTA ....................: /Users/siripongtongjai/ST_Bioinformatics/ST_ANVIO_Work/TEST_20221212_PanGenomics/StaphAU__USANEE_20221212/combined-aas.fa.unique

WARNING

You elected to use NCBI's blastp for amino acid sequence search. Running blastp will be significantly slower than DIAMOND, but in some cases, slightly more sensitive. We are unsure about whether the slight increase in sensitivity may justify significant increase in run time, but you are the boss.

NCBI BLAST MAKEDB

BLAST search db ..............................: /Users/siripongtongjai/ST_Bioinformatics/ST_ANVIO_Work/TEST_20221212_PanGenomics/StaphAU__USANEE_20221212/combined-aas.fa.unique

NCBI BLAST SEARCH

BLAST results ................................: /Users/siripongtongjai/ST_Bioinformatics/ST_ANVIO_Work/TEST_20221212_PanGenomics/StaphAU__USANEE_20221212/blast-search-results.txt

MCL INPUT

Min percent identity .........................: 0.0
Minbit .......................................: 0.5 Filtered search results ......................: 4,899,964 edges stored
MCL input ....................................: /Users/siripongtongjai/ST_Bioinformatics/ST_ANVIO_Work/TEST_20221212_PanGenomics/StaphAU__USANEE_20221212/mcl-input.txt

MCL

MCL inflation ................................: 10.0 MCL output ...................................: /Users/siripongtongjai/ST_Bioinformatics/ST_ANVIO_Work/TEST_20221212_PanGenomics/StaphAU__USANEE_20221212/mcl-clusters.txt Number of MCL clusters .......................: 5,983

CITATION

The workflow you are using will likely use 'muscle' by Edgar, doi:10.1093/nar/gkh340 (http://www.drive5.com/muscle) to align your sequences. If you publish your findings, please do not forget to properly credit this tool.

[13 Dec 22 00:01:36 Aligning amino acid sequences for genes in gene clusters] Processed 1214 of 3287 non-singl (...) ETA: 1m56sTraceback (most recent call last): File "/opt/conda/envs/anvioenv/bin/anvi-pan-genome", line 117, in pan.process() File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 925, in process gene_clusters_dict, unsuccessful_alignments = self.compute_alignments_for_gene_clusters(gene_clusters_dict) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 794, in compute_alignments_for_gene_clusters gene_clusters_item = output_queue.get() File "", line 2, in get File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/managers.py", line 819, in _callmethod kind, result = conn.recv() File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 253, in recv buf = self._recv_bytes() File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 410, in _recv_bytes buf = self._recv(4) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 386, in _recv raise EOFError EOFError Process Process-14: Process Process-9: Process Process-6: Process Process-3: Process Process-7: Traceback (most recent call last): Process Process-15: Traceback (most recent call last): Traceback (most recent call last): File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 297, in _bootstrap self.run() File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 885, in alignment_worker output_queue.put(output) File "", line 2, in put File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/managers.py", line 818, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 407, in _send_bytes self._send(header + buf) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 371, in _send n = write(self._handle, buf) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 297, in _bootstrap self.run() File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 885, in alignment_worker output_queue.put(output) File "", line 2, in put BrokenPipeError: [Errno 32] Broken pipe File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/managers.py", line 818, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 407, in _send_bytes self._send(header + buf) Traceback (most recent call last): File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 371, in _send n = write(self._handle, buf) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 297, in _bootstrap self.run() File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 885, in alignment_worker output_queue.put(output) File "", line 2, in put File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/managers.py", line 818, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 407, in _send_bytes self._send(header + buf) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 371, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe BrokenPipeError: [Errno 32] Broken pipe File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 297, in _bootstrap self.run() File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 885, in alignment_worker output_queue.put(output) File "", line 2, in put File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/managers.py", line 818, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 407, in _send_bytes self._send(header + buf) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 371, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe Process Process-11: Traceback (most recent call last): Traceback (most recent call last): File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 297, in _bootstrap self.run() File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 885, in alignment_worker output_queue.put(output) File "", line 2, in put File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/managers.py", line 818, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 407, in _send_bytes self._send(header + buf) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 371, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 297, in _bootstrap self.run() File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 885, in alignment_worker output_queue.put(output) File "", line 2, in put File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/managers.py", line 818, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 407, in _send_bytes self._send(header + buf) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 371, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe Traceback (most recent call last): File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 297, in _bootstrap self.run() File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 885, in alignment_worker output_queue.put(output) File "", line 2, in put File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/managers.py", line 818, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 407, in _send_bytes self._send(header + buf) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 371, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe Process Process-8: Traceback (most recent call last): File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 297, in _bootstrap self.run() File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 885, in alignment_worker output_queue.put(output) File "", line 2, in put File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/managers.py", line 818, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 407, in _send_bytes self._send(header + buf) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 371, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe Process Process-4: Process Process-5: Traceback (most recent call last): File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 297, in _bootstrap self.run() File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 885, in alignment_worker output_queue.put(output) File "", line 2, in put File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/managers.py", line 818, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 407, in _send_bytes self._send(header + buf) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 371, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe Traceback (most recent call last): File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 297, in _bootstrap self.run() File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 885, in alignment_worker output_queue.put(output) File "", line 2, in put File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/managers.py", line 818, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 407, in _send_bytes self._send(header + buf) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 371, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe Process Process-13: Traceback (most recent call last): File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 297, in _bootstrap self.run() File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 885, in alignment_worker output_queue.put(output) File "", line 2, in put File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/managers.py", line 818, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 407, in _send_bytes self._send(header + buf) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 371, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe Process Process-10: Traceback (most recent call last): File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 297, in _bootstrap self.run() File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 885, in alignment_worker output_queue.put(output) File "", line 2, in put File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/managers.py", line 818, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 407, in _send_bytes self._send(header + buf) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 371, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe Process Process-2: Traceback (most recent call last): File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 297, in _bootstrap self.run() File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/anvio/panops.py", line 885, in alignment_worker output_queue.put(output) File "", line 2, in put File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/managers.py", line 818, in _callmethod conn.send((self._id, methodname, args, kwds)) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 400, in _send_bytes self._send(header) File "/opt/conda/envs/anvioenv/lib/python3.7/site-packages/multiprocess/connection.py", line 371, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

[meren]

It seems one of the gene clusters anvi'o is trying to get alignments for from muscle, is exploding muscle :/ This is an issue anvi'o can do very little. I think there should be a better error message instead of this cryptic output. But essentially I may suggest you to use famsa as an aligner and see what happens, or skip alignment completely to get your pangenome (this is not a critical step to get your pangenome working).

Alternatively, you can make your genome storage and the exact command available to me via a Dropbox download link, and I can test it :)

[valentynbez]

@meren just hinting on possibly low effort-high yield replacement of muscle with PyFamsa.

[meren]

PyFamsa looks very promising and I feel like perhaps anvi'o should completely switch to that and use famsa via a native Python interface, and dump muscle and all the command line tricks once and for all.