Oyster_River_Protocol
Oyster_River_Protocol copied to clipboard
macmanes-lab
PySlice_Unpack error
oyster.mk main fails on the test data set with the following error. Quick google search reveals similar errors in conda, sometimes reported fixed by using newer Python.
======= SPAdes pipeline finished.
SPAdes log can be found here: /home/tc/sampledata/assemblies/test.spades_k55/spades.log
Thank you for using SPAdes!
Traceback (most recent call last):
File "/lab/pr/tc/ORP/software/transabyss/transabyss", line 18, in
d symbol: PySlice_Unpack
make: *** [/home/tc/sampledata/assemblies/test.transabyss.fasta] Error 1
Fixed it by editing py36_env.yml to specify a newer python version (>=3.6.1), adding git to dependencies, and getting busco directly from the gitlab repository (v3.1) instead of the out-of-date anaconda repository (v3.02). Don't know if this latter bit was necessary, but I saw in the commits that there were problems with busco when using python>3.6. Anyway, it didn't give me any problems on the included test data. Here is my new py36_env.yml:
name: orp_v2
channels:
- bioconda
- conda-forge
- defaults
- r
- antoined
dependencies:
- python>=3.6.1
- git
- zsh
- ruby
- make
- glib
- zlib==1.2.11=0
- libgcc>=7.2.0
- salmon>=0.10.2
- seqtk
- trinity
- spades
- blast
- hmmer
- samtools
- blat
- parallel
- last
- rcorrector==1.0.3
- mcl
- abyss>=2.0.1
- python-igraph
- diamond
- cd-hit
- bwa
- pip:
- biopython
- cvxopt
- cython
- numpy
- pandas
- pip
- scipy
- bashplotlib
- "git+https://gitlab.com/ezlab/busco"
So just to confirm, the test dataset now runs properly, and gives an output something like
6| #
5| ##
4| ###
3| ###
2| ### #
1| ### ### #
-----------
------------------------------------
| Summary |
------------------------------------
| observations: 20 |
| min value: -1.000000 |
| mean : -0.986450 |
| max value: -0.935000 |
------------------------------------
***** See the following link for interpretation *****
***** https://oyster-river-protocol.readthedocs.io/en/latest/strandexamine.html *****
***** QUALITY REPORT FOR: test4 using the ORP version 2.1.0 ****
***** THE ASSEMBLY CAN BE FOUND HERE: /mnt/lustre/macmaneslab/macmanes/fastq_files/assemblies/test4.ORP.fasta ****
***** BUSCO SCORE ~~~~~> C:0.0%[S:0.0%,D:0.0%],F:0.3%,M:99.7%,n:303
***** TRANSRATE SCORE ~~~~~> 0.42269
***** TRANSRATE OPTIMAL SCORE ~~~~~> 0.57644
***** UNIQUE GENES ORP ~~~~~> 39
***** UNIQUE GENES TRINITY ~~~~~> 31
***** UNIQUE GENES SPADES55 ~~~~~> 22
***** UNIQUE GENES SPADES75 ~~~~~> 25
***** UNIQUE GENES TRANSABYSS ~~~~~> 36
Here is the output:
[bam_sort_core] merging from 0 files and 10 in-memory blocks...
-parsing file: test.sorted.bam
-done parsing file, examining orientations of reads.
7| #
6| ##
5| ##
4| ##
3| ###
2| ### #
1| ### ## #
-----------
------------------------
| Summary |
------------------------
| observations: 20 |
| min value: -1.000000 |
| mean : -0.987400 |
| max value: -0.935000 |
------------------------
***** See the following link for interpretation *****
***** https://oyster-river-protocol.readthedocs.io/en/latest/strandexamine.html *****
***** QUALITY REPORT FOR: test using the ORP version 2.1.0 ****
***** THE ASSEMBLY CAN BE FOUND HERE: /home/tc/sampledata2/assemblies/test.ORP.fasta ****
***** BUSCO SCORE ~~~~~> C:0.0%[S:0.0%,D:0.0%],F:0.3%,M:99.7%,n:303
***** TRANSRATE SCORE ~~~~~> 0.41937
***** TRANSRATE OPTIMAL SCORE ~~~~~> 0.58487
***** UNIQUE GENES ORP ~~~~~> 39
***** UNIQUE GENES TRINITY ~~~~~> 31
***** UNIQUE GENES SPADES55 ~~~~~> 22
***** UNIQUE GENES SPADES75 ~~~~~> 23
***** UNIQUE GENES TRANSABYSS ~~~~~> 35
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
ok good.. the other issues aside, have you been able to successfully run a real dataset through the pipeline?
Sample problem not solved following the instructions above. This is the way py36_env.yml looks after editing with nano (note:
(orp_v2) [ant@hillary Oyster_River_Protocol]$ cat py36_env.yml name: orp_v2 channels:
- bioconda
- conda-forge
- defaults
- r
- antoined dependencies:
- python==3.6.0 {note: I check the version in my download}
- git
- zsh
- ruby
- make
- glib
- zlib==1.2.11=0
- libgcc>=7.2.0
- salmon>=0.10.2
- seqtk
- trinity
- spades
- busco
- blast
- hmmer
- samtools
- blat
- parallel
- last
- rcorrector==1.0.3
- mcl
- abyss>=2.0.1
- python-igraph
- diamond
- cd-hit
- bwa
- pip:
- biopython
- cvxopt
- cython
- numpy
- pandas
- pip
- scipy
- bashplotlib
- "git+https://gitlab.com/ezlab/busco" (orp_v2) [ant@hillary Oyster_River_Protocol]$
The error message I get is the same than before editing py36 but slightly different to the one reported above by cooketho:
Thank you for using SPAdes!
Traceback (most recent call last):
File "/home/ant/Oyster_River_Protocol/software/transabyss/transabyss", line 18, in
Any thoughts? Thanks beforehand
You have python==3.6.0. You probably need python>=3.6.1, which is what I used (see previous post).
Yes, I did try it too: I edited again py36 so that it looks like:
(orp_v2) [ant@hillary Oyster_River_Protocol]$ cat py36_env.yml name: orp_v2 channels:
- bioconda
- conda-forge
- defaults
- r
- antoined dependencies:
- python>=3.6.1
- git
- zsh
- ruby
- make
- glib
- zlib==1.2.11=0
- libgcc>=7.2.0
- salmon>=0.10.2
- seqtk
- trinity
- spades
- busco
- blast
- hmmer
- samtools
- blat
- parallel
- last
- rcorrector==1.0.3
- mcl
- abyss>=2.0.1
- python-igraph
- diamond
- cd-hit
- bwa
- pip:
- biopython
- cvxopt
- cython
- numpy
- pandas
- pip
- scipy
- bashplotlib
- "git+https://gitlab.com/ezlab/busco"
But I get the same error:
Thank you for using SPAdes!
Traceback (most recent call last):
File "/home/ant/Oyster_River_Protocol/software/transabyss/transabyss", line 18, in
Thanks!
You still have - busco in the main dependencies list. That will install the outdated anaconda repository for busco. Try deleting it. That's why I used the gitlab busco repository installed via pip. You can just cut and paste the code from my previous post.
Hmm. Sorry! I don't have any more advice, except you might want to try deleting the entire installation and starting from scratch by downloading from github again, and following the installation instructions. But before you enter make, copy-paste my post above into your py36_env.yml
I've updated the repo with the Python changes. You should be able to git update -> make at this point.
Let me know if starting from scratch works - I'm committed to helping you get this up and running.
I uninstalled everything, and all went smoothly. Now, before doing make, I modified the py36 file, but then I am getting this error message after make:
Collecting git+https://gitlab.com/ezlab/busco (from -r /home/ant/Oyster_River_Protocol/condaenv._g9hpm_0.requirements.txt (line 9)) Cloning https://gitlab.com/ezlab/busco to /tmp/pip-req-build-kzyqy7sh error: RPC failed; result=22, HTTP code = 404
This problem was solved by adding '.git' at the end of the web address in the py36 file
The installation went smoothly: I modified the py36 file according to the post above before 'make' and all run well. The previous error was resolved. However, I have a new issue. When I test the protocol, this is the message:
(orp_v2) [ant@hillary Oyster_River_Protocol]$ cd sampledata/
(orp_v2) [ant@hillary sampledata]$ /home/ant/Oyster_River_Protocol/oyster.mk main \
STRAND=RF
MEM=15
CPU=8
READ1=test.1.fq.gz
READ2=test.2.fq.gz
RUNOUT=test which: no run_BUSCO.py in which: no run_BUSCO.py in (/home/ant/Oyster_River_Protocol/software/anaconda/install/envs/orp_v2/bin:/home/ant/Oyster_River_Protocol/software/anaconda/install/condabin:/usr/local/rvm/gems/ruby-1.9.3-p551/bin:/usr/local/rvm/gems/ruby-1.9.3-p551@global/bin:/usr/local/rvm/rubies/ruby-1.9.3-p551/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/rvm/bin:/home/ant/bin:/home/ant/Oyster_River_Protocol/software/anaconda/install/bin:/home/ant/Oyster_River_Protocol/software/OrthoFinder/orthofinder:/home/ant/Oyster_River_Protocol/software/orp-transrate:/home/ant/Oyster_River_Protocol/software/transabyss) /home/ant/Oyster_River_Protocol/oyster.mk:96: *** "\n\n*** BUSCO is not installed, must fix ***". Stop. (orp_v2) [ant@hillary sampledata]$
So, I went to github and downloaded the script 'run_BUSCO.py' to anaconda/bin When I run the test sample, the protocol is getting stuck in orthofuse with the following message:
parallel: Error: --tollef has been retired. parallel: Error: Remove --tollef or use --gnu to override --tollef. make: *** [/home/ant/Oyster_River_Protocol/sampledata/orthofuse/test/orthotransrate.done] Error 255 (orp_v2) [ant@hillary sampledata]$
trying to solve this issue now: problem solved by adding flag --gnu to each parallel command found in each *.mk file.
I wonder about what can be done if you are working on a linux centos machine with no administrator privileges. I got stuck because the version of coreutils in the machine I am using is old enough so that the command sort does not have --parallel. I installed a new version of coreutils with conda in the ORP_v2 environment [together with other mandatory updates requested by conda when installing coreutils]. Of course, this last action [mandatory script updates] broke the pipeline. I assume that is too much work to rework the pipeline so to have coreutils be integral part of it? Any comment will be appreciated. Thanks