EDTA
EDTA copied to clipboard
oushujun
TIR learner error - Illegal instructions
I'm getting this error in TIR learner when running the test genome.
Fri Dec 10 16:15:45 EST 2021 Identify TIR candidates from scratch.
Species: others
/Users/benweinberg/GitHub/EDTA/bin/TIR-Learner2.5/TIR-Learner2.5.sh: line 290: 39174 Illegal instruction: 4 python3 $path/Module3_New/getDataset.py -g $genomeFile -name $genomeName -p $path -t $t -d $dir"/Module3_New"
cat: '*-+-DTA.fa': No such file or directory
cat: '*-+-DTC.fa': No such file or directory
cat: '*-+-DTH.fa': No such file or directory
cat: '*-+-DTM.fa': No such file or directory
cat: '*-+-DTT.fa': No such file or directory
cat: '*-+-NonTIR.fa': No such file or directory
cat: '*-+-*-+-*.gff3': No such file or directory
rm: cannot remove '*-+-*-+-*.gff3': No such file or directory
Traceback (most recent call last):
File "/Users/benweinberg/GitHub/EDTA/bin/TIR-Learner2.5/Module3_New/CombineAll.py", line 75, in <module>
f_m3=removeDupinSingle("%s.gff3"%(genome_Name+spliter+"Module3"))
File "/Users/benweinberg/GitHub/EDTA/bin/TIR-Learner2.5/Module3_New/CombineAll.py", line 57, in removeDupinSingle
f=pd.read_csv(file,header=None,sep="\t") #shujun
File "/Users/benweinberg/opt/anaconda3/envs/EDTA/lib/python3.6/site-packages/pandas/io/parsers.py", line 688, in read_csv
return _read(filepath_or_buffer, kwds)
File "/Users/benweinberg/opt/anaconda3/envs/EDTA/lib/python3.6/site-packages/pandas/io/parsers.py", line 454, in _read
parser = TextFileReader(fp_or_buf, **kwds)
File "/Users/benweinberg/opt/anaconda3/envs/EDTA/lib/python3.6/site-packages/pandas/io/parsers.py", line 948, in __init__
self._make_engine(self.engine)
File "/Users/benweinberg/opt/anaconda3/envs/EDTA/lib/python3.6/site-packages/pandas/io/parsers.py", line 1180, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "/Users/benweinberg/opt/anaconda3/envs/EDTA/lib/python3.6/site-packages/pandas/io/parsers.py", line 2010, in __init__
self._reader = parsers.TextReader(src, **kwds)
File "pandas/_libs/parsers.pyx", line 540, in pandas._libs.parsers.TextReader.__cinit__
pandas.errors.EmptyDataError: No columns to parse from file
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/Users/benweinberg/opt/anaconda3/envs/EDTA/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/Users/benweinberg/opt/anaconda3/envs/EDTA/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/Users/benweinberg/GitHub/EDTA/bin/TIR-Learner2.5/Module3/GetAllSeq.py", line 32, in GetListFromFile
f=open(file,"r+")
FileNotFoundError: [Errno 2] No such file or directory: 'TIR-Learner_FinalAnn_filter.gff3'
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/benweinberg/GitHub/EDTA/bin/TIR-Learner2.5/Module3/GetAllSeq.py", line 63, in <module>
pool.map(GetListFromFile,fileList) #shujun
File "/Users/benweinberg/opt/anaconda3/envs/EDTA/lib/python3.6/multiprocessing/pool.py", line 266, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/Users/benweinberg/opt/anaconda3/envs/EDTA/lib/python3.6/multiprocessing/pool.py", line 644, in get
raise self._value
FileNotFoundError: [Errno 2] No such file or directory: 'TIR-Learner_FinalAnn_filter.gff3'
mv: cannot stat 'TIR-Learner/*FinalAnn*.gff3': No such file or directory
mv: cannot stat 'TIR-Learner/*FinalAnn*.fa': No such file or directory
Can't open ./TIR-Learner-Result/TIR-Learner_FinalAnn.fa: No such file or directory at /Users/benweinberg/GitHub/EDTA/util/rename_tirlearner.pl line 19.
Warning: LOC list genome.fa.mod.TIR.ext30.list is empty.
Error: Error while loading sequenceCan't open ./TIR-Learner-Result/TIR-Learner_FinalAnn.gff3: No such file or directory.
Warning: The TIR result file has 0 bp!
I'm also realizing when I try to relaunch the EDTA environment again, I'm getting a "Illegal instruction: 4" error.
Hello @benweinberg89,
This is clearly an installation issue. You may need to create a new conda env and reinstall EDTA. Please try the methods provided in Readme.
Best, Shujun
Hello @oushujun
I had the same problem when running the test data.
perl ../EDTA.pl --genome genome.fa --cds genome.cds.fa --curatedlib ../database/rice6.9.5.liban --exclude genome.exclude.bed --overwrite 1 --sensitive 1 --anno 1 --evaluate 1 --threads 10
########################################################
##### Extensive de-novo TE Annotator (EDTA) v2.0.0 ####
##### Shujun Ou ([email protected]) ####
########################################################
Fri Dec 17 15:52:37 CST 2021 Dependency checking:
All passed!
A custom library ../database/rice6.9.5.liban is provided via --curatedlib. Please make sure this is a manually curated library but not machine generated.
A CDS file genome.cds.fa is provided via --cds. Please make sure this is the DNA sequence of coding regions only.
A BED file is provided via --exclude. Regions specified by this file will be excluded from TE annotation and masking.
Fri Dec 17 15:52:39 CST 2021 Obtain raw TE libraries using various structure-based programs:
Fri Dec 17 15:52:39 CST 2021 EDTA_raw: Check dependencies, prepare working directories.
Fri Dec 17 15:52:40 CST 2021 Start to find LTR candidates.
Fri Dec 17 15:52:40 CST 2021 Identify LTR retrotransposon candidates from scratch.
Warning: LOC list genome.fa.mod.ltrTE.veryfalse is empty.
Fri Dec 17 15:53:04 CST 2021 Finish finding LTR candidates.
Fri Dec 17 15:53:04 CST 2021 Start to find TIR candidates.
Fri Dec 17 15:53:04 CST 2021 Identify TIR candidates from scratch.
Species: others
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/data2/zhengzhuqing/00.software/EDTA-master/share/EDTA/bin/TIR-Learner2.5/Module3_New/getDataset.py", line 110, in Predict
model = load_model(path+'/CNN0912.h5')
File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/site-packages/tensorflow/python/keras/saving/save.py", line 146, in load_model
return hdf5_format.load_model_from_hdf5(filepath, custom_objects, compile)
File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/site-packages/tensorflow/python/keras/saving/hdf5_format.py", line 210, in load_model_from_hdf5
model_config = json.loads(model_config.decode('utf-8'))
AttributeError: 'str' object has no attribute 'decode'
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/data2/zhengzhuqing/00.software/EDTA-master/share/EDTA/bin/TIR-Learner2.5/Module3_New/getDataset.py", line 139, in <module>
d = pool.map(Predict,files)
File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/multiprocessing/pool.py", line 266, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/multiprocessing/pool.py", line 644, in get
raise self._value
AttributeError: 'str' object has no attribute 'decode'
cat: '*-+-DTA.fa': No such file or directory
cat: '*-+-DTC.fa': No such file or directory
cat: '*-+-DTH.fa': No such file or directory
cat: '*-+-DTM.fa': No such file or directory
cat: '*-+-DTT.fa': No such file or directory
cat: '*-+-NonTIR.fa': No such file or directory
cat: '*-+-*-+-*.gff3': No such file or directory
rm: cannot remove '*-+-*-+-*.gff3': No such file or directory
Traceback (most recent call last):
File "/data2/zhengzhuqing/00.software/EDTA-master/share/EDTA/bin/TIR-Learner2.5/Module3_New/CombineAll.py", line 75, in <module>
f_m3=removeDupinSingle("%s.gff3"%(genome_Name+spliter+"Module3"))
File "/data2/zhengzhuqing/00.software/EDTA-master/share/EDTA/bin/TIR-Learner2.5/Module3_New/CombineAll.py", line 57, in removeDupinSingle
f=pd.read_csv(file,header=None,sep="\t") #shujun
File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/site-packages/pandas/io/parsers.py", line 688, in read_csv
return _read(filepath_or_buffer, kwds)
File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/site-packages/pandas/io/parsers.py", line 454, in _read
parser = TextFileReader(fp_or_buf, **kwds)
File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/site-packages/pandas/io/parsers.py", line 948, in __init__
self._make_engine(self.engine)
File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/site-packages/pandas/io/parsers.py", line 1180, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/site-packages/pandas/io/parsers.py", line 2010, in __init__
self._reader = parsers.TextReader(src, **kwds)
File "pandas/_libs/parsers.pyx", line 540, in pandas._libs.parsers.TextReader.__cinit__
pandas.errors.EmptyDataError: No columns to parse from file
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/data2/zhengzhuqing/00.software/EDTA-master/share/EDTA/bin/TIR-Learner2.5/Module3/GetAllSeq.py", line 32, in GetListFromFile
f=open(file,"r+")
FileNotFoundError: [Errno 2] No such file or directory: 'TIR-Learner_FinalAnn_filter.gff3'
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/data2/zhengzhuqing/00.software/EDTA-master/share/EDTA/bin/TIR-Learner2.5/Module3/GetAllSeq.py", line 63, in <module>
pool.map(GetListFromFile,fileList) #shujun
File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/multiprocessing/pool.py", line 266, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/multiprocessing/pool.py", line 644, in get
raise self._value
FileNotFoundError: [Errno 2] No such file or directory: 'TIR-Learner_FinalAnn_filter.gff3'
mv: cannot stat 'TIR-Learner/*FinalAnn*.gff3': No such file or directory
mv: cannot stat 'TIR-Learner/*FinalAnn*.fa': No such file or directory
Can't open ./TIR-Learner-Result/TIR-Learner_FinalAnn.fa: No such file or directory at /data2/zhengzhuqing/00.software/EDTA-master/share/EDTA/util/rename_tirlearner.pl line 19.
Warning: LOC list genome.fa.mod.TIR.ext30.list is empty.
Error: Error while loading sequenceCan't open ./TIR-Learner-Result/TIR-Learner_FinalAnn.gff3: No such file or directory.
Warning: The TIR result file has 0 bp!
Fri Dec 17 15:53:46 CST 2021 Start to find Helitron candidates.
Fri Dec 17 15:53:46 CST 2021 Identify Helitron candidates from scratch.
Fri Dec 17 15:54:27 CST 2021 Finish finding Helitron candidates.
Fri Dec 17 15:54:27 CST 2021 Execution of EDTA_raw.pl is finished!
ERROR: Raw TIR results not found in genome.fa.mod.EDTA.raw/genome.fa.mod.TIR.raw.fa
If you believe the program is working properly, this may be caused by the lack of intact TIRs in your genome. Consider to use the --force 1 parameter to overwrite this check
Best wishes, Zheng zhuqing
Hello Zhuqing,
Looks like an installation issue to me. Did you use the new conda or the previous version to run 2.0?
Shujun
On Fri, Dec 17, 2021 at 1:58 AM biozzq @.***> wrote:
Hello @oushujun https://github.com/oushujun
I had the same problem when running the test data.
perl ../EDTA.pl --genome genome.fa --cds genome.cds.fa --curatedlib ../database/rice6.9.5.liban --exclude genome.exclude.bed --overwrite 1 --sensitive 1 --anno 1 --evaluate 1 --threads 10
########################################################
Extensive de-novo TE Annotator (EDTA) v2.0.0
Shujun Ou @.***)
########################################################
Fri Dec 17 15:52:37 CST 2021 Dependency checking: All passed!
A custom library ../database/rice6.9.5.liban is provided via --curatedlib. Please make sure this is a manually curated library but not machine generated.
A CDS file genome.cds.fa is provided via --cds. Please make sure this is the DNA sequence of coding regions only.
A BED file is provided via --exclude. Regions specified by this file will be excluded from TE annotation and masking.
Fri Dec 17 15:52:39 CST 2021 Obtain raw TE libraries using various structure-based programs: Fri Dec 17 15:52:39 CST 2021 EDTA_raw: Check dependencies, prepare working directories.
Fri Dec 17 15:52:40 CST 2021 Start to find LTR candidates.
Fri Dec 17 15:52:40 CST 2021 Identify LTR retrotransposon candidates from scratch.
Warning: LOC list genome.fa.mod.ltrTE.veryfalse is empty. Fri Dec 17 15:53:04 CST 2021 Finish finding LTR candidates.
Fri Dec 17 15:53:04 CST 2021 Start to find TIR candidates.
Fri Dec 17 15:53:04 CST 2021 Identify TIR candidates from scratch.
Species: others multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/multiprocessing/pool.py", line 119, in worker result = (True, func(*args, **kwds)) File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar return list(map(*args)) File "/data2/zhengzhuqing/00.software/EDTA-master/share/EDTA/bin/TIR-Learner2.5/Module3_New/getDataset.py", line 110, in Predict model = load_model(path+'/CNN0912.h5') File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/site-packages/tensorflow/python/keras/saving/save.py", line 146, in load_model return hdf5_format.load_model_from_hdf5(filepath, custom_objects, compile) File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/site-packages/tensorflow/python/keras/saving/hdf5_format.py", line 210, in load_model_from_hdf5 model_config = json.loads(model_config.decode('utf-8')) AttributeError: 'str' object has no attribute 'decode' """
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/data2/zhengzhuqing/00.software/EDTA-master/share/EDTA/bin/TIR-Learner2.5/Module3_New/getDataset.py", line 139, in
d = pool.map(Predict,files) File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/multiprocessing/pool.py", line 266, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/multiprocessing/pool.py", line 644, in get raise self._value AttributeError: 'str' object has no attribute 'decode' cat: '-+-DTA.fa': No such file or directory cat: '-+-DTC.fa': No such file or directory cat: '-+-DTH.fa': No such file or directory cat: '-+-DTM.fa': No such file or directory cat: '-+-DTT.fa': No such file or directory cat: '-+-NonTIR.fa': No such file or directory cat: '-+--+-.gff3': No such file or directory rm: cannot remove '-+--+-.gff3': No such file or directory Traceback (most recent call last): File "/data2/zhengzhuqing/00.software/EDTA-master/share/EDTA/bin/TIR-Learner2.5/Module3_New/CombineAll.py", line 75, in f_m3=removeDupinSingle("%s.gff3"%(genome_Name+spliter+"Module3")) File "/data2/zhengzhuqing/00.software/EDTA-master/share/EDTA/bin/TIR-Learner2.5/Module3_New/CombineAll.py", line 57, in removeDupinSingle f=pd.read_csv(file,header=None,sep="\t") #shujun File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/site-packages/pandas/io/parsers.py", line 688, in read_csv return _read(filepath_or_buffer, kwds) File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/site-packages/pandas/io/parsers.py", line 454, in _read parser = TextFileReader(fp_or_buf, **kwds) File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/site-packages/pandas/io/parsers.py", line 948, in init self._make_engine(self.engine) File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/site-packages/pandas/io/parsers.py", line 1180, in _make_engine self._engine = CParserWrapper(self.f, **self.options) File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/site-packages/pandas/io/parsers.py", line 2010, in init self._reader = parsers.TextReader(src, **kwds) File "pandas/_libs/parsers.pyx", line 540, in pandas._libs.parsers.TextReader.cinit pandas.errors.EmptyDataError: No columns to parse from file multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/multiprocessing/pool.py", line 119, in worker result = (True, func(*args, **kwds)) File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar return list(map(*args)) File "/data2/zhengzhuqing/00.software/EDTA-master/share/EDTA/bin/TIR-Learner2.5/Module3/GetAllSeq.py", line 32, in GetListFromFile f=open(file,"r+") FileNotFoundError: [Errno 2] No such file or directory: 'TIR-Learner_FinalAnn_filter.gff3' """ The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/data2/zhengzhuqing/00.software/EDTA-master/share/EDTA/bin/TIR-Learner2.5/Module3/GetAllSeq.py", line 63, in
pool.map(GetListFromFile,fileList) #shujun File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/multiprocessing/pool.py", line 266, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/data2/zhengzhuqing/00.software/EDTA-master/lib/python3.6/multiprocessing/pool.py", line 644, in get raise self._value FileNotFoundError: [Errno 2] No such file or directory: 'TIR-Learner_FinalAnn_filter.gff3' mv: cannot stat 'TIR-Learner/FinalAnn.gff3': No such file or directory mv: cannot stat 'TIR-Learner/FinalAnn.fa': No such file or directory Can't open ./TIR-Learner-Result/TIR-Learner_FinalAnn.fa: No such file or directory at /data2/zhengzhuqing/00.software/EDTA-master/share/EDTA/util/rename_tirlearner.pl line 19. Warning: LOC list genome.fa.mod.TIR.ext30.list is empty. Error: Error while loading sequenceCan't open ./TIR-Learner-Result/TIR-Learner_FinalAnn.gff3: No such file or directory. Warning: The TIR result file has 0 bp!
Fri Dec 17 15:53:46 CST 2021 Start to find Helitron candidates.
Fri Dec 17 15:53:46 CST 2021 Identify Helitron candidates from scratch.
Fri Dec 17 15:54:27 CST 2021 Finish finding Helitron candidates.
Fri Dec 17 15:54:27 CST 2021 Execution of EDTA_raw.pl is finished!
ERROR: Raw TIR results not found in genome.fa.mod.EDTA.raw/genome.fa.mod.TIR.raw.fa If you believe the program is working properly, this may be caused by the lack of intact TIRs in your genome. Consider to use the --force 1 parameter to overwrite this check
Best wishes, Zheng zhuqing
— Reply to this email directly, view it on GitHub https://github.com/oushujun/EDTA/issues/245#issuecomment-996511798, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNX4NBPXH434YWZYWVGPQ3URLURHANCNFSM5J2CUWWQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.Message ID: @.***>
Dear @oushujun
Yes, I used a new conda environment to install the 2.0;
conda create -n EDTA
conda activate EDTA
conda install -c bioconda edta
Best wishes, Zheng zhuqing
Dear Zhuqing,
You may use the old env to drive the 2.0 and see if you have issues. The new conda env is essentially just had the version number changed and nothing else. Dependencies for EDTA have not changed for quite a while. You may try other conda installation approaches if you really want the 2.0 env.
Best, Shujun
Dear @oushujun
Thanks for your prompt reply. I have removed my old conda env. To keep the dependencies for EDTA, I changed to install the packages using following command, however, it also gives me above error when running the test data.
git clone https://github.com/oushujun/EDTA.git
conda env create -f EDTA.yml
Best wishes. Zheng zhuqing
This problem is caused by having user installed PYTHON packages because even inside a container, python will look in ~/.local/lib/python3.6/site-packages and it will use those rather than the ones inside /opt/conda/lib/python3.6/site-packages. This happened for me running it on singularity and I solved it by setting PYTHONPATH prior to calling the container. It is worth setting PYTHONPATH to just be /opt/conda/lib/python3.6/site-packages in the Dockerfile to prevent this issue for users.
In case my previous comment wasn't enough to help, here's how I ran the tests (inside the test directory) with Singularity on my setup:
$ export PYTHONNOUSERSITE=1
$ singularity exec ../EDTA.sif EDTA.pl --genome genome.fa --cds genome.cds.fa --curatedlib ../database/rice6.9.5.liban --exclude genome.exclude.bed --overwrite 1 --sensitive 1 --anno 1 --evaluate 1 --threads 10
There are better ways to handle this such as fixing the Dockerfile to set the variable when the container is built, or other methods depending on what you're running to pass the value in but the above is simple and works.
@shanesturrock Thank you for helping to identify the issue and providing solutions. I was trying to incorporate and test that in Dockerfile but have python issues due to the Apple M1 chip. I will need to come back to this when I have root access to a non-M1 computer.