"CESAR wrapper command crashed"

[mprincipato]

Hello, dev team!

I'm very new to TOGA, and I'm trying it out on some public data for a class project.

When I run it on my personal Windows machine (using Windows Subsystem for Linux), it runs for about eight and a half hours before saying:

`### CESAR jobs done ###

Checking whether all CESAR results are complete !!CRITICAL: Too many (4576) CESAR jobs died, please check your input data and re-run TOGA !!CRITICAL: Too many (4576) CESAR jobs died, please check your input data and re-run TOGA Program finished with exit code 1`

When I look in the CESAR job logs, I find several error messages like this: (Slightly edited to remove personal information.)

!!CESAR wrapper command crashed: /home/m/code/TOGA/CESAR_wrapper.py ENST00000356220 70 /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/toga_filt_ref_annot.hdf5 /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/genome_alignment.bst /home/m/code/toga_input/hg38.2bit /home/m/code/toga_input/HLpipKuh2.2bit --cesar_binary /home/m/code/TOGA/CESAR2.0/cesar --uhq_flank 50 --temp_dir /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/cesar_temp_files --check_loss --alt_frame_del CESAR failed run for ['/home/m/code/TOGA/CESAR2.0/cesar', '/dev/stdin', '-x', '3']:  
!!FAILED COMMAND: /home/m/code/TOGA/CESAR_wrapper.py ENST00000356220 70 /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/toga_filt_ref_annot.hdf5 /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/genome_alignment.bst /home/m/code/toga_input/hg38.2bit /home/m/code/toga_input/HLpipKuh2.2bit --cesar_binary /home/m/code/TOGA/CESAR2.0/cesar --uhq_flank 50 --temp_dir /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/cesar_temp_files --check_loss --alt_frame_del
!!At attempt 0
!!CESAR wrapper command crashed: /home/m/code/TOGA/CESAR_wrapper.py ENST00000356220 70 /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/toga_filt_ref_annot.hdf5 /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/genome_alignment.bst /home/m/code/toga_input/hg38.2bit /home/m/code/toga_input/HLpipKuh2.2bit --cesar_binary /home/m/code/TOGA/CESAR2.0/cesar --uhq_flank 50 --temp_dir /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/cesar_temp_files --check_loss --alt_frame_del CESAR failed run for ['/home/m/code/TOGA/CESAR2.0/cesar', '/dev/stdin', '-x', '3']:  
!!FAILED COMMAND: /home/m/code/TOGA/CESAR_wrapper.py ENST00000356220 70 /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/toga_filt_ref_annot.hdf5 /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/genome_alignment.bst /home/m/code/toga_input/hg38.2bit /home/m/code/toga_input/HLpipKuh2.2bit --cesar_binary /home/m/code/TOGA/CESAR2.0/cesar --uhq_flank 50 --temp_dir /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/cesar_temp_files --check_loss --alt_frame_del
!!At attempt 1
!!FAILED TO EXECUTE COMMAND /home/m/code/TOGA/CESAR_wrapper.py ENST00000356220 70 /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/toga_filt_ref_annot.hdf5 /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/genome_alignment.bst /home/m/code/toga_input/hg38.2bit /home/m/code/toga_input/HLpipKuh2.2bit --cesar_binary /home/m/code/TOGA/CESAR2.0/cesar --uhq_flank 50 --temp_dir /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/cesar_temp_files --check_loss --alt_frame_del in 2 attempts
cesar_runner::cesar_job_1_0: return code: 1
cesar_runner::cesar_job_1_0: CESAR JOB FAILED: /home/m/code/TOGA/CESAR_wrapper.py ENST00000356220 70 /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/toga_filt_ref_annot.hdf5 /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/genome_alignment.bst /home/m/code/toga_input/hg38.2bit /home/m/code/toga_input/HLpipKuh2.2bit --cesar_binary /home/m/code/TOGA/CESAR2.0/cesar --uhq_flank 50 --temp_dir /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/cesar_temp_files --check_loss --alt_frame_del
cesar_runner::cesar_job_1_0: ERROR MESSAGE: CESAR wrapper command crashed: /home/m/code/TOGA/CESAR_wrapper.py ENST00000356220 70 /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/toga_filt_ref_annot.hdf5 /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/genome_alignment.bst /home/m/code/toga_input/hg38.2bit /home/m/code/toga_input/HLpipKuh2.2bit --cesar_binary /home/m/code/TOGA/CESAR2.0/cesar --uhq_flank 50 --temp_dir /home/m/code/TOGA/TOGA_project_on_2023.11.12_at_13:58:58/temp/cesar_temp_files --check_loss --alt_frame_del CESAR failed run for ['/home/m/code/TOGA/CESAR2.0/cesar', '/dev/stdin', '-x', '3']:   

Here was the command I originally used to run TOGA: ./toga.py ../toga_input/hg38.HLpipKuh2.allfilled.chain supply/hg38.wgEncodeGencodeCompV34.bed ../toga_input/hg38.2bit ../toga_input/HLpipKuh2.2bit --kt -i supply/hg38.wgEncodeGencodeCompV34.isoforms.txt --cesar_mem_limit 5 --cesar_jobs_num 500

Do you have any guesses what might be going wrong to cause the CESAR wrapper command to crash?

As far as my inputs go, I got hg38.HLpipKuh2.allfilled.chain and HLpipKuh2.fa from this Bat1K page, and I converted HLpipKuh2.fa to 2bit using faToTwoBit.