`statistics` not found, and cannot resume at step "afterdedup"

[agilly]

My compute environment is a bit unstable at the moment, with I/O failures causing steps to sometimes fail. I therefore often have to use the -S flag to resume at a particular point in the pipeline. Recently, I saw the following:

$> ./opt/scripts/juicer.sh -S dedup -D $optdir -z hg38.p13.fa.gz -d $workdir -p hg38
(-:  Mark duplicates done successfully
Picked up _JAVA_OPTIONS: -Xmx1024m -Xms1024m
Exception in thread "main" java.lang.RuntimeException: Unknown command: statistics
        at juicebox.tools.HiCTools.main(HiCTools.java:98)
Picked up _JAVA_OPTIONS: -Xmx1024m -Xms1024m
Exception in thread "main" java.lang.RuntimeException: Unknown command: statistics
        at juicebox.tools.HiCTools.main(HiCTools.java:98)
Picked up _JAVA_OPTIONS: -Xmx150000m -Xms150000m
Not including fragment map
Error while reading graphs file: java.io.FileNotFoundException: /lustre/groups/itg/teams/zeggini/projects/child_diabesity/analysis/initial_sample/qc/Juicer/L79851_Track-123454/aligned/inter_hists.m (N
o such file or directory)
Start preprocess
Writing header
Writing body
........................................................................................................................................................................................................
................................................................................................java.io.FileNotFoundException: /tmp/blocks17658201768437070861bin (Cannot send after transport endpoint
shutdown)

The above looks like a genuine error followed by an I/O error. Any pointers on how to solve the unknown command: statistics error?

Following repairs on the file system, I wanted to resume after dedup. According to the help, this is done with the afterdedup stage. However, I get the following

$> ./opt/scripts/juicer.sh -S afterdedup -D $optdir -z hg38.p13.fa.gz -d $workdir -p hg38
Usage: juicer.sh [-g genomeID] [-d topDir] [-s site]\n                 [-a about] [-S stage] [-p chrom.sizes path]\n                 [-y restriction site file] [-z reference genome file]\n                 [-D Juicer scripts parent dir] [-b ligation] [-t threads]\n                 [-T threadsHic] [-i sample] [-k library] [-w wobble]\n                 [-e] [-h] [-f] [-j] [-u] [-m] [--assembly] [--cleanup] [--qc]
* [stage]: must be one of "chimeric", "merge", "dedup", "afterdedup", "final", "postproc", or "early".\n    -Use "chimeric" when alignment has finished or to start from previously\n     aligned files\n    -Use "merge" when chimeric handling has finished but the merged_sort file\n     has not yet been created.\n    -Use "dedup" when the files have been merged into merged_sort but\n     merged_dedup has not yet been created.\n    -Use "afterdedup" when dedup is complete but statistics haven't been run\n    -Use "final" when the reads have been deduped into merged_dedup but the\n     final hic files have not yet been created.\n    -Use "postproc" when the hic files have been created and only\n     postprocessing feature annotation remains to be completed.\n    -Use "early" for an early exit, before the final creation of the hic files\n    Can also use -e flag to exit early

The same command proceeds with -S final and -S dedup. It actually seems that -S final picks up after dedup, since it tries to generate the stats again AFAICS:

$>./opt/scripts/juicer.sh -S dedup -D $optdir -z hg38.p13.fa.gz -d $workdir -p hg38
Picked up _JAVA_OPTIONS: -Xmx1024m -Xms1024m
Exception in thread "main" java.lang.RuntimeException: Unknown command: statistics
        at juicebox.tools.HiCTools.main(HiCTools.java:98)
Picked up _JAVA_OPTIONS: -Xmx1024m -Xms1024m
Exception in thread "main" java.lang.RuntimeException: Unknown command: statistics
        at juicebox.tools.HiCTools.main(HiCTools.java:98)
Picked up _JAVA_OPTIONS: -Xmx150000m -Xms150000m
Not including fragment map
Error while reading graphs file: java.io.FileNotFoundException: /lustre/groups/itg/teams/zeggini/projects/child_diabesity/analysis/initial_sample/qc/Juicer/L79851_Track-123454/aligned/inter_hists.m (No such file or directory)
Start preprocess
Writing header
Writing body

Is the afterdedup stage properly recognized?

[agilly]

may be linked to issue #255

[sa501428]

Exception in thread "main" java.lang.RuntimeException: Unknown command: statistics at juicebox.tools.HiCTools.main(HiCTools.java:98) Is due to using an old Juicer Tools jar that is not compatible with Juicer 2. You'll need to grab the latest from Github Releases.

[agilly]

Can you confirm which version of the tools I need?

Currently, I have Juicer Tools Version 1.9.9 when I run the CPU/common/juicer_tools.jar.

I used https://github.com/aidenlab/juicer/wiki/Installation as an install guide.

The most recent release in GH is Juicer 1.6.

[sa501428]

If it's requesting the statistics command, try using a new Juicer Tools jar, e.g. https://github.com/aidenlab/Juicebox/releases/tag/v.2.13.07

Did you clone the github repo for Juicer, or download the 1.6 release?

[agilly]

I cloned the github repo but for Juicer. Will try with the release you recommend, however, what you sent me is Juicebox. Did I try to use the wrong software?

[sa501428]

You'll see a juicer_tools.jar listed under the assets for that release (even though it's in the Juicebox github repo). This is mentioned here: https://github.com/aidenlab/juicer/wiki/Download#version-2xxxx

If you cloned the repo, you are not using Juicer 1.6, but rather the latest Juicer 2 under development. See here: https://github.com/aidenlab/juicer#read-this-first

[agilly]

All right. Perhaps it would make sense to add this to the Juicer installation wiki, as it currently specifically instructs users to clone the repo, and then download juicer tools 1.9.9

[Aannaw]

Hello About the step "afterdedup", have you solved? Recently I also am stuck in the problem. When I use -S afterdedup, the command can not run and generates the USAGE.... Best wishes!

[sa501428]

@Aannaw can you make a separate github issue post and include more details about the exact error you are getting?

[Aannaw]

Hello @sa501428 This is the problem I met when I use juicer version 2.0. However I have changed juicer version 1.6 by SLURM. I encounter another issue I have submit the #267. I will create new issue to describe the problem. Could you give me any suggestions about #267 . Best wishes!

[zhangyangfan]

Try to use the latest juicer_ tools。 I have also had such problems, which have been solved.

wget https://github.com/aidenlab/Juicebox/releases/download/v2.17.00/juicer_tools_2.17.00.jar

[qdu-beep]

You'll see a juicer_tools.jar listed under the assets for that release (even though it's in the Juicebox github repo). This is mentioned here: https://github.com/aidenlab/juicer/wiki/Download#version-2xxxx

If you cloned the repo, you are not using Juicer 1.6, but rather the latest Juicer 2 under development. See here: https://github.com/aidenlab/juicer#read-this-first

Hello @sa501428 I hope you can help me to confirm whether my operation is correct, thank you for your help！ I cloned the github repo for Juicer. (git clone https://github.com/aidenlab/juicer.git) And, I placed this Git repository in software/ directory. In another directory juicer_work/ I created, I set up the references/ , restriction_sites/ , and work/ directories.

In juicer_work/ : ln -s /home/dq/software/juicer/CPU scripts I have the same problem as agilly, I downloaded the Juicer Tools jar 1.9.9 in the juicer_work/scripts/common/ . However, I found that the file juicer/work/aligned/inter.hic is still being generated continuously, and other files have not been generated new content after the error is reported, such as "inter.txt".

If I should download the latest version and re-run juicer.sh?

In the juicer_work/scripts/common/ : wget https://github.com/aidenlab/Juicebox/releases/download/v2.20.00/juicer_tools.2.20.00.jar ln -s juicer_tools.2.20.00.jar juicer_tools.jar

export PATH=/home/dq/software/juicer/:$PATH export PATH=/home/dq/software/miniconda3/envs/juicer/bin/:$PATH /juicer_work/scripts/juicer.sh
-g genomic
-z /juicer_work/references/genomic.fna
-p /juicer_work/restriction_sites/chrom.sizes
-y /juicer_work/restriction_sites/DpnII.txt
-s DpnII
-d /juicer_work/work
-D /juicer_work/
-t 90

Thank you so much for taking the time to answer my question, I have another doubt.

In the output log file: " (-: Aligning files matching /juicer_work/work/fastq/_R.fastq* to genome /juicer_work/references/genomic.fna with no fragment delimited maps." In the website https://github.com/aidenlab/juicer/wiki/Installation: "(If you don't use fragment delimited resolutions, you must run Juicer with the -x flag)"

I'm not sure if it's my restriction file not being used during the juicer run.

My output log file content is as follows: Using /juicer_work/restriction_sites/DpnII.txt as site file (-: Looking for fastq files...fastq files exist (-: Created /juicer_work/work/splits. (-: Aligning files matching /juicer_work/work/fastq/_R.fastq* to genome /juicer_work/references/genomic.fna with no fragment delimited maps. Running command bwa mem -SP5M -t 90 -R @RG\tID:13271\tSM:HiC_sample\tPL:ILM\tLB:HiC_library /juicer_work/references/genomic.fna /juicer_work/work/splits/13271_R1.fastq.gz /juicer_work/work/splits/13271_R2.fastq.gz > /juicer_work/work/splits/13271.fastq.gz.sam (-: Align of /juicer_work/work/splits/13271.fastq.gz.sam done successfully (-: Finished sorting all sorted files into a single merge. (-: Mark duplicates done successfully Not including fragment map Start preprocess Writing header Writing body

My error output log file content is as follows: [main] Version: 0.7.17-r1188 [main] CMD: bwa mem -K 320000000 -SP5M -t 90 -R @RG\tID:13271\tSM:HiC_sample\tPL:ILM\tLB:HiC_library /juicer_work/references/genomic.fna /juicer_work/work/splits/13271_R1.fastq.gz /juicer_work/work/splits/13271_R2.fastq.gz [main] Real time: 29331.845 sec; CPU: 2567696.332 sec [bam_sort_core] merging from 17 files and 90 in-memory blocks... Picked up _JAVA_OPTIONS: -Xmx1024m -Xms1024m Exception in thread "main" java.lang.RuntimeException: Unknown command: statistics at juicebox.tools.HiCTools.main(HiCTools.java:98) Picked up _JAVA_OPTIONS: -Xmx1024m -Xms1024m Exception in thread "main" java.lang.RuntimeException: Unknown command: statistics at juicebox.tools.HiCTools.main(HiCTools.java:98) Picked up _JAVA_OPTIONS: -Xmx150000m -Xms150000m Error while reading graphs file: java.io.FileNotFoundException: /juicer_work/work/aligned/inter_hists.m (No such file or directory)