Juicebox [BUG] Exception in thread "pool-2-thread-1" java.lang.ArrayIndexOutOfBoundsException: -1

Hi, I am running into the error below with hiccups. I am trying to reproduce some published data, which used defined settings with hiccups. I downloaded the pairsam file:

#columns: readID chrom1 pos1 chrom2 pos2 strand1 strand2 pair_type mapq1 mapq2
35363968    chr1    3000024    chr1    3000259    +    -    UU    10    14
23858251    chr1    3000025    chr1    3000259    +    -    UU    49    60

Then generated .hic file: "$juicer_tools" pre -r 1000,2500,5000,10000 -k KR,KR,KR,KR "$infile" "$outfile" "mm10"

Then tried to run hiccups:

"$juicer_tools" hiccups --cpu --threads 0 -k KR -f 0.1,0.1,0.1,0.1 -r 1000,2500,5000,10000 -i 10000,20000,30000,40000 -d 2500,5000,10000,20000 "$infile" "$workdir"/hiccups

Getting this message below. Thanks for your help!

WARNING - You are using the CPU version of HiCCUPS.
The GPU version of HiCCUPS is the official version and has been tested extensively.
The CPU version only searches for loops within 8MB (by default) of the diagonal and is still experimental.
Using 1 CPU thread(s)
Unable to assess map sparsity; continuing with HiCCUPS
Running HiCCUPS for resolution 10000
Data not available for Y at 10000 resolution
Exception in thread "pool-2-thread-1" java.lang.ArrayIndexOutOfBoundsException: -1
    at juicebox.tools.utils.juicer.hiccups.GPUController.runCPUVersion(GPUController.java:381)
    at juicebox.tools.utils.juicer.hiccups.GPUController.process(GPUController.java:141)
    at juicebox.tools.clt.juicer.HiCCUPS.runCoreCodeForHiCCUPS(HiCCUPS.java:521)
    at juicebox.tools.clt.juicer.HiCCUPS.access$200(HiCCUPS.java:158)
    at juicebox.tools.clt.juicer.HiCCUPS$1.run(HiCCUPS.java:434)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Exception in thread "pool-4-thread-1" java.lang.ArrayIndexOutOfBoundsException: -1
    at juicebox.tools.utils.juicer.hiccups.GPUController.runCPUVersion(GPUController.java:381)
    at juicebox.tools.utils.juicer.hiccups.GPUController.process(GPUController.java:141)
    at juicebox.tools.clt.juicer.HiCCUPS.runCoreCodeForHiCCUPS(HiCCUPS.java:521)
    at juicebox.tools.clt.juicer.HiCCUPS.access$200(HiCCUPS.java:158)
    at juicebox.tools.clt.juicer.HiCCUPS$1.run(HiCCUPS.java:434)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Running HiCCUPS for resolution 5000
Data not available for Y at 5000 resolution

May 20 '20 16:05 dgoekbuget

As suggested, I tried running just hiccups with "-c 1" and without 1000. It gave the same error.

Then I tried running pre as well with those settings and now pre also gave an error message:

WARN [2020-05-20T09:50:34,164] [Globals.java:138] [main] Development mode is enabled Using 1 CPU thread(s) Not including fragment map Start preprocess Writing header Writing body java.lang.ArrayIndexOutOfBoundsException: -1 at juicebox.data.ChromosomeHandler.getChromosomeFromIndex(ChromosomeHandler.java:289) at juicebox.tools.utils.original.Preprocessor.computeWholeGenomeMatrix(Preprocessor.java:577) at juicebox.tools.utils.original.Preprocessor.writeBody(Preprocessor.java:669) at juicebox.tools.utils.original.Preprocessor.preprocess(Preprocessor.java:419) at juicebox.tools.clt.old.PreProcessing.run(PreProcessing.java:153) at juicebox.tools.HiCTools.main(HiCTools.java:94)

May 20 '20 18:05 dgoekbuget

A .hic file was generated for "-c 1" option above, but size was a few hundred bytes and "pre" finishes after 10 seconds. The genome is mm10.

May 20 '20 18:05 dgoekbuget

Did you use -c1 or -c 1? Can you try -c chr1? Although I wasn't suggesting re-running Pre - I was saying to use -c 1 or -c chr1 with HiCCUPS CPU version.

May 20 '20 18:05 sa501428

Ok. Yes I used -c 1 and also just tried -c chr1 with HiCCUPS (just tried in parallel with "pre"). Unfortunately, it also resulted in error below. HiCCUPS generates two files for 5000 and 10000 resolution, which look like:

0	10000.0	10000.0	10000.0	10000.0
1	10000.0	10000.0	10000.0	10000.0
2	10000.0	10000.0	10000.0	10000.0

Call: "$juicer_tools" hiccups --cpu --threads 0 -c chr1 -k KR -f 0.1,0.1,0.1 -r 2500,5000,10000 -i 20000,30000,40000 -d 5000,10000,20000 "$infile" "$ workdir"/hiccups Stdout:

Discarding invalid configuration: Config res: 2500 peak: -1 window: 20000 fdr: 10% radius: 5000
Using the following configurations for HiCCUPS:
Config res: 10000 peak: 2 window: 40000 fdr: 10% radius: 20000
Config res: 5000 peak: 4 window: 30000 fdr: 10% radius: 10000
WARNING - You are using the CPU version of HiCCUPS.
The GPU version of HiCCUPS is the official version and has been tested extensively.
The CPU version only searches for loops within 8MB (by default) of the diagonal and is still experimental.
Using 1 CPU thread(s)
Unable to assess map sparsity; continuing with HiCCUPS
Running HiCCUPS for resolution 10000
Exception in thread "pool-2-thread-1" java.lang.ArrayIndexOutOfBoundsException: -1
	at juicebox.tools.utils.juicer.hiccups.GPUController.runCPUVersion(GPUController.java:381)
	at juicebox.tools.utils.juicer.hiccups.GPUController.process(GPUController.java:141)
	at juicebox.tools.clt.juicer.HiCCUPS.runCoreCodeForHiCCUPS(HiCCUPS.java:521)
	at juicebox.tools.clt.juicer.HiCCUPS.access$200(HiCCUPS.java:158)
	at juicebox.tools.clt.juicer.HiCCUPS$1.run(HiCCUPS.java:434)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Exception in thread "pool-4-thread-1" java.lang.ArrayIndexOutOfBoundsException: -1
	at juicebox.tools.utils.juicer.hiccups.GPUController.runCPUVersion(GPUController.java:381)
	at juicebox.tools.utils.juicer.hiccups.GPUController.process(GPUController.java:141)
	at juicebox.tools.clt.juicer.HiCCUPS.runCoreCodeForHiCCUPS(HiCCUPS.java:521)
	at juicebox.tools.clt.juicer.HiCCUPS.access$200(HiCCUPS.java:158)
	at juicebox.tools.clt.juicer.HiCCUPS$1.run(HiCCUPS.java:434)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Running HiCCUPS for resolution 5000
Exception in thread "pool-6-thread-1" java.lang.ArrayIndexOutOfBoundsException: -1
	at juicebox.tools.utils.juicer.hiccups.GPUController.runCPUVersion(GPUController.java:381)
	at juicebox.tools.utils.juicer.hiccups.GPUController.process(GPUController.java:141)
	at juicebox.tools.clt.juicer.HiCCUPS.runCoreCodeForHiCCUPS(HiCCUPS.java:521)
	at juicebox.tools.clt.juicer.HiCCUPS.access$200(HiCCUPS.java:158)
	at juicebox.tools.clt.juicer.HiCCUPS$1.run(HiCCUPS.java:434)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Exception in thread "pool-8-thread-1" java.lang.ArrayIndexOutOfBoundsException: -1
	at juicebox.tools.utils.juicer.hiccups.GPUController.runCPUVersion(GPUController.java:381)
	at juicebox.tools.utils.juicer.hiccups.GPUController.process(GPUController.java:141)
	at juicebox.tools.clt.juicer.HiCCUPS.runCoreCodeForHiCCUPS(HiCCUPS.java:521)
	at juicebox.tools.clt.juicer.HiCCUPS.access$200(HiCCUPS.java:158)
	at juicebox.tools.clt.juicer.HiCCUPS$1.run(HiCCUPS.java:434)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
HiCCUPS complete

May 20 '20 18:05 dgoekbuget

Does it still crash if you just do: "$juicer_tools" hiccups --cpu --threads 0 -r 5000,10000 "$infile" "$ workdir"/hiccups

May 20 '20 18:05 sa501428

It hasn't crashed and still runs, which is promising.. will report back when done.

May 20 '20 18:05 dgoekbuget

That suggests there is some issue with these non-standard resolutions (either 1kb or 2.5kb). Can you confirm if you open your file with Juicebox that it has those resolutions and nothing looks weird about the map?

Also, I just realized you're not calling the -p flag. When using custom resolutions, all flags need to be utilized so that it knows what the custom configurations are (you've already got i/f/d).

May 20 '20 18:05 sa501428

Program still running, and I see a bedpe file appearing in folder, which wasn't there before. 1 and 2.5 (which is seemingly displayed as 3) resolutions look fine to me in Juicebox (not an experience Juicebox user, though). Looking at 1 chromosome increasing resolutions essentially results in finer granularity of the interactions as one would expect.

Tried running it by including the -p parameter which I forgot (my mistake). Getting this message below. This time is generated these two files in addition, though: fdr_thresholds_1000, fdr_thresholds_2500

WARN [2020-05-20T14:35:03,750]  [Globals.java:138] [main]  Development mode is enabled
Reading file: /mnt/iscsi_speed/blelloch/deniz/hsieh2020/4DNFILZ1CPT8.pairs.hic
Using the following configurations for HiCCUPS:
Config res: 10000 peak: 20000 window: 40000 fdr: 10% radius: 20000
Config res: 2500 peak: 10000 window: 20000 fdr: 10% radius: 5000
Config res: 1000 peak: 4000 window: 10000 fdr: 10% radius: 2500
Config res: 5000 peak: 15000 window: 30000 fdr: 10% radius: 10000
WARNING - You are using the CPU version of HiCCUPS.
The GPU version of HiCCUPS is the official version and has been tested extensively.
The CPU version only searches for loops within 8MB (by default) of the diagonal and is still experimental.
Using 1 CPU thread(s)
Unable to assess map sparsity; continuing with HiCCUPS
Running HiCCUPS for resolution 10000
Data not available for Y at 10000 resolution
Exception in thread "pool-2-thread-1" java.lang.ArrayIndexOutOfBoundsException: -19980
	at juicebox.tools.utils.juicer.hiccups.GPUController.runCPUVersion(GPUController.java:333)
	at juicebox.tools.utils.juicer.hiccups.GPUController.process(GPUController.java:141)
	at juicebox.tools.clt.juicer.HiCCUPS.runCoreCodeForHiCCUPS(HiCCUPS.java:521)
	at juicebox.tools.clt.juicer.HiCCUPS.access$200(HiCCUPS.java:158)
	at juicebox.tools.clt.juicer.HiCCUPS$1.run(HiCCUPS.java:434)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Exception in thread "pool-4-thread-1" java.lang.ArrayIndexOutOfBoundsException: -19980
	at juicebox.tools.utils.juicer.hiccups.GPUController.runCPUVersion(GPUController.java:333)
	at juicebox.tools.utils.juicer.hiccups.GPUController.process(GPUController.java:141)
	at juicebox.tools.clt.juicer.HiCCUPS.runCoreCodeForHiCCUPS(HiCCUPS.java:521)
	at juicebox.tools.clt.juicer.HiCCUPS.access$200(HiCCUPS.java:158)
	at juicebox.tools.clt.juicer.HiCCUPS$1.run(HiCCUPS.java:434)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Running HiCCUPS for resolution 2500
Data not available for Y at 2500 resolution
Exception in thread "pool-6-thread-1" java.lang.ArrayIndexOutOfBoundsException: -9980
	at juicebox.tools.utils.juicer.hiccups.GPUController.runCPUVersion(GPUController.java:333)
	at juicebox.tools.utils.juicer.hiccups.GPUController.process(GPUController.java:141)
	at juicebox.tools.clt.juicer.HiCCUPS.runCoreCodeForHiCCUPS(HiCCUPS.java:521)
	at juicebox.tools.clt.juicer.HiCCUPS.access$200(HiCCUPS.java:158)
	at juicebox.tools.clt.juicer.HiCCUPS$1.run(HiCCUPS.java:434)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Exception in thread "pool-8-thread-1" java.lang.ArrayIndexOutOfBoundsException: -9980
	at juicebox.tools.utils.juicer.hiccups.GPUController.runCPUVersion(GPUController.java:333)
	at juicebox.tools.utils.juicer.hiccups.GPUController.process(GPUController.java:141)
	at juicebox.tools.clt.juicer.HiCCUPS.runCoreCodeForHiCCUPS(HiCCUPS.java:521)
	at juicebox.tools.clt.juicer.HiCCUPS.access$200(HiCCUPS.java:158)
	at juicebox.tools.clt.juicer.HiCCUPS$1.run(HiCCUPS.java:434)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Running HiCCUPS for resolution 1000
Data not available for Y at 1000 resolution
Exception in thread "pool-10-thread-1" java.lang.ArrayIndexOutOfBoundsException: -3980
	at juicebox.tools.utils.juicer.hiccups.GPUController.runCPUVersion(GPUController.java:333)
	at juicebox.tools.utils.juicer.hiccups.GPUController.process(GPUController.java:141)
	at juicebox.tools.clt.juicer.HiCCUPS.runCoreCodeForHiCCUPS(HiCCUPS.java:521)
	at juicebox.tools.clt.juicer.HiCCUPS.access$200(HiCCUPS.java:158)
	at juicebox.tools.clt.juicer.HiCCUPS$1.run(HiCCUPS.java:434)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Exception in thread "pool-12-thread-1" java.lang.ArrayIndexOutOfBoundsException: -3980
	at juicebox.tools.utils.juicer.hiccups.GPUController.runCPUVersion(GPUController.java:333)
	at juicebox.tools.utils.juicer.hiccups.GPUController.process(GPUController.java:141)
	at juicebox.tools.clt.juicer.HiCCUPS.runCoreCodeForHiCCUPS(HiCCUPS.java:521)
	at juicebox.tools.clt.juicer.HiCCUPS.access$200(HiCCUPS.java:158)
	at juicebox.tools.clt.juicer.HiCCUPS$1.run(HiCCUPS.java:434)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Running HiCCUPS for resolution 5000
Data not available for Y at 5000 resolution
Exception in thread "pool-14-thread-1" java.lang.ArrayIndexOutOfBoundsException: -14980
	at juicebox.tools.utils.juicer.hiccups.GPUController.runCPUVersion(GPUController.java:333)
	at juicebox.tools.utils.juicer.hiccups.GPUController.process(GPUController.java:141)
	at juicebox.tools.clt.juicer.HiCCUPS.runCoreCodeForHiCCUPS(HiCCUPS.java:521)
	at juicebox.tools.clt.juicer.HiCCUPS.access$200(HiCCUPS.java:158)
	at juicebox.tools.clt.juicer.HiCCUPS$1.run(HiCCUPS.java:434)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Exception in thread "pool-16-thread-1" java.lang.ArrayIndexOutOfBoundsException: -14980
	at juicebox.tools.utils.juicer.hiccups.GPUController.runCPUVersion(GPUController.java:333)
	at juicebox.tools.utils.juicer.hiccups.GPUController.process(GPUController.java:141)
	at juicebox.tools.clt.juicer.HiCCUPS.runCoreCodeForHiCCUPS(HiCCUPS.java:521)
	at juicebox.tools.clt.juicer.HiCCUPS.access$200(HiCCUPS.java:158)
	at juicebox.tools.clt.juicer.HiCCUPS$1.run(HiCCUPS.java:434)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
HiCCUPS complete

May 20 '20 21:05 dgoekbuget

Update on "$juicer_tools" hiccups --cpu --threads 0 -r 5000,10000 "$infile" "$ workdir"/hiccups It seemingly ran successfully.

May 20 '20 23:05 dgoekbuget

Any advise on how to run the original command successfully?

"$juicer_tools" hiccups --cpu --threads 0 -k KR -f 0.1,0.1,0.1,0.1 -r 1000,2500,5000,10000 -i 10000,20000,30000,40000 -d 2500,5000,10000,20000 -p 4000,10000,15000,20000 "$infile" "$workdir"/hiccups

May 26 '20 17:05 dgoekbuget

Not including fragment map Start preprocess Writing header Writing body java.io.IOException: Unexpected column count. Check file format at juicebox.tools.utils.original.AsciiPairIterator.advance(AsciiPairIterator.java:151) at juicebox.tools.utils.original.AsciiPairIterator.(AsciiPairIterator.java:75) at juicebox.tools.utils.original.Preprocessor.computeWholeGenomeMatrix(Preprocessor.java:482) at juicebox.tools.utils.original.Preprocessor.writeBody(Preprocessor.java:580) at juicebox.tools.utils.original.Preprocessor.preprocess(Preprocessor.java:346) at juicebox.tools.clt.old.PreProcessing.run(PreProcessing.java:116) at juicebox.tools.HiCTools.main(HiCTools.java:96) java.lang.RuntimeException: No reads in Hi-C contact matrices. This could be because the MAPQ filter is set too high (-q) or because all reads map to the same fragment. at juicebox.tools.utils.original.Preprocessor$MatrixZoomDataPP.mergeAndWriteBlocks(Preprocess or.java:1650) at juicebox.tools.utils.original.Preprocessor$MatrixZoomDataPP.access$000(Preprocessor.java:1 419) at juicebox.tools.utils.original.Preprocessor.writeMatrix(Preprocessor.java:832) at juicebox.tools.utils.original.Preprocessor.writeBody(Preprocessor.java:582) at juicebox.tools.utils.original.Preprocessor.preprocess(Preprocessor.java:346) at juicebox.tools.clt.old.PreProcessing.run(PreProcessing.java:116) at juicebox.tools.HiCTools.main(HiCTools.java:96)

请问这样的报错是哪里的问题

Apr 18 '23 03:04 xiaoyan-wang99