SmallPedigree-WGS System.ArgumentOutOfRangeException: Index was out of range.

[curtisjamison]

Hi -

We encountered an error running canvas Germline WGS. The bam seems to be okay as both the strelka and Manta VCF both have data that has passed filter, and the ploidy file has been used multiple time without incident. (reference 1055 is GCHr37) Any help/advice would be greatly appreciated.

Cheers, Curt

Calling parameters: 2019-04-23T12:42:58+00:00,Running Canvas Germline-WGS 1.39.0.1598+master 2019-04-23T12:42:58+00:00,Command-line arguments: Germline-WGS -b ../sorted1055.bam --sample-b-allele-vcf ../sift.vcf.gz -o canvasOut -r /mnt/data/refData/1055/1055.kmer.fa -g /mnt/data/refData/1055 -n default -f /mnt/data/refData/1055/filter13.bed --ploidy-vcf /mnt/data/refData/1055/female-ploidy.vcf

$ cat CanvasError.txt 2019-04-23T13:17:04+00:00,ERROR: Job CNV.vcf.gz failed with exit code 134. Job logs: /mnt/data/workDir/1156157/canvasRun/canvasOut/Logging/CNV.vcf.gz.stdout /mnt/data/workDir/1156157/canvasRun/canvasOut/Logging/CNV.vcf.gz.stderr Job error message: Unhandled Exception: System.InvalidOperationException: Sequence contains no elements at System.Linq.Enumerable.First[TSource](IEnumerable1 source) at CanvasCommon.CanvasSegmentWriter.WriteVariants(IEnumerable1 segmentsOfAllSamples, List1 ploidies, GenomeMetadata genome, BgzipOrS treamWriter writer, Nullable1 denovoQualityThreshold) in D:\TeamCity\buildAgent\work\a29a190a11771d97\Src\Canvas\CanvasCommon\CanvasSegm entWriter.cs:line 124 at CanvasCommon.CanvasSegmentWriter.WriteSegments(String outVcfPath, List1 segments, Nullable1 diploidCoverage, String wholeGenomeFa staDirectory, String sampleName, List1 extraHeaders, PloidyInfo ploidy, Int32 qualityThreshold, Boolean isPedigreeInfoSupplied, Nullable 1 denovoQualityThreshold, Nullable1 sizeThreshold) in D:\TeamCity\buildAgent\work\a29a190a11771d97\Src\Canvas\CanvasCommon\CanvasSegmen tWriter.cs:line 283 at CanvasDiploidCaller.CanvasDiploidCaller.CallVariants(String variantFrequencyFile, String inFile, String outFile, String ploidyVcfPa th, String referenceFolder, String sampleName, String truthDataPath) in D:\TeamCity\buildAgent\work\a29a190a11771d97\Src\Canvas\CanvasDip loidCaller\CanvasDiploidCaller.cs:line 290 at CanvasDiploidCaller.Program.Main(String[] args) in D:\TeamCity\buildAgent\work\a29a190a11771d97\Src\Canvas\CanvasDiploidCaller\Prog ram.cs:line 98 2019-04-23T13:17:04+00:00,ERROR: Canvas workflow error: Isas.Framework.WorkManagement.JobFailedException: Job CNV.vcf.gz failed with exit code 134 at Isas.Framework.WorkManagement.JobLaunching.BasicJobLauncher.RunSystemProcessCheckExitAsync(JobInfo job, Action1 stdoutCallback, Ac tion1 stderrCallback) at Isas.Framework.WorkManagement.JobLaunching.RuntimeTrackingJobLauncher.LaunchJobAsync(JobInfo jobInfo, Action1 stdoutCallback, Acti on1 stderrCallback) at Isas.Framework.WorkManagement.IJobLauncherExtensions.LaunchJob(IJobLauncher jobLauncher, JobInfo jobInfo, Action1 stdoutCallback, Action1 stderrCallback) at Isas.Framework.WorkManagement.IWorkDoerJobExtensions.<>c__DisplayClass0_01.<DoWork>b__0(WorkResources sr, IJobLauncher jobLauncher ) at Isas.Framework.WorkManagement.ResourceManagement.ThreadedWorkResourceManager.<>c__DisplayClass11_01.<RunWithResources>b__3(WorkRes ources resources) --- End of stack trace from previous location where exception was thrown --- at Isas.Framework.WorkManagement.ResourceManagement.ThreadedWorkResourceManager.<RunWithResources>b__11_1[T](Task1 resourceRequestTas k, Task1 functionTask) at Isas.Framework.WorkManagement.AsTaskExtensions.Await[T](Task1 task) at Canvas.CanvasRunner.RunGermlineCalling(IFileLocation partitionedPath, CanvasCallset callset, String ploidyVcfPath, IFileLocation ca nvasSnvPath) at Isas.Framework.Checkpointing.Legacy.LegacyCheckpointRunner.<>c__DisplayClass0_0.<RunCheckpoint>b__0() at Isas.Framework.Checkpointing.Internals.SerializingCheckpointRunner.<>c__DisplayClass8_01.<RunCheckpoint>b__0(Checkpoint checkpoint ) at Isas.Framework.Checkpointing.Internals.CoreCheckpointRunner.RunCheckpoint[TResult](String name, Func2 run) at Isas.Framework.Checkpointing.Legacy.LegacyCheckpointRunner.RunCheckpoint(ICheckpointRunner runner, String name, Action a) at Canvas.CanvasRunner.CallSampleInternal(CanvasCallset callset) at Canvas.CanvasRunner.CallSample(CanvasCallset callset) at Canvas.ModeLauncher.Launch()

[eroller]

Germline-WGS is deprecated. Please run SmallPedigree-WGS, even for a single sample.

[curtisjamison]

Hmm. Okay, thanks. Will switch to that.

[curtisjamison]

The SmallPedigree-WGS workflow also failed:

2019-04-24T21:16:29+00:00,ERROR: Job default.partitioned failed with exit code 134. Job logs: /mnt/data/workDir/1156157/canvasRun/canvasOut/Logging/default.partitioned.stdout /mnt/data/workDir/1156157/canvasRun/canvasOut/Logging/default.partitioned.stderr Job error message: Unhandled Exception: System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. Parameter name: index at CanvasCommon.Utilities.Quartiles(List1 x) in D:\TeamCity\buildAgent\work\a29a190a11771d97\Src\Canvas\CanvasCommon\Utilities.cs:line 382 at CanvasPartition.HiddenMarkovModelsRunner.Run(List1 segmentation, Boolean isPerSample) in D:\TeamCity\buildAgent\work\a29a190a11771d97\Src\Canvas\CanvasPartition\HiddenMarkovModelsRunner.cs:line 45 at CanvasPartition.CanvasPartition.Main(String[] args) in D:\TeamCity\buildAgent\work\a29a190a11771d97\Src\Canvas\CanvasPartition\CanvasPartition.cs:line 170 2019-04-24T21:16:29+00:00,ERROR: Canvas workflow error: Isas.Framework.WorkManagement.JobFailedException: Job default.partitioned failed with exit code 134 at Isas.Framework.WorkManagement.JobLaunching.BasicJobLauncher.RunSystemProcessCheckExitAsync(JobInfo job, Action1 stdoutCallback, Action1 stderrCallback) at Isas.Framework.WorkManagement.JobLaunching.RuntimeTrackingJobLauncher.LaunchJobAsync(JobInfo jobInfo, Action1 stdoutCallback, Action1 stderrCallback) at Isas.Framework.WorkManagement.IJobLauncherExtensions.LaunchJob(IJobLauncher jobLauncher, JobInfo jobInfo, Action1 stdoutCallback, Action1 stderrCallback) at Isas.Framework.WorkManagement.IWorkDoerJobExtensions.<>c__DisplayClass0_01.<DoWork>b__0(WorkResources sr, IJobLauncher jobLauncher) at Isas.Framework.WorkManagement.ResourceManagement.ThreadedWorkResourceManager.<>c__DisplayClass11_01.<RunWithResources>b__3(WorkResources resources) --- End of stack trace from previous location where exception was thrown --- at Isas.Framework.WorkManagement.ResourceManagement.ThreadedWorkResourceManager.<RunWithResources>b__11_1[T](Task1 resourceRequestTask, Task1 functionTask) at Isas.Framework.WorkManagement.AsTaskExtensions.Await[T](Task1 task) at Canvas.CanvasRunner.InvokeCanvasPartitionMultisample(SmallPedigreeCallset callsets, List1 cleanedPaths, String canvasBedPath, String commonCnvsBed) at Isas.Framework.Checkpointing.Internals.SerializingCheckpointRunner.<>c__DisplayClass8_01.<RunCheckpoint>b__0(Checkpoint checkpoint) at Isas.Framework.Checkpointing.Internals.CoreCheckpointRunner.RunCheckpoint[TResult](String name, Func2 run) at Canvas.CanvasRunner.CallSampleInternal(SmallPedigreeCallset callset) at Canvas.CanvasRunner.CallPedigree(SmallPedigreeCallset callset) at Canvas.ModeLauncher.Launch()

[curtisjamison]

Hi Guys -- any word on having the same failure with small-pedigree?

[eroller]

This error is happening because there don't appear to be any coverage values to operate on to perform segmentation.

Can you check the input files to the partition command that is failing? I believe it is the .cleaned file that contains the coverage values. Is that file empty? We have seen weird behavior when there was a mismatch between the chromosome names in the bam vs. the reference genome. Make sure they are all the same convention (e.g. hg19 vs GRCh37).

[curtisjamison]

Hi Eric - We've set up a pipeline of Isaac/Strelka/Canvas/Manta. The BAM is directly from Isaac and all using the same GRCh37 reference. So far we've run 500+ files through the pipeline with only this one failure.

Looking at the input files, I see no real coverage values in the default.cleaned.

ll default.cleaned -rw-r--r-- 1 root root 20 Apr 24 21:16 default.cleaned more default.cleaned

[eroller]

Is this a reproducible error or maybe something strange going on with the filesystem?

If this is reproducible then the coverage data must be getting dropped somewhere. Does the .binned file have the expected coverage data. For WGS it should be hundreds of MB in size.

[curtisjamison]

Hi Eric - This is very reproducible -- it has happened on two different virtual machines and on multiple retries.

The bin job only runs for a minute: Job default_0.binned duration: 00:01:14.1 and the .binned file is empty, but the chromosome files are populated. ll canvasOut3/TempCNV total 3165264 drwxr-xr-x 2 root root 4096 May 3 21:22 ./ drwxr-xr-x 6 root root 4096 May 3 21:02 ../ -rw-r--r-- 1 root root 20 May 3 21:22 default_0.binned -rw-r--r-- 1 root root 280406991 May 3 21:03 default_0_chr1.dat -rw-r--r-- 1 root root 152476636 May 3 21:04 default_0_chr10.dat -rw-r--r-- 1 root root 151882376 May 3 21:04 default_0_chr11.dat -rw-r--r-- 1 root root 150583427 May 3 21:04 default_0_chr12.dat -rw-r--r-- 1 root root 129566158 May 3 21:05 default_0_chr13.dat -rw-r--r-- 1 root root 120768278 May 3 21:05 default_0_chr14.dat -rw-r--r-- 1 root root 115347859 May 3 21:05 default_0_chr15.dat -rw-r--r-- 1 root root 101649143 May 3 21:06 default_0_chr16.dat -rw-r--r-- 1 root root 91344657 May 3 21:06 default_0_chr17.dat -rw-r--r-- 1 root root 87836947 May 3 21:06 default_0_chr18.dat -rw-r--r-- 1 root root 66520151 May 3 21:21 default_0_chr19.dat -rw-r--r-- 1 root root 273599337 May 3 21:03 default_0_chr2.dat -rw-r--r-- 1 root root 70903753 May 3 21:06 default_0_chr20.dat -rw-r--r-- 1 root root 54146177 May 3 21:21 default_0_chr21.dat -rw-r--r-- 1 root root 57717682 May 3 21:21 default_0_chr22.dat -rw-r--r-- 1 root root 222775276 May 3 21:03 default_0_chr3.dat -rw-r--r-- 1 root root 215048603 May 3 21:03 default_0_chr4.dat -rw-r--r-- 1 root root 203529710 May 3 21:03 default_0_chr5.dat -rw-r--r-- 1 root root 192504493 May 3 21:04 default_0_chr6.dat -rw-r--r-- 1 root root 179031038 May 3 21:04 default_0_chr7.dat -rw-r--r-- 1 root root 164659567 May 3 21:05 default_0_chr8.dat -rw-r--r-- 1 root root 158865152 May 3 21:05 default_0_chr9.dat

[eroller]

So something odd is happening when merging the bin data from each of the chromosomes. Can you check the Logging directory to see if there is anything in stderr or stdout for the CanvasBin invocations?

[curtisjamison]

Is the bin count supposed to have a minus sign in front?

04/24/2019 08:57:09 Deserialization complete -2147483648 binSize 04/24/2019 08:57:31 Launch BinCountsForChromosome jobs... 04/24/2019 08:57:39 Completed BinCountsForChromosome jobs. 04/24/2019 08:57:39 Output binned counts: 04/24/2019 08:57:39 Output complete

[eroller]

-2147483648 is int.MinValue and is probably the result of overflow in the bin calculation. I haven't seen this before and I haven't been able to figure out how it could happen. If you are able to share the binned data I could dig deeper to see what is happening.

[curtisjamison]

Thanks for looking into this deeper. I can share the binned data. Please let me know which folder to tar up and I'll get onto a share drive for you.

[eroller]

I should only need the .dat files. Can you also send the last command line for CanvasBin.dll . It should contain each of the .dat files along with some other parameters.

[eroller]

@osowiecki That might be unrelated to this issue. Can you create a separate issue and include additional log messages including the contents of any files mentioned in the error log.

[osowiecki]

@osowiecki That might be unrelated to this issue. Can you create a separate issue and include additional log messages including the contents of any files mentioned in the error log.

It must be related to https://github.com/Illumina/canvas/issues/92 And you can find my extended post there.