gatk icon indicating copy to clipboard operation
gatk copied to clipboard

Published 20 hours ago • verified publisher icon broadinstitute

Funcotator keep running but no results or com.google.cloud.storage Connection reset

Open kaixinxiaonvwa-hub opened this issue 2 years ago • 2 comments

Bug Report

Affected tool(s) or class(es) mutect2 and Funcotator

Affected version(s) gatk-4.1.8.1

Description

I tried to use Funcotator to annotate a vcf file. The vcf file had 436 records. I had download database funcotator_dataSources.v1.7.20200521s to my linux machine, and use --data-sources-path to appoint it

  1. when Funcotator with contidion "gnomAD_genome.tar.gz ",no results got. The error is message: All 20 retries failed. Waited a total of 1918000 ms between attempts Caused by: com.google.cloud.storage.StorageException: Connection reset Caused by: java.net.SocketException: Connection reset
  2. when Funcotator with contidion "gnomAD_genome ", results got but has nothing, this is a file with title annotation message but no records. The error is java.lang.IllegalArgumentException: Unexpected value: MANE_Select

###Steps to reproduce vcf file from mutect2-FilterMutectCalls,both snv and indel record

code1 java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -DGATK_STACKTRACE_ON_USER_EXCEPTION=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /root/software/gatk-4.1.8.1/gatk-package-4.1.8.1-local.jar Funcotator -R /data/Homo_sapiens/Homo_sapiens_38/Homo_sapiens_assembly38.fasta -V /data/project/naangda_panel_20230228/output/mutect/probe/normal_mutect/sssss4.filtered.mutect2.vcf -O /data/project/naangda_panel_20230228/output/mutect/Funcotator/sssss4.funcocator.maf --output-file-format MAF --data-sources-path /data/Homo_sapiens/Homo_sapiens_38/funcotator/funcotator_dataSources.v1.7.20200521s/ --ref-version hg38

code2 java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -DGATK_STACKTRACE_ON_USER_EXCEPTION=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /root/software/gatk-4.1.8.1/gatk-package-4.1.8.1-local.jar Funcotator -R /data/Homo_sapiens/Homo_sapiens_38/Homo_sapiens_assembly38.fasta -V /data/project/naangda_panel_20230228/output/mutect/probe/normal_mutect/sssss4.filtered.mutect2.vcf -O /data/project/naangda_panel_20230228/output/mutect/Funcotator/sssss4.funcocator.maf --output-file-format MAF --data-sources-path /data/Homo_sapiens/Homo_sapiens_38/funcotator/funcotator_dataSources.v1.7.20200521s/ --ref-version hg38

what's the problem with the program?

  1. I had download database, why error "com.google.cloud.storage.StorageException: Connection reset" happened?
  2. why I got only a file with title but no records like this, what should I do to solve this problem？

Funcotator 4.1.8.1 | Date 20233731T033757 | Gencode 34 CANONICAL | Achilles 110303 | ClinVar_VCF 20180429_hg38 | Cosmic v84 | CosmicFusion v84 | CosmicTissue v83 | Familial_Cancer_Genes 20110905 | Gencode_XHGNC 90_38 | Gencode_XRefSeq 90_38 | HGNC Nov302017 | Oreganno 20160119 | Simple_Uniprot 2014_12 | dbSNP 9606_b151

Hugo_Symbol Entrez_Gene_Id Center NCBI_Build Chromosome Start_Position End_Position Strand Variant_Classification Variant_Type Reference_Allele Tumor_Seq_Allele1 Tumor_Seq_Allele2 dbSNP_RSdbSNP_Val_Status Tumor_Sample_Barcode Matched_Norm_Sample_Barcode Match_Norm_Seq_Allele1 Match_Norm_Seq_Allele2 Tumor_Validation_Allele1 Tumor_Validation_Allele2 Match_Norm_Validation_Allele1 Match_Norm_Valida

kaixinxiaonvwa-hub avatar Mar 31 '23 03:03 kaixinxiaonvwa-hub

Hi - @kaixinxiaonvwa-hub . I have a couple of questions:

  • Can you post your full stack trace with the errors?
  • Did you attempt to enable gnomAD data sources or is it doing this without any changes to the data sources directory? Did you do any other configuration steps after downloading the datasources and before running funcotator?

If you enable gnomAD, the datasources are hosted on google cloud. If you don't have an internet connection or google cloud is blocked, Funcotator will not be able to connect to read the gnomAD data and will show the error in your 1 case above.

jonn-smith avatar Apr 03 '23 17:04 jonn-smith

Hello, Could you tell me the exact source websites of funcotator_dataSources.v1.7.20200521g? I did not find it in your Google Cloud (genomics-public-data). Besides, I used this code to download./gatk-4.1.9.0/gatk FuncotatorDataSourceDownloader --germline --validate-integrity --extract-after-download , but the error appeared as following:Nov 18, 2023 1:15:05 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine INFO: Failed to detect whether we are running on Google Compute Engine. 13:15:05.202 INFO FuncotatorDataSourceDownloader - ------------------------------------------------------------ 13:15:05.203 INFO FuncotatorDataSourceDownloader - The Genome Analysis Toolkit (GATK) v4.1.9.0 13:15:05.203 INFO FuncotatorDataSourceDownloader - For support and documentation go to https://software.broadinstitute.org/gatk/ 13:15:05.203 INFO FuncotatorDataSourceDownloader - Executing as yaoxq@mu01 on Linux v3.10.0-693.el7.x86_64 amd64 13:15:05.203 INFO FuncotatorDataSourceDownloader - Java runtime: Java HotSpot(TM) 64-Bit Server VM v1.8.0_151-b12 13:15:05.203 INFO FuncotatorDataSourceDownloader - Start Date/Time: November 18, 2023 1:15:04 PM CST 13:15:05.203 INFO FuncotatorDataSourceDownloader - ------------------------------------------------------------ 13:15:05.203 INFO FuncotatorDataSourceDownloader - ------------------------------------------------------------ 13:15:05.204 INFO FuncotatorDataSourceDownloader - HTSJDK Version: 2.23.0 13:15:05.204 INFO FuncotatorDataSourceDownloader - Picard Version: 2.23.3 13:15:05.204 INFO FuncotatorDataSourceDownloader - HTSJDK Defaults.COMPRESSION_LEVEL : 2 13:15:05.204 INFO FuncotatorDataSourceDownloader - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false 13:15:05.204 INFO FuncotatorDataSourceDownloader - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true 13:15:05.204 INFO FuncotatorDataSourceDownloader - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false 13:15:05.204 INFO FuncotatorDataSourceDownloader - Deflater: IntelDeflater 13:15:05.204 INFO FuncotatorDataSourceDownloader - Inflater: IntelInflater 13:15:05.204 INFO FuncotatorDataSourceDownloader - GCS max retries/reopens: 20 13:15:05.204 INFO FuncotatorDataSourceDownloader - Requester pays: disabled 13:15:05.204 INFO FuncotatorDataSourceDownloader - Initializing engine 13:15:05.205 INFO FuncotatorDataSourceDownloader - Done initializing engine 13:15:05.205 INFO FuncotatorDataSourceDownloader - Germline data sources selected. 13:15:05.207 INFO FuncotatorDataSourceDownloader - Collecting expected checksum... 13:19:33.264 INFO FuncotatorDataSourceDownloader - Shutting down engine [November 18, 2023 1:19:33 PM CST] org.broadinstitute.hellbender.tools.funcotator.FuncotatorDataSourceDownloader done. Elapsed time: 4.48 minutes. Runtime.totalMemory()=1967128576 code: 0 message: All 3 retries failed. Waited a total of 14000 ms between attempts reason: null location: null retryable: false com.google.cloud.storage.StorageException: All 3 retries failed. Waited a total of 14000 ms between attempts at com.google.cloud.storage.contrib.nio.CloudStorageRetryHandler.handleRetryForStorageException(CloudStorageRetryHandler.java:135) at com.google.cloud.storage.contrib.nio.CloudStorageRetryHandler.handleStorageException(CloudStorageRetryHandler.java:115) at com.google.cloud.storage.contrib.nio.CloudStorageReadChannel.fetchSize(CloudStorageReadChannel.java:253) at com.google.cloud.storage.contrib.nio.CloudStorageReadChannel.<init>(CloudStorageReadChannel.java:110) at com.google.cloud.storage.contrib.nio.CloudStorageReadChannel.create(CloudStorageReadChannel.java:90) at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.newReadChannel(CloudStorageFileSystemProvider.java:390) at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.newByteChannel(CloudStorageFileSystemProvider.java:305) at java.nio.file.Files.newByteChannel(Files.java:361) at java.nio.file.Files.newByteChannel(Files.java:407) at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.newInputStream(CloudStorageFileSystemProvider.java:482) at java.nio.file.Files.newInputStream(Files.java:152) at java.nio.file.Files.newBufferedReader(Files.java:2784) at java.nio.file.Files.lines(Files.java:3744) at java.nio.file.Files.lines(Files.java:3785) at org.broadinstitute.hellbender.tools.funcotator.FuncotatorDataSourceDownloader.readSha256SumFromPath(FuncotatorDataSourceDownloader.java:277) at org.broadinstitute.hellbender.tools.funcotator.FuncotatorDataSourceDownloader.downloadDataSources(FuncotatorDataSourceDownloader.java:252) at org.broadinstitute.hellbender.tools.funcotator.FuncotatorDataSourceDownloader.downloadAndValidateDataSources(FuncotatorDataSourceDownloader.java:230) at org.broadinstitute.hellbender.tools.funcotator.FuncotatorDataSourceDownloader.doWork(FuncotatorDataSourceDownloader.java:203) at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:140) at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:192) at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:211) at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:160) at org.broadinstitute.hellbender.Main.mainEntry(Main.java:203) at org.broadinstitute.hellbender.Main.main(Main.java:289) Caused by: com.google.cloud.storage.StorageException: connect timed out at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:229) at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:439) at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:244) at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:241) at shaded.cloud_nio.com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105) at shaded.cloud_nio.com.google.cloud.RetryHelper.run(RetryHelper.java:76) at shaded.cloud_nio.com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50) at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:240) at com.google.cloud.storage.contrib.nio.CloudStorageReadChannel.fetchSize(CloudStorageReadChannel.java:232) ... 22 more Caused by: java.net.SocketTimeoutException: connect timed out at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:673) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264) at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177) at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:162) at shaded.cloud_nio.com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:143) at shaded.cloud_nio.com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:79) at shaded.cloud_nio.com.google.api.client.http.HttpRequest.execute(HttpRequest.java:996) at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:541) at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:474) at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:591) at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:436) ... 29 more Thanks for your time!

wqrao avatar Nov 18 '23 08:11 wqrao