[Bug]: Google BigQuery - ERROR: com.google.common.util.concurrent.ExecutionError: java.lang.NoSuchMethodError: 'java.lang.String com.google.auth.Credentials.getUniverseDomain()'

[Tianoklein]

Apache Hop version?

2.9.0

Java version?

java 22 2024-03-19

Operating system

Windows

What happened?

When I set a limit of 39,999 on a query, it works fine. However, if I remove the limit or change it to => 40,000, the query fails with the following error:

"ERROR: com.google.common.util.concurrent.ExecutionError: java.lang.NoSuchMethodError: 'java.lang.String com.google.auth.Credentials.getUniverseDomain()'"

Issue Priority

Priority: 0

Issue Component

Component: Database

[Tianoklein]

Does anyone have a chance to check this? Is this a bug or is there a workaround?

[hansva]

.take-issue

[hansva]

which libraries have you copied over from the Simba JDBC driver? I think you will have to replace the google-auth-library-credentials and google-auth-library-oauth2-http under lib/beam folder

[Tianoklein]

Hi, Thanks for your help.

I'm using the version 1.23.0 in both folders (\hop\lib\beam) and (\hop\plugins\databases\googlebigquery\lib): Below the output for all files from:

\hop\plugins\databases\googlebigquery\lib: api-common-2.32.0.jar gax-2.49.0.jar gax-grpc-2.43.0.jar google-api-client-2.3.0.jar google-api-services-bigquery-v2-rev20240211-2.0.0.jar google-auth-library-oauth2-http-1.23.0.jar google-cloud-bigquerystorage-3.2.0.jar google-http-client-jackson2-1.42.3.jar google-oauth-client-1.36.0.jar GoogleBigQueryJDBC42.jar grpc-google-cloud-bigquerystorage-v1-3.2.0.jar grpc-google-cloud-bigquerystorage-v1beta1-0.174.0.jar grpc-google-cloud-bigquerystorage-v1beta2-0.174.0.jar json-20240205.jar proto-google-cloud-bigquerystorage-v1-3.2.0.jar proto-google-cloud-bigquerystorage-v1beta1-0.174.0.jar proto-google-cloud-bigquerystorage-v1beta2-0.174.0.jar threetenbp-1.6.9.jar

\hop\lib\beam: akka-protobuf_2.11-2.5.32.jar akka-slf4j_2.11-2.5.32.jar antlr4-runtime-4.7.jar auto-value-annotations-1.8.2.jar avro-1.11.3.jar beam-model-job-management-2.56.0.jar beam-model-pipeline-2.56.0.jar beam-runners-direct-java-2.56.0.jar beam-sdks-java-core-2.56.0.jar beam-vendor-bytebuddy-1_11_0-0.1.jar beam-vendor-grpc-1_60_1-0.2.jar beam-vendor-guava-32_1_2-jre-0.1.jar byte-buddy-1.12.18.jar classgraph-4.8.162.jar commons-codec-1.15.jar commons-compress-1.26.0.jar commons-io-2.15.1.jar commons-lang3-3.9.jar commons-logging-1.1.3.jar compress-lzf-1.1.jar conscrypt-openjdk-uber-2.5.2.jar curator-client-5.4.0.jar curator-framework-5.4.0.jar curator-recipes-5.4.0.jar error_prone_annotations-2.10.0.jar google-api-client-2.2.0.jar google-auth-library-credentials-1.23.0.jar google-auth-library-oauth2-http-1.23.0.jar google-http-client-1.42.3.jar google-http-client-gson-1.42.3.jar grpc-alts-1.60.1.jar grpc-api-1.60.1.jar grpc-auth-1.60.1.jar grpc-context-1.60.1.jar grpc-core-1.60.1.jar grpc-grpclb-1.60.1.jar grpc-netty-1.60.1.jar grpc-netty-shaded-1.60.1.jar grpc-protobuf-1.60.1.jar grpc-protobuf-lite-1.60.1.jar grpc-stub-1.60.1.jar gson-2.10.jar guava-32.1.2-jre.jar hamcrest-2.1.jar hop-engine-beam-2.9.0.jar httpclient-4.5.13.jar httpcore-4.4.15.jar jackson-annotations-2.15.0.jar joda-time-2.12.1.jar json-simple-1.1.1.jar json4s-ast_2.12-3.7.0-M11.jar json4s-core_2.12-3.7.0-M11.jar json4s-jackson_2.12-3.7.0-M11.jar json4s-scalap_2.12-3.7.0-M11.jar jsr305-3.0.2.jar kryo-5.3.0.jar kryo-shaded-4.0.2.jar log4j-api-2.19.0.jar log4j-core-2.19.0.jar metrics-core-4.2.12.jar metrics-jvm-4.2.12.jar metrics-logback-4.2.12.jar minlog-1.3.1.jar netty-all-4.1.89.Final.jar netty-buffer-4.1.89.Final.jar netty-codec-4.1.89.Final.jar netty-codec-dns-4.1.89.Final.jar netty-codec-haproxy-4.1.89.Final.jar netty-codec-http-4.1.89.Final.jar netty-codec-http2-4.1.89.Final.jar netty-codec-memcache-4.1.89.Final.jar netty-codec-mqtt-4.1.89.Final.jar netty-codec-redis-4.1.89.Final.jar netty-codec-smtp-4.1.89.Final.jar netty-codec-socks-4.1.89.Final.jar netty-codec-stomp-4.1.89.Final.jar netty-codec-xml-4.1.89.Final.jar netty-common-4.1.89.Final.jar netty-handler-4.1.89.Final.jar netty-handler-proxy-4.1.89.Final.jar netty-handler-ssl-ocsp-4.1.89.Final.jar netty-resolver-4.1.89.Final.jar netty-resolver-dns-4.1.89.Final.jar netty-resolver-dns-classes-macos-4.1.89.Final.jar netty-transport-4.1.89.Final.jar netty-transport-classes-epoll-4.1.89.Final.jar netty-transport-classes-kqueue-4.1.89.Final.jar netty-transport-native-unix-common-4.1.89.Final.jar netty-transport-rxtx-4.1.89.Final.jar netty-transport-sctp-4.1.89.Final.jar netty-transport-udt-4.1.89.Final.jar opencensus-api-0.31.1.jar opencensus-contrib-http-util-0.31.1.jar paranamer-2.8.jar perfmark-api-0.26.0.jar proto-google-common-protos-2.14.0.jar protobuf-java-3.23.2.jar protobuf-java-util-3.23.2.jar scala-library-2.12.17.jar slf4j-api-2.0.4.jar woodstox-core-6.4.0.jar zookeeper-3.8.4.jar zstd-jni-1.5.2-2.jar

Tks!

[hansva]

You still have the error? it seems we still shipped /hop/lib/beam/google-auth-library-credentials-1.12.1.jar and this one did not have that Method

[Tianoklein]

Hi, google-auth-library-credentials-1.12.1.jar is the original and I get the original error. Today I updated to google-auth-library-credentials-1.23.0.jar and I still have the same error.

[hansva]

yup, it seems the Simba dependencies are not matching with our own at the moment and there is no easy fix for this. I'll try and get it sorted out but there won't be a short term fix

[hansva]

I suggest using the bigquery input, this one works with the Hop engine too

[Tianoklein]

Unless I'm wrong, "Beam BigQuery Input" doesn't work with "Local Engine" - only on DataFlow engine - which is too expensive for me for now. Please let me know if there is anything I can do to work around this issue.

Tks

[hansva]

works with the local engine, just tested it. And it is also marked this way in the documentation. https://hop.apache.org//manual/latest/pipeline/transforms/beambigqueryinput.html

[Tianoklein]

Hi, Ok. For Input they works well as a workaround. Thanks!

[hansva]

lowering Prio as there is a solution.

[shubhammapara-sprih]

Do we have a fix for this? I'm facing same issue inside, here's the stack trace

com.google.common.util.concurrent.ExecutionError: java.lang.NoSuchMethodError: 'java.lang.String com.google.auth.Credentials.getUniverseDomain()'
	at com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1388) ~[guava-31.1-jre.jar:?]
	at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1381) ~[guava-31.1-jre.jar:?]
	at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:53) ~[gax-2.49.0.jar:2.49.0]
	at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112) ~[gax-2.49.0.jar:2.49.0]
	at com.google.cloud.vertexai.generativeai.GenerativeModel.generateContent(GenerativeModel.java:400) ~[google-cloud-vertexai-1.5.0.jar:1.5.0]
	at com.google.cloud.vertexai.generativeai.GenerativeModel.generateContent(GenerativeModel.java:387) ~[google-cloud-vertexai-1.5.0.jar:1.5.0]
	at com.google.cloud.vertexai.generativeai.GenerativeModel.generateContent(GenerativeModel.java:374) ~[google-cloud-vertexai-1.5.0.jar:1.5.0]