google-cloud-java icon indicating copy to clipboard operation
google-cloud-java copied to clipboard
verified publisher icon googleapis

Java VerifyError occurs when using com.google.cloud libraries-bom > 26.31.0

Open rnaval opened this issue 1 year ago • 13 comments

Environment details

  1. API tested: CloudFilestoreManagerClient.create(), SubscriptionAdminClient.create()
  2. OS type and version: CentOS 7 docker
  3. Java version: Java 17
  4. Version(s):
  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>com.google.cloud</groupId>
        <artifactId>libraries-bom</artifactId>
        <version>26.37.0</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-monitoring</artifactId>
    </dependency>
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-compute</artifactId>
    </dependency>
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-storage</artifactId>
    </dependency>
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-filestore</artifactId>
    </dependency>
    <dependency>
      <groupId>com.google.protobuf</groupId>
      <artifactId>protobuf-java-util</artifactId>
      <version>4.26.1</version>
    </dependency>
  </dependencies>

Code example

  private CloudFilestoreManagerClient createClient() {
    try {
      return CloudFilestoreManagerClient.create();
    } catch (Exception e) {
      logger.get().error("{} Error when creating client for CloudFilestoreManagerClient : {}",
          this.getClass().getSimpleName(), e.getMessage());
    }
    return null;
  }

Stack trace

java.lang.VerifyError: Bad return type
Exception Details:
  Location:
    com/google/cloud/filestore/v1/Instance.internalGetMapFieldReflection(I)Lcom/google/protobuf/MapFieldReflectionAccessor; @24: areturn
  Reason:
    Type 'com/google/protobuf/MapField' (current frame, stack[0]) is not assignable to 'com/google/protobuf/MapFieldReflectionAccessor' (from method signature)
  Current Frame:
    bci: @24
    flags: { }
    locals: { 'com/google/cloud/filestore/v1/Instance', integer }
    stack: { 'com/google/protobuf/MapField' }
  Bytecode:
    0000000: 1bab 0000 0000 0018 0000 0001 0000 0009
    0000010: 0000 0013 2ab7 0004 b0bb 001d 59bb 001e
    0000020: 59b7 001f 1220 b600 211b b600 22b6 0023
    0000030: b700 24bf                              
  Stackmap Table:
    same_frame(@20)
    same_frame(@25)

	at com.google.cloud.filestore.v1.stub.GrpcCloudFilestoreManagerStub.<clinit>(GrpcCloudFilestoreManagerStub.java:88)
	at com.google.cloud.filestore.v1.stub.CloudFilestoreManagerStubSettings.createStub(CloudFilestoreManagerStubSettings.java:494)
	at com.google.cloud.filestore.v1.CloudFilestoreManagerClient.<init>(CloudFilestoreManagerClient.java:517)
	at com.google.cloud.filestore.v1.CloudFilestoreManagerClient.create(CloudFilestoreManagerClient.java:499)
	at com.google.cloud.filestore.v1.CloudFilestoreManagerClient.create(CloudFilestoreManagerClient.java:490)

Another one:

java.lang.VerifyError: Bad return type
Exception Details:
Location:
com/google/pubsub/v1/Subscription.internalGetMapFieldReflection(I)Lcom/google/protobuf/MapFieldReflectionAccessor; @24: areturn
Reason:
Type 'com/google/protobuf/MapField' (current frame, stack[0]) is not assignable to 'com/google/protobuf/MapFieldReflectionAccessor' (from method signature)
Current Frame:
bci: @24
flags: { }
locals: { 'com/google/pubsub/v1/Subscription', integer }
stack: { 'com/google/protobuf/MapField' }
Bytecode:
0000000: 1bab 0000 0000 0018 0000 0001 0000 0009
0000010: 0000 0013 2ab7 0007 b0bb 0069 59bb 006b
0000020: 59b7 006d 126e b600 701b b600 74b6 0077
0000030: b700 7bbf
Stackmap Table:
same_frame(@20)
same_frame(@25)

    at com.google.cloud.pubsub.v1.stub.GrpcSubscriberStub.<clinit>(GrpcSubscriberStub.java:78)
    at com.google.cloud.pubsub.v1.stub.SubscriberStubSettings.createStub(SubscriberStubSettings.java:376)
    at com.google.cloud.pubsub.v1.SubscriptionAdminClient.<init>(SubscriptionAdminClient.java:544)
    at com.google.cloud.pubsub.v1.SubscriptionAdminClient.create(SubscriptionAdminClient.java:526)
    at com.google.cloud.pubsub.v1.SubscriptionAdminClient.create(SubscriptionAdminClient.java:517)

Any additional information below

Issue only occurs when we use 26.32.0 onwards. 26.31.0 works fine.

Unit tests seem to be working fine for all versions, but not sure what is happening in the docker image.

rnaval avatar Apr 15 '24 09:04 rnaval

Hi @rnaval, thanks for reporting this. Would you mind sharing the tag of the docker image you are using? Has this occurred when running your java application on your host machine? Thanks!

diegomarquezp avatar Apr 16 '24 18:04 diegomarquezp

It looks like protobuf-java-util may be involved (thanks @suztomo).

diegomarquezp avatar Apr 16 '24 18:04 diegomarquezp

@rnaval our libraries are not using protobuf 4.26 just yet. Would you mind trying to modify your dependency on java-protobuf-util to ~3.25.3~ 3.25.2 (the one our libraries use)?

From https://github.com/googleapis/java-cloud-bom/releases/tag/v26.37.0 image

diegomarquezp avatar Apr 16 '24 19:04 diegomarquezp

Thanks for the response.

Tried it out with the recommended protobuf library, issue still occurs.

java.lang.VerifyError: Bad return type
Exception Details:
  Location:
    com/google/cloud/filestore/v1/Instance.internalGetMapFieldReflection(I)Lcom/google/protobuf/MapFieldReflectionAccessor; @24: areturn
  Reason:
    Type 'com/google/protobuf/MapField' (current frame, stack[0]) is not assignable to 'com/google/protobuf/MapFieldReflectionAccessor' (from method signature)
  Current Frame:
    bci: @24
    flags: { }
    locals: { 'com/google/cloud/filestore/v1/Instance', integer }
    stack: { 'com/google/protobuf/MapField' }
  Bytecode:
    0000000: 1bab 0000 0000 0018 0000 0001 0000 0009
    0000010: 0000 0013 2ab7 0004 b0bb 001d 59bb 001e
    0000020: 59b7 001f 1220 b600 211b b600 22b6 0023
    0000030: b700 24bf                              
  Stackmap Table:
    same_frame(@20)
    same_frame(@25)

	at com.google.cloud.filestore.v1.stub.GrpcCloudFilestoreManagerStub.<clinit>(GrpcCloudFilestoreManagerStub.java:88)
	at com.google.cloud.filestore.v1.stub.CloudFilestoreManagerStubSettings.createStub(CloudFilestoreManagerStubSettings.java:494)
	at com.google.cloud.filestore.v1.CloudFilestoreManagerClient.<init>(CloudFilestoreManagerClient.java:517)
	at com.google.cloud.filestore.v1.CloudFilestoreManagerClient.create(CloudFilestoreManagerClient.java:499)
	at com.google.cloud.filestore.v1.CloudFilestoreManagerClient.create(CloudFilestoreManagerClient.java:490)

Docker image is based on CentOS 7:

FROM centos:7
...

Here's a look at the dependency tree:

INFO] +- com.google.cloud:google-cloud-monitoring:jar:3.42.0:compile
[INFO] |  +- io.grpc:grpc-api:jar:1.62.2:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] |  +- com.google.errorprone:error_prone_annotations:jar:2.26.1:compile
[INFO] |  +- io.grpc:grpc-stub:jar:1.62.2:compile
[INFO] |  +- io.grpc:grpc-protobuf:jar:1.62.2:compile
[INFO] |  +- io.grpc:grpc-protobuf-lite:jar:1.62.2:runtime
[INFO] |  +- com.google.api:api-common:jar:2.29.1:compile
[INFO] |  +- com.google.auto.value:auto-value-annotations:jar:1.10.4:compile
[INFO] |  +- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] |  +- com.google.j2objc:j2objc-annotations:jar:3.0.0:compile
[INFO] |  +- com.google.protobuf:protobuf-java:jar:3.25.2:compile
[INFO] |  +- com.google.api.grpc:proto-google-common-protos:jar:2.37.1:compile
[INFO] |  +- com.google.api.grpc:proto-google-cloud-monitoring-v3:jar:3.42.0:compile
[INFO] |  +- com.google.guava:guava:jar:33.1.0-jre:provided
[INFO] |  +- com.google.guava:failureaccess:jar:1.0.2:compile
[INFO] |  +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] |  +- org.checkerframework:checker-qual:jar:3.42.0:compile
[INFO] |  +- com.google.api:gax:jar:2.46.1:compile
[INFO] |  +- com.google.auth:google-auth-library-credentials:jar:1.23.0:compile
[INFO] |  +- io.opencensus:opencensus-api:jar:0.31.1:compile
[INFO] |  +- io.grpc:grpc-context:jar:1.62.2:compile
[INFO] |  +- io.opentelemetry:opentelemetry-api:jar:1.36.0:compile
[INFO] |  +- io.opentelemetry:opentelemetry-context:jar:1.36.0:compile
[INFO] |  +- com.google.api:gax-grpc:jar:2.46.1:compile
[INFO] |  +- io.grpc:grpc-inprocess:jar:1.62.2:compile
[INFO] |  +- io.grpc:grpc-core:jar:1.62.2:compile
[INFO] |  +- com.google.android:annotations:jar:4.1.1.4:runtime
[INFO] |  +- org.codehaus.mojo:animal-sniffer-annotations:jar:1.23:runtime
[INFO] |  +- io.grpc:grpc-alts:jar:1.62.2:compile
[INFO] |  +- io.grpc:grpc-grpclb:jar:1.62.2:compile
[INFO] |  +- org.conscrypt:conscrypt-openjdk-uber:jar:2.5.2:compile
[INFO] |  +- io.grpc:grpc-auth:jar:1.62.2:compile
[INFO] |  +- io.grpc:grpc-netty-shaded:jar:1.62.2:compile
[INFO] |  +- io.grpc:grpc-util:jar:1.62.2:runtime
[INFO] |  +- io.perfmark:perfmark-api:jar:0.27.0:runtime
[INFO] |  +- io.grpc:grpc-googleapis:jar:1.62.2:runtime
[INFO] |  +- io.grpc:grpc-xds:jar:1.62.2:runtime
[INFO] |  +- io.opencensus:opencensus-proto:jar:0.2.0:runtime
[INFO] |  +- io.grpc:grpc-services:jar:1.62.2:runtime
[INFO] |  +- com.google.re2j:re2j:jar:1.7:runtime
[INFO] |  +- org.threeten:threetenbp:jar:1.6.8:compile
[INFO] |  +- com.google.auth:google-auth-library-oauth2-http:jar:1.23.0:compile
[INFO] |  +- com.google.http-client:google-http-client:jar:1.44.1:compile
[INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.5.14:compile
[INFO] |  +- commons-codec:commons-codec:jar:1.16.1:compile
[INFO] |  +- org.apache.httpcomponents:httpcore:jar:4.4.16:compile
[INFO] |  +- io.opencensus:opencensus-contrib-http-util:jar:0.31.1:compile
[INFO] |  +- com.google.http-client:google-http-client-gson:jar:1.44.1:compile
[INFO] |  \- com.google.code.gson:gson:jar:2.10.1:compile
[INFO] +- com.google.cloud:google-cloud-compute:jar:1.51.0:compile
[INFO] |  +- com.google.api.grpc:proto-google-cloud-compute-v1:jar:1.51.0:compile
[INFO] |  \- com.google.api:gax-httpjson:jar:2.46.1:compile
[INFO] +- com.google.cloud:google-cloud-storage:jar:2.36.1:compile
[INFO] |  +- com.google.http-client:google-http-client-jackson2:jar:1.44.1:compile
[INFO] |  +- com.google.api-client:google-api-client:jar:2.4.0:compile
[INFO] |  +- com.google.oauth-client:google-oauth-client:jar:1.35.0:compile
[INFO] |  +- com.google.http-client:google-http-client-apache-v2:jar:1.44.1:compile
[INFO] |  +- com.google.apis:google-api-services-storage:jar:v1-rev20240311-2.0.0:compile
[INFO] |  +- com.google.cloud:google-cloud-core:jar:2.36.1:compile
[INFO] |  +- com.google.cloud:google-cloud-core-http:jar:2.36.1:compile
[INFO] |  +- com.google.http-client:google-http-client-appengine:jar:1.44.1:compile
[INFO] |  +- com.google.cloud:google-cloud-core-grpc:jar:2.36.1:compile
[INFO] |  +- com.google.api.grpc:proto-google-iam-v1:jar:1.32.1:compile
[INFO] |  +- com.google.api.grpc:proto-google-cloud-storage-v2:jar:2.36.1-alpha:compile
[INFO] |  +- com.google.api.grpc:grpc-google-cloud-storage-v2:jar:2.36.1-alpha:compile
[INFO] |  +- com.google.api.grpc:gapic-google-cloud-storage-v2:jar:2.36.1-alpha:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-core:jar:2.16.1:compile
[INFO] |  \- io.grpc:grpc-rls:jar:1.62.2:runtime
[INFO] +- com.google.cloud:google-cloud-filestore:jar:1.42.0:compile
[INFO] |  +- com.google.api.grpc:proto-google-cloud-filestore-v1:jar:1.42.0:compile
[INFO] |  \- com.google.api.grpc:proto-google-cloud-filestore-v1beta1:jar:0.44.0:compile
[INFO] +- com.google.protobuf:protobuf-java-util:jar:3.25.2:compile

rnaval avatar Apr 17 '24 03:04 rnaval

@rnaval Do you observe the problem only happens in the CentOS docker container?

suztomo avatar Apr 17 '24 14:04 suztomo

cc: @suztomo Hi @rnaval, I created a reproducer based on our comments here.

I could run the main function without any issues in IntelliJ. +1 to @suztomo asking if the problem occurs in your host machine as well.

diegomarquezp avatar Apr 17 '24 14:04 diegomarquezp

Sorry for the late response. Based on testing results, we have verified that it works on:

  • MacOS host machine
  • Windows host machine

We also tried it on a Centos 7 VM - same error was encountered.

rnaval avatar Apr 29 '24 06:04 rnaval

@diegomarquezp @suztomo Also tried it out with Ubuntu 22.04 docker, seems to work.

rnaval avatar May 17 '24 05:05 rnaval

What's the image tag of this CentOS image? Are you using a dockerfile with this image as base or are you running your code directly on it? Maybe a small reproducer such as a bash script showing how you setup your code can help us find out what's missing in this CentOS image.

If no dockerfile is involved, then we should look at the jdk versions in each environment you tested, or more generally the differences in each execution environment.

diegomarquezp avatar May 17 '24 05:05 diegomarquezp

I have the exact same issue after doing 3.25.1 --> 3.25.3.

mdetemad avatar Jun 17 '24 16:06 mdetemad

The circleCI used to generate the docs for our project is failing due to this issue as well although the dependency is from Google's Beam SDK :

    [org.apache.beam/beam-sdks-java-io-google-cloud-platform "2.57.0"

       [com.google.protobuf/protobuf-java-util "3.25.3" :exclusions [[org.checkerframework/jdk8] [com.google.guava/guava-jdk5] [jdk.tools] [com.google.protobuf/protobuf-lite] [org.hamcrest/hamcrest-all]]]

https://app.circleci.com/pipelines/github/cljdoc/builder/53299/workflows/d2e886c4-683d-415a-a8dd-040a88745c5f/jobs/69674

edporras avatar Jul 17 '24 09:07 edporras