apache
[Bug] Code that is generated with the protobuf version 3.22.0+ doesn't work and logs a IllegalAccessError exception
Search before asking
- [X] I searched in the issues and found nothing similar.
Version
3.0.0
Minimal reproduce step
add the pulsar-client-3.0.0.jar as a dependency, try to create an admin client
What did you expect to see?
no warnings or a warning that reports a reason for method emptyList() to be unavailable
What did you see instead?
java.lang.IllegalAccessError: class com.company.grpc.inventory.Command tried to access method 'com.google.protobuf.LazyStringArrayList com.google.protobuf.LazyStringArrayList.emptyList()' (com.company.grpc.inventory.Command and com.google.protobuf.LazyStringArrayList are in unnamed module of loader 'app’)
Anything else?
I also use dependencies:
- JavaVersion = '17’
- protobufJavaVersion = '3.22.3'
- grpcJavaVersion = '1.54.2'
I think I have this problem because Pulsar uses protobuf version 3.19.6, which the emptyList() method is package private https://github.com/protocolbuffers/protobuf/blob/5cba162a5d93f8df786d828621019e03e50edb4f/java/core/src/main/java/com/google/protobuf/LazyStringArrayList.java#L70 They have exposed the method to the public since 3.22.0 protocolbuffers/protobuf@c658e27
When I use these dependencies, I don't have this problem:
- protobufJavaVersion = '3.21.7'
- grpcJavaVersion = '1.54.2'
Do you know about such a bug and in which release it will be fixed?
Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
I don't think this is a bug. It's more lack of documentation. :)
It seems that protobuf and grpc codegen should be done using a codegen version of the oldest version of protobuf and grpc-java to support at runtime. Generated code is designed to be compatible with newer runtimes, but not necessarily compatible with older runtimes. This is explained in comment https://github.com/grpc/grpc-java/issues/10202#issuecomment-1553739615 .
Could you use 3.19.6 protoc / 1.55.3 protoc-gen-grpc-java to generate the Java classes? Using a newer protobuf & grpc library at runtime should be fine.
Actually protobuf's compatibility is more limited than what I mentioned in my previous comment.
The cross version runtime compatibility is documented here: https://protobuf.dev/support/cross-version-runtime-guarantee/
Protobuf will not support mixing generated code and runtimes across major version boundaries. We will add “poison pills” where possible to detect these mismatches.
@artsiomau The problem you have reported doesn't seem to be a problem with the Pulsar client. Do you agree?
I'd recommend using a compatible grpc & protobuf version. One way to find out is to look in poms at https://search.maven.org/artifact/io.grpc/grpc-protobuf for a specific version. For example in grpc 1.58.0 , the protobuf-java version is 3.24.0 . If you happen to have dependencies to an older protobuf version, it might be better to downgrade the grpc version to a compatible version. For example, grpc 1.57.2 uses protobuf-java 3.22.3
I'm having similar issues. We have a protobuf project which has been building with protoc 3.21.7 and also provides transitive dependencies for protobuf-java-util:3.21.7 and protobuf-java:3.21.7. There is a security vulnerability in protobuf-java-util:3.21.7 in its transitive dependencies.
I upgraded to 4.26.1 for all 3 libraries. The protobuf library we build worked fine in a project that doesn't use pulsar, but in the project that does use pulsar, I get: "tried to access method 'com.google.protobuf.LazyStringArrayList com.google.protobuf.LazyStringArrayList.emptyList()'" errors.
The problem is the pulsar-client-all jar is including that com.google.protobuf.LazyStringArrayList class.
So I cannot upgrade to the latest protobuf libraries and use pulsar client in the same project. I think this is a bug. I think pulsar should shade the classes from protobuf that it is including in its jar.
The problem is the pulsar-client-all jar is including that com.google.protobuf.LazyStringArrayList class.
So I cannot upgrade to the latest protobuf libraries and use pulsar client in the same project. I think this is a bug. I think pulsar should shade the classes from protobuf that it is including in its jar.
@davidtrobinson You might be able to workaround the problem by depending on pulsar-client-original and pulsar-admin-original dependencies instead of pulsar-client-all, pulsar-client or pulsar-admin. pulsar-client-original and pulsar-admin-original are unshaded. Those work in most cases when used with a recent compatible Netty version.
Thank you for the tip. I am using spring-boot-starter-pulsar. It was including the pulsar-client-all.jar as a dependency. I tried excluding the pulsar-client-all jar and then adding the original jars like you said but I get a weird class cast exception:
Caused by: java.lang.LinkageError: ClassCastException: attempting to castjar:file:/Users/drobinson/.gradle/caches/modules-2/files-2.1/javax.ws.rs/javax.ws.rs-api/2.1/426a0862406536e690c7caa8bb6ed32191986fac/javax.ws.rs-api-2.1.jar!/javax/ws/rs/client/ClientBuilder.class to jar:file:/Users/drobinson/.gradle/caches/modules-2/files-2.1/javax.ws.rs/javax.ws.rs-api/2.1/426a0862406536e690c7caa8bb6ed32191986fac/javax.ws.rs-api-2.1.jar!/javax/ws/rs/client/ClientBuilder.class
at javax.ws.rs.client.ClientBuilder.newBuilder(ClientBuilder.java:105) ~[javax.ws.rs-api-2.1.jar:2.1]
at org.apache.pulsar.client.admin.internal.PulsarAdminImpl.
It's weird because the classes are identical. I'm giving up on this for now and I'll just use the older protobuf version. Hopefully a better solution comes along in the future.