[MSHADE-260] shading does not rewrite Lambda deserializers

[jira-importer]

James Moore opened MSHADE-260 and commented

In our spark applications, we've encountered issues around shading and lambdas. Where the javac inserted deserialization function ( included below ) will not be updated to reflect the shaded libraries The relocation rule is the following

<relocation>
  <pattern>com.google.protobuf</pattern>
  <shadedPattern>com.google.protobuf.v3</shadedPattern>
</relocation>

But while the majority of the signatures are updated to com.google.protobuf.v3 in the class file, the following byte code shows the string where it is not.

(Ljava/lang/Class;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Lcom/google/protobuf/Message

I believe this problem affects any shaded library which has a classed passed into a serialized lambda instance.

private static java.lang.Object $deserializeLambda$(java.lang.invoke.SerializedLambda);
  Code:
     0: aload_0
     1: invokevirtual #64                 // Method java/lang/invoke/SerializedLambda.getImplMethodName:()Ljava/lang/String;
     4: astore_1
     5: iconst_m1
     6: istore_2
     7: aload_1
     8: invokevirtual #70                 // Method java/lang/String.hashCode:()I
    11: lookupswitch  { // 1
          1182156382: 28
             default: 39
        }
    28: aload_1
    29: ldc           #71                 // String lambda$convert$60ebd0cc$1
    31: invokevirtual #75                 // Method java/lang/String.equals:(Ljava/lang/Object;)Z
    34: ifeq          39
    37: iconst_0
    38: istore_2
    39: iload_2
    40: lookupswitch  { // 1
                   0: 60
             default: 143
        }
    60: aload_0
    61: invokevirtual #78                 // Method java/lang/invoke/SerializedLambda.getImplMethodKind:()I
    64: bipush        6
    66: if_icmpne     143
    69: aload_0
    70: invokevirtual #81                 // Method java/lang/invoke/SerializedLambda.getFunctionalInterfaceClass:()Ljava/lang/String;
    73: ldc           #83                 // String org/apache/spark/api/java/function/Function
    75: invokevirtual #84                 // Method java/lang/Object.equals:(Ljava/lang/Object;)Z
    78: ifeq          143
    81: aload_0
    82: invokevirtual #87                 // Method java/lang/invoke/SerializedLambda.getFunctionalInterfaceMethodName:()Ljava/lang/String;
    85: ldc           #88                 // String call
    87: invokevirtual #84                 // Method java/lang/Object.equals:(Ljava/lang/Object;)Z
    90: ifeq          143
    93: aload_0
    94: invokevirtual #91                 // Method java/lang/invoke/SerializedLambda.getFunctionalInterfaceMethodSignature:()Ljava/lang/String;
    97: ldc           #92                 // String (Ljava/lang/Object;)Ljava/lang/Object;
    99: invokevirtual #84                 // Method java/lang/Object.equals:(Ljava/lang/Object;)Z
   102: ifeq          143
   105: aload_0
   106: invokevirtual #95                 // Method java/lang/invoke/SerializedLambda.getImplClass:()Ljava/lang/String;
   109: ldc           #96                 // String StockHubSpotClass
   111: invokevirtual #84                 // Method java/lang/Object.equals:(Ljava/lang/Object;)Z
   114: ifeq          143
   117: aload_0
   118: invokevirtual #99                 // Method java/lang/invoke/SerializedLambda.getImplMethodSignature:()Ljava/lang/String;
   121: ldc           #101                // String (Ljava/lang/Class;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Lcom/google/protobuf/Message;
   123: invokevirtual #84                 // Method java/lang/Object.equals:(Ljava/lang/Object;)Z
   126: ifeq          143
   129: aload_0
   130: iconst_0
   131: invokevirtual #105                // Method java/lang/invoke/SerializedLambda.getCapturedArg:(I)Ljava/lang/Object;
   134: checkcast     #107                // class java/lang/Class
   137: invokedynamic #46,  0             // InvokeDynamic #0:call:(Ljava/lang/Class;)Lorg/apache/spark/api/java/function/Function;
   142: areturn
   143: new           #109                // class java/lang/IllegalArgumentException
   146: dup
   147: ldc           #111                // String Invalid lambda deserialization
   149: invokespecial #114                // Method java/lang/IllegalArgumentException."<init>":(Ljava/lang/String;)V
   152: athrow

---

Affects: 3.0.0, 3.1.0

Issue Links:

• FLINK-18006 It will throw Invalid lambda deserialization Exception when writing to elastic search with new format ("causes")
• FLINK-28043 "Invalid lambda deserialization" in AvroParquetReaders ("causes")

3 votes, 5 watchers

[jira-importer]

Lorenzo Affetti commented

Hello people!

This bug seems quite a big deal and It happens to me also on version 3.2.4 (java11).

Is somebody working on this?

Is this fixed in later versions?