Reflection error with Unity when using il2cpp

[leohahn]

What version of protobuf and what language are you using? Version: v3.6.1 Language: C#

What operating system (Linux, Windows, ...) and version? MacOS Mojave 10.14

What runtime / compiler are you using (e.g., python version or gcc version) Apple LLVM version 10.0.0 (clang-1000.11.45.2) Target: x86_64-apple-darwin18.0.0 Thread model: posix

What did you do?

• Open the following project on Unity 2018.3.0b10 (I think that this happens with any Unity 2018, maybe other years as well)
• Run on the editor to see that the file Test.cs executes correctly.
• Compile with il2cpp to iOS and run. The code fails with the following stacktrace:

DescriptorValidationException: google.protobuf.Value.kind: Method ClearKind not found in Google.Protobuf.WellKnownTypes.Value
  at Google.Protobuf.Reflection.OneofDescriptor.CreateAccessor (System.String clrName) [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.Reflection.OneofDescriptor..ctor (Google.Protobuf.Reflection.OneofDescriptorProto proto, Google.Protobuf.Reflection.FileDescriptor file, Google.Protobuf.Reflection.MessageDescriptor parent, System.Int32 index, System.String clrName) [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.Reflection.MessageDescriptor+<>c__DisplayClass4_0.<.ctor>b__0 (Google.Protobuf.Reflection.OneofDescriptorProto oneof, System.Int32 index) [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.Reflection.DescriptorUtil.ConvertAndMakeReadOnly[TInput,TOutput] (System.Collections.Generic.IList`1[T] input, Google.Protobuf.Reflection.DescriptorUtil+IndexedConverter`2[TInput,TOutput] converter) [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.Reflection.MessageDescriptor..ctor (Google.Protobuf.Reflection.DescriptorProto proto, Google.Protobuf.Reflection.FileDescriptor file, Google.Protobuf.Reflection.MessageDescriptor parent, System.Int32 typeIndex, Google.Protobuf.Reflection.GeneratedClrTypeInfo generatedCodeInfo) [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.Reflection.FileDescriptor+<>c__DisplayClass1_0.<.ctor>b__0 (Google.Protobuf.Reflection.DescriptorProto message, System.Int32 index) [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.Reflection.DescriptorUtil.ConvertAndMakeReadOnly[TInput,TOutput] (System.Collections.Generic.IList`1[T] input, Google.Protobuf.Reflection.DescriptorUtil+IndexedConverter`2[TInput,TOutput] converter) [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.Reflection.FileDescriptor..ctor (Google.Protobuf.ByteString descriptorData, Google.Protobuf.Reflection.FileDescriptorProto proto, Google.Protobuf.Reflection.FileDescriptor[] dependencies, Google.Protobuf.Reflection.DescriptorPool pool, System.Boolean allowUnknownDependencies, Google.Protobuf.Reflection.GeneratedClrTypeInfo generatedCodeInfo) [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.Reflection.FileDescriptor.BuildFrom (Google.Protobuf.ByteString descriptorData, Google.Protobuf.Reflection.FileDescriptorProto proto, Google.Protobuf.Reflection.FileDescriptor[] dependencies, System.Boolean allowUnknownDependencies, Google.Protobuf.Reflection.GeneratedClrTypeInfo generatedCodeInfo) [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.Reflection.FileDescriptor.FromGeneratedCode (System.Byte[] descriptorData, Google.Protobuf.Reflection.FileDescriptor[] dependencies, Google.Protobuf.Reflection.GeneratedClrTypeInfo generatedCodeInfo) [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.WellKnownTypes.StructReflection..cctor () [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.WellKnownTypes.Value.get_Descriptor () [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.JsonParser..cctor () [0x00000] in <00000000000000000000000000000000>:0 
  at Test.Start () [0x00000] in <00000000000000000000000000000000>:0 
Rethrow as ArgumentException: Invalid embedded descriptor for "google/protobuf/struct.proto".
  at Google.Protobuf.Reflection.FileDescriptor.FromGeneratedCode (System.Byte[] descriptorData, Google.Protobuf.Reflection.FileDescriptor[] dependencies, Google.Protobuf.Reflection.GeneratedClrTypeInfo generatedCodeInfo) [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.WellKnownTypes.StructReflection..cctor () [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.WellKnownTypes.Value.get_Descriptor () [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.JsonParser..cctor () [0x00000] in <00000000000000000000000000000000>:0 
  at Test.Start () [0x00000] in <00000000000000000000000000000000>:0 
Rethrow as TypeInitializationException: The type initializer for 'Google.Protobuf.WellKnownTypes.StructReflection' threw an exception.
  at Google.Protobuf.WellKnownTypes.Value.get_Descriptor () [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.JsonParser..cctor () [0x00000] in <00000000000000000000000000000000>:0 
  at Test.Start () [0x00000] in <00000000000000000000000000000000>:0 
Rethrow as TypeInitializationException: The type initializer for 'Google.Protobuf.JsonParser' threw an exception.
  at Test.Start () [0x00000] in <00000000000000000000000000000000>:0 
 
(Filename: currently not available on il2cpp Line: -1)

[jskeet]

Hmm. I've managed to get the same code to pass in a standalone app with il2cpp:

using Google.Protobuf;
using System;

namespace TestApp
{
    class Program
    {
        static void Main(string[] args)
        {
            try
            {
                var res = (IMessage) Activator.CreateInstance(typeof(TestApp.Error));
                var msg = JsonParser.Default.Parse("{\"code\":\"PIT-000\",\"msg\":\"GetError is returning a custom error\"}", res.Descriptor);
                Console.WriteLine(msg);
            }
            catch (TypeInitializationException ex)
            {
                Console.WriteLine("Type initialization exception");
                Console.WriteLine(ex.InnerException);
            }
        }
    }
}

Output:

{ "code": "PIT-000", "msg": "GetError is returning a custom error" }

Here's the il2cpp invocation I'm using:

"$IL2CPP" --convert-to-cpp \
  $GENERIC_SHARING \
  -architecture=x64 \
  -configuration=Release \
  -platform=WindowsDesktop \
  --dotnetprofile=Net45 \
  --libil2cpp-static \
  --emit-null-checks \
  --enable-array-bounds-check \
  --enable-stats \
  --compile-cpp \
  --directory=$IL2CPP_INPUT \
  --generatedcppdir=$IL2CPP_OUTPUT \
  --outputpath=$IL2CPP_OUTPUT/Program.exe \
  --cachedirectory=$IL2CPP_CACHE \
  --libil2cpp-cache-directory=$IL2CPP_CACHE

That's using Google.Protobuf 3.6.1 as well. I don't know which il2cpp flags are used by Unity. Hmm.

[leohahn]

That's interesting. I'll maybe try with another version of Unity.

[jskeet]

The version of Unity I have installed is 2018.2.0b2. Unfortunately I can't see a way of finding a version of il2cpp...

Here's my complete build script, if you want to try my test app for yourself - change the directories in the obvious way:

#!/bin/bash

set -e

declare -r UNITY_HOME=/c/Program\ Files/Unity/Hub/Editor/2018.2.0b2
declare -r IL2CPP=$UNITY_HOME/Editor/Data/il2cpp/build/il2cpp.exe
declare -r AOT_ASSEMBLIES=$UNITY_HOME/Editor/Data/MonoBleedingEdge/lib/mono/unityaot
declare -r IL2CPP_CACHE=il2cppcache
declare -r IL2CPP_OUTPUT=il2cppoutput
declare -r IL2CPP_INPUT=il2cppinput
declare -r PROTOBUF_ROOT=../../packages/Google.Protobuf.Tools.3.5.1/tools
declare -r PROTOC=$PROTOBUF_ROOT/windows_x64/protoc.exe
#declare -r GENERIC_SHARING=--enable-primitive-value-type-generic-sharing


rm -rf $IL2CPP_OUTPUT
mkdir $IL2CPP_OUTPUT
rm -rf $IL2CPP_INPUT
mkdir $IL2CPP_INPUT

$PROTOC -ITestApp -I$(echo $PROTOBUF_ROOT | sed 's/\//\\/g') --csharp_out=TestApp TestApp/*.proto

dotnet build

cp TestApp/bin/Debug/net471/* $IL2CPP_INPUT
cp "$AOT_ASSEMBLIES/mscorlib.dll" $IL2CPP_INPUT
cp "$AOT_ASSEMBLIES/Mono.Security.dll" $IL2CPP_INPUT
cp "$AOT_ASSEMBLIES/System.dll" $IL2CPP_INPUT
cp "$AOT_ASSEMBLIES/System.Core.dll" $IL2CPP_INPUT
cp "$AOT_ASSEMBLIES/System.Xml.dll" $IL2CPP_INPUT


"$IL2CPP" --convert-to-cpp \
  $GENERIC_SHARING \
  -architecture=x64 \
  -configuration=Release \
  -platform=WindowsDesktop \
  --dotnetprofile=Net45 \
  --libil2cpp-static \
  --emit-null-checks \
  --enable-array-bounds-check \
  --enable-stats \
  --compile-cpp \
  --directory=$IL2CPP_INPUT \
  --generatedcppdir=$IL2CPP_OUTPUT \
  --outputpath=$IL2CPP_OUTPUT/Program.exe \
  --cachedirectory=$IL2CPP_CACHE \
  --libil2cpp-cache-directory=$IL2CPP_CACHE

echo Running...

$IL2CPP_OUTPUT/Program.exe

(I note that I'm using protoc from 3.5.1, but I'd be astonished if that's changed enough to be an issue.)

[aaron-bray]

Hey,

My team also recently had a few problems with C# reflections using Unity with il2cpp and enums using the 3.6.1 build. Note I am not the unity dev who dug into this, but this is a list of things we worked through. This sounds like the same issue as this, but maybe it's not?

1. That code accessed only through reflection was stripped because IL2CPP did not know it was needed, and that was solved by link.xml with the contents (DataModelBindings is a .NET lib of all my protoc generated code)

<linker> <assembly fullname="DataModelBindings" preserve="all"/> </linker>

This is probably something that should be mentioned in Protobuf documentation?

Relevant part of Unity documentation: https://docs.unity3d.com/Manual/IL2CPP-BytecodeStripping.html

2. Next this error came up:

Attempting to call method 'Google.Protobuf.Reflection.ReflectionUtil+ReflectionHelper`2[[Cdm.SerializeStateData, DataModelBindings, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null],[Cdm.eSerialization+Types+Type, DataModelBindings, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null]]::.ctor' for which no ahead of time (AOT) code was generated.

(eSerialization is an enum)

The issue appears to be with this file in particular: https://github.com/protocolbuffers/protobuf/blob/master/csharp/src/Google.Protobuf/Reflection/ReflectionUtil.cs GetReflectionHelper() with its call to Activator.CreateInstance() is where things seem to go wrong.

The use of generics through reflection + not letting the IL2CPP compiler know what classes will be needed is what caused the error - code was requested that was not generated, and there was no way to generate it on demand (as it would normally happen with JIT runtimes). Somewhat relevant part of Unity documentation: https://docs.unity3d.com/Manual/ScriptingRestrictions.html ("Ahead-of-time compile" section)

To get past this we ended up replacing the smarts and quickness of the generics via reflection in ReflectionUtils.cs with pretty simple code that calls invoke instead.

But I don't think this is going to be something we want in protubuf as invoke can be kind of slow, but there is probably some way to apply Google.Protobuf.Reflection.FileDescriptor.ForceReflectionInitialization to the custom types to make sure AOT code is generated for them but I'm not sure how that's done exactly, and how easy it would be.

Again, this issue description is a summary of notes of what we did over a week or so and maybe difficult to follow. Please let me know if parts need further explanation. But with all these changes we were able to create and pass protobuf JSON from C# to C++ in our packaged Unity UWP game.

[joshpeterson]

Unfortunately I can't see a way of finding a version of il2cpp

Note that we don't version il2cpp separately from Unity. The IL2CPP version is the same as the Unity editor version.

[na-kubota]

is there any update on this issue?

[jskeet]

@na-ka-na: Not from me, I'm afraid. This is only an area I can dip into occasionally, and this problem requires significantly more time. (This sort of brittleness is precisely why I've always been very nervous of supporting Unity - things that work on one version then failing on a different one.)

[na-kubota]

i try Unity 18.3.8f1 same Error

DescriptorValidationException: google.protobuf.Value.kind: Method ClearKind not found in Google.Protobuf.WellKnownTypes.Value
  at Google.Protobuf.Reflection.OneofDescriptor.CreateAccessor (System.String clrName) [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.Reflection.OneofDescriptor..ctor (Google.Protobuf.Reflection.OneofDescriptorProto proto, Google.Protobuf.Reflection.FileDescriptor file, Google.Protobuf.Reflection.MessageDescriptor parent, System.Int32 index, System.String clrName) [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.Reflection.MessageDescriptor+<>c__DisplayClass4_0.<.ctor>b__0 (Google.Protobuf.Reflection.OneofDescriptorProto oneof, System.Int32 index) [0x00000] in <00000000000000000000000000000000>:0 
  at Google.Protobuf.Reflection.DescriptorUtil.ConvertAndMakeReadOnly[TInput,TOutput] (System.Collections.Generic.IList`1[T] input, Google.Protobuf.Reflection.DescriptorUtil+IndexedConverter`2[TInput,TOutput] converter) [0x00000] in <0000000000

in this Simple Code

public class NewBehaviourScript : MonoBehaviour
{
    // Start is called before the first frame update
    void Start()
    {
        var data = ProtoData.Parser.ParseJson("{\"id\":1,\"name\":\"\u58f0\u512a1\"}");
    }
}

Is the cause Unity Version? please...

[jskeet]

Is the cause Unity Version?

Well sort of - in that I've had this working with some versions of Unity. But I'm not suggesting that you should change which version of Unity you're using. I'm suggesting that fixing this properly requires a significant amount of time and effort, ideally with automated testing against multiple versions of il2cpp.

[na-kubota]

ok, very thank you. What should I do?

[jskeet]

Unfortunately at the moment I would suggest avoiding any requirement on reflection - that's calling ToString() or parsing JSON - within Unity. Depending on the JSON and your protos, you may be able to use another JSON parser to populate your proto model, and then use it as normal with binary serialization, which doesn't require reflection.

[na-kubota]

ok i try other Parse thank you!

[leohahn]

I have managed to fix the issue with the latest version of protobuf by adding a link.xml file in the Assets folder of the project with the following contents:

<linker>
    <assembly fullname="Google.Protobuf" preserve="all"/>
</linker>

Since we also have to support .NET3.5, we compiled protobuf for that platform as well.

[jartsa]

I have managed to fix the issue with the latest version of protobuf by adding a link.xml file in the Assets folder of the project with the following contents:

<linker>
    <assembly fullname="Google.Protobuf" preserve="all"/>
</linker>

Since we also have to support .NET3.5, we compiled protobuf for that platform as well.

Thank You!

[ghost]

1. That code accessed only through reflection was stripped because IL2CPP did not know it was needed, and that was solved by link.xml with the contents (DataModelBindings is a .NET lib of all my protoc generated code)

<linker> <assembly fullname="DataModelBindings" preserve="all"/> </linker>

Fixed the issue for me - thanks!

[vitorbaraujo]

I'm having a similar issue: when trying to get a message descriptor I got a ExecutionEngineException error. This is the stack:

ExecutionEngineException: Attempting to call method 'Google.Protobuf.Reflection.ReflectionUtil+ReflectionHelper`2[[Google.Protobuf.Reflection.FieldDescriptorProto, Google.Protobuf, Version=3.6.1.0, Culture=neutral, PublicKeyToken=a7d26565bac4d604],[Google.Protobuf.Reflection.FieldDescriptorProto+Types+Label, Google.Protobuf, Version=3.6.1.0, Culture=neutral, PublicKeyToken=a7d26565bac4d604]]::.ctor' for which no ahead of time (AOT) code was generated.
  at System.Reflection.MonoCMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00000] in <filename unknown>:0 
  at Google.Protobuf.Reflection.ReflectionUtil.GetReflectionHelper (System.Type t1, System.Type t2) [0x00000] in <filename unknown>:0 
  at Google.Protobuf.Reflection.ReflectionUtil.CreateFuncIMessageObject (System.Reflection.MethodInfo method) [0x00000] in <filename unknown>:0 
  at Google.Protobuf.Reflection.FieldAccessorBase..ctor (System.Reflection.PropertyInfo property, Google.Protobuf.Reflection.FieldDescriptor descriptor) [0x00000] in <filename unknown>:0 
  at Google.Protobuf.Reflection.SingleFieldAccessor..ctor (System.Reflection.PropertyInfo property, Google.Protobuf.Reflection.FieldDescriptor descriptor) [0x00000] in <filename unknown>:0 
  at Google.Protobuf.Reflection.FieldDescriptor.CreateAccessor () [0x00000] in <filename unknown>:0 
  at Google.Protobuf.Reflection.FieldDescriptor.CrossLink () [0x00000] in <filename unknown>:0 
  at Google.Protobuf.Reflection.MessageDescriptor.CrossLink () [0x00000] in <filename unknown>:0 
  at Google.Protobuf.Reflection.FileDescriptor.CrossLink () [0x00000] in <filename unknown>:0 
  at Google.Protobuf.Reflection.FileDescriptor.BuildFrom (Google.Protobuf.ByteString descriptorData, Google.Protobuf.Reflection.FileDescriptorProto proto, Google.Protobuf.Reflection.FileDescriptor[] dependencies, Boolean allowUnknownDependencies, Google.Protobuf.Reflection.GeneratedClrTypeInfo generatedCodeInfo) [0x00000] in <filename unknown>:0 
  at Google.Protobuf.Reflection.FileDescriptor.FromGeneratedCode (System.Byte[] descriptorData, Google.Protobuf.Reflection.FileDescriptor[] dependencies, Google.Protobuf.Reflection.GeneratedClrTypeInfo generatedCodeInfo) [0x00000] in <filename unknown>:0 
  at Google.Protobuf.Reflection.DescriptorReflection..cctor () [0x00000] in <filename unknown>:0 
  at Google.Protobuf.Reflection.FileDescriptor.get_DescriptorProtoFileDescriptor () [0x00000] in <filename unknown>:0 

I can't seem to use FileDescriptor.ForceReflectionInitialization<T> here since Google.Protobuf.Reflection.FieldDescriptorProto.Types.Label enum is internal.

I'm using version 3.6.1 compiled to .NET 3.5 and using Unity 2018.4.23f1.

[github-actions[bot]]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please add a comment.

This issue is labeled inactive because the last activity was over 90 days ago.

[github-actions[bot]]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please reopen it.

This issue was closed and archived because there has been no new activity in the 14 days since the inactive label was added.