PSI on Oculus Quest (Android)

[emielch]

Hi! I'm trying to use PSI on an Oculus Quest in combination with StereoKit. To start, I cloned the StereoKit universal template (https://github.com/StereoKit/SKTemplate-Universal) and added a Microsoft.Psi.Runtime NuGet package reference to the StereoKitApp project. I adapted the first example from the Brief Introduction wiki page and added it to App.cs to test the basic functionality.

using Microsoft.Psi;
...
...
...
public void Init() {
...
...
...
    PsiTest();
}

public async void PsiTest() {
    await Task.Run(() => {
        try {
            using (var p = Pipeline.Create()) {
                var timer = Timers.Timer(p, TimeSpan.FromSeconds(0.1));
                timer.Do(t => Console.WriteLine(t));
                p.Run();
            }
        } catch (Exception ex) {
            Console.WriteLine("Message: ");
            Console.WriteLine(ex.Message);
            Console.WriteLine("InnerException: ");
            Console.WriteLine(ex.InnerException);
            Console.WriteLine("StackTrace: ");
            Console.WriteLine(ex.StackTrace);
            Console.WriteLine("");
        }
    });
}

When running this on desktop by setting the StereoKit_DotNet project as the startup project it works as expected; every 100ms a timestamp is printed to the console. When setting the StereoKit_Android project as the startup project and deploying it to the Oculus Quest, the following exception is caught:

Message: 
The type initializer for 'Microsoft.Psi.Serialization.KnownSerializers' threw an exception.
InnerException: 
System.NullReferenceException: Object reference not set to an instance of an object.
  at System.TypeSpec.Resolve (System.Func`2[T,TResult] assemblyResolver, System.Func`4[T1,T2,T3,TResult] typeResolver, System.Boolean throwOnError, System.Boolean ignoreCase, System.Threading.StackCrawlMark& stackMark) [0x0008a] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/corlib/System/TypeSpec.cs:322 
  at System.TypeNameParser.GetType (System.String typeName, System.Func`2[T,TResult] assemblyResolver, System.Func`4[T1,T2,T3,TResult] typeResolver, System.Boolean throwOnError, System.Boolean ignoreCase, System.Threading.StackCrawlMark& stackMark) [0x00006] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/corlib/ReferenceSources/TypeNameParser.cs:17 
  at System.Type.GetType (System.String typeName, System.Func`2[T,TResult] assemblyResolver, System.Func`4[T1,T2,T3,TResult] typeResolver) [0x00002] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/corlib/ReferenceSources/Type.cs:171 
  at Microsoft.Psi.TypeResolutionHelper.GetVerifiedType (System.String typeName) [0x00000] in <6bd4d33a21f248318e30737cf8f86d32>:0 
  at Microsoft.Psi.Serialization.KnownSerializers.RegisterGenericSerializer (System.Type genericSerializer) [0x00033] in <6bd4d33a21f248318e30737cf8f86d32>:0 
  at Microsoft.Psi.Serialization.KnownSerializers..ctor (System.Boolean isDefault, Microsoft.Psi.Common.RuntimeInfo runtimeVersion) [0x000ca] in <6bd4d33a21f248318e30737cf8f86d32>:0 
  at Microsoft.Psi.Serialization.KnownSerializers..cctor () [0x00073] in <6bd4d33a21f248318e30737cf8f86d32>:0 
StackTrace: 
  at (wrapper managed-to-native) System.Object.__icall_wrapper_mono_generic_class_init(intptr)
  at Microsoft.Psi.Serializer.IsImmutableType[T] () [0x00000] in <6bd4d33a21f248318e30737cf8f86d32>:0 
  at Microsoft.Psi.RecyclingPool.Create[T] (System.Diagnostics.StackTrace debugTrace) [0x00000] in <6bd4d33a21f248318e30737cf8f86d32>:0 
  at Microsoft.Psi.Receiver`1[T]..ctor (System.Int32 id, System.String name, Microsoft.Psi.Executive.PipelineElement element, System.Object owner, System.Action`1[T] onReceived, Microsoft.Psi.Scheduling.SynchronizationLock context, Microsoft.Psi.Pipeline pipeline, System.Boolean enforceIsolation) [0x000b3] in <6bd4d33a21f248318e30737cf8f86d32>:0 
  at Microsoft.Psi.Pipeline.CreateReceiver[T] (System.Object owner, System.Action`1[T] action, System.String name, System.Boolean autoClone) [0x0001c] in <6bd4d33a21f248318e30737cf8f86d32>:0 
  at Microsoft.Psi.Pipeline.CreateReceiver[T] (System.Object owner, System.Action`2[T1,T2] action, System.String name, System.Boolean autoClone) [0x0000d] in <6bd4d33a21f248318e30737cf8f86d32>:0 
  at Microsoft.Psi.Components.ConsumerProducer`2[TIn,TOut]..ctor (Microsoft.Psi.Pipeline pipeline, System.String name) [0x00030] in <6bd4d33a21f248318e30737cf8f86d32>:0 
  at Microsoft.Psi.Components.Processor`2[TIn,TOut]..ctor (Microsoft.Psi.Pipeline pipeline, System.Action`3[T1,T2,T3] transform, System.Action`2[T1,T2] onClose, System.String name) [0x0000d] in <6bd4d33a21f248318e30737cf8f86d32>:0 
  at Microsoft.Psi.Operators.Process[TIn,TOut] (Microsoft.Psi.IProducer`1[TOut] source, System.Action`3[T1,T2,T3] transform, Microsoft.Psi.DeliveryPolicy`1[T] deliveryPolicy, System.String name) [0x0000b] in <6bd4d33a21f248318e30737cf8f86d32>:0 
  at Microsoft.Psi.Operators.Do[T] (Microsoft.Psi.IProducer`1[TOut] source, System.Action`2[T1,T2] action, Microsoft.Psi.DeliveryPolicy`1[T] deliveryPolicy, System.String name) [0x0000d] in <6bd4d33a21f248318e30737cf8f86d32>:0 
  at Microsoft.Psi.Operators.Do[T] (Microsoft.Psi.IProducer`1[TOut] source, System.Action`1[T] action, Microsoft.Psi.DeliveryPolicy`1[T] deliveryPolicy, System.String name) [0x0000d] in <6bd4d33a21f248318e30737cf8f86d32>:0 
  at StereoKitApp.App+<>c.<PsiTest>b__7_0 () [0x00029] in F:\PSI_questTest\SKTemplate-Universal\App.cs:36 

I tried creating a new Android App (Xamarin) project in which I added the same PsiTest() function and tried running it on my Android phone which resulted in the same error.

I read on the Mixed Reality Overview wiki page that the Oculus Quest is not tested but that it should be possible to target it (with a little extra effort). Could you point me in the right direction on how to get this to work? Thanks in advance!

[sandrist]

Hello, just a quick heads up that we are still looking into this. We have never tested Psi with an Android/Xamarin project, so we're not quite sure what is going wrong, but we'll ping this thread when we know more.

[chitsaw]

I looked into this, and there appears to be an issue with the call to the System.Type.GetType overload which takes an assembly resolver argument that is causing it to throw inside System.TypeSpec.Resolve. I'm not sure why, but it looks like the assembly resolver is being bypassed internally for some reason. Unfortunately, because the .NET framework in Xamarin is based on Mono, it is not likely to support running \psi at this time.

[emielch]

Hey! Thanks for looking into it. I indeed came to the same conclusion about System.Type.GetType and the assembly resolver argument. I tried digging around a bit more and found that calling System.Type.GetType without the resolver argument works just as well. So I changed https://github.com/microsoft/psi/blob/419d35f89a5f8f110d10ca4e28f78bf2275d1f89/Sources/Runtime/Microsoft.Psi/Common/TypeResolutionHelper.cs#L24 to

var type = Type.GetType(typeName);

and did the same to the Type.GetType call on line 33.

I have to say I don't really know how the custom assembly resolver is different from the standard assembly resolution that is used when calling System.Type.GetType with only the typeName string as an argument and whether using the standard resolution has any unwanted side effects.

Anyway, when using this adapted version of PSI in a Xamarin Android project, calling the PsiTest() example from my first post runs as expected and timestamps are being printed to the console.

Next, I tried playing back a store made on another device. I made a store on my PC using the code from https://github.com/microsoft/psi/wiki/Brief-Introduction#3-saving-data and copied it over to my phone. To play it I added the read file permissions to AndroidManifest.xml and replaced the "timer" code from before with the playback code from https://github.com/microsoft/psi/wiki/Brief-Introduction#4-replaying-data:

public async void PsiTest()
{
    await Xamarin.Essentials.Permissions.RequestAsync<Xamarin.Essentials.Permissions.StorageRead>();
    await Task.Run(() =>
    {
        try
        {
            using (var p = Pipeline.Create())
            {
                // Open the store
                var store = PsiStore.Open(p, "demo", @"storage/emulated/0/Store");

                // Open the Sequence stream
                var sequence = store.OpenStream<double>("Sequence");

                // Compute derived streams
                var sin = sequence.Select(Math.Sin).Do(t => Console.WriteLine($"Sin: {t}"));
                var cos = sequence.Select(Math.Cos);

                // Run the pipeline
                p.Run();
            }
        }
        catch (Exception ex)
        {
            Console.WriteLine("Message: ");
            Console.WriteLine(ex.Message);
            Console.WriteLine("InnerException: ");
            Console.WriteLine(ex.InnerException);
            Console.WriteLine("StackTrace: ");
            Console.WriteLine(ex.StackTrace);
            Console.WriteLine("");
        }
    });
}

This resulted in the following exception:

Message: 
Specified method is not supported.
InnerException: 
StackTrace: 
  at System.Threading.Mutex.TryOpenExisting (System.String name, System.Threading.Mutex& result) [0x00000] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/corlib/System.Threading/Mutex.cs:205 
  at Microsoft.Psi.Persistence.InfiniteFileReader..ctor (System.String path, System.String fileName, System.Int32 fileId) [0x0001d] in C:\PSITest\psi\Sources\Runtime\Microsoft.Psi\Persistence\InfiniteFileReader.cs:32 
  at Microsoft.Psi.Persistence.MessageReader..ctor (System.String fileName, System.String path) [0x00022] in C:\PSITest\psi\Sources\Runtime\Microsoft.Psi\Persistence\MessageReader.cs:28 
  at Microsoft.Psi.Persistence.PsiStoreReader..ctor (System.String name, System.String path, System.Action`2[T1,T2] metadataUpdateHandler, System.Boolean autoOpenAllStreams) [0x00053] in C:\PSITest\psi\Sources\Runtime\Microsoft.Psi\Persistence\PsiStoreReader.cs:45 
  at Microsoft.Psi.Data.PsiStoreStreamReader..ctor (System.String name, System.String path) [0x00034] in C:\PSITest\psi\Sources\Runtime\Microsoft.Psi\Data\PsiStoreStreamReader.cs:35 
  at Microsoft.Psi.Data.PsiImporter..ctor (Microsoft.Psi.Pipeline pipeline, System.String name, System.String path, System.Boolean usePerStreamReaders) [0x00000] in C:\PSITest\psi\Sources\Runtime\Microsoft.Psi\Data\PsiImporter.cs:27 
  at Microsoft.Psi.PsiStore.Open (Microsoft.Psi.Pipeline pipeline, System.String name, System.String rootPath, System.Boolean usePerStreamReaders) [0x00001] in C:\PSITest\psi\Sources\Runtime\Microsoft.Psi\Data\PsiStore.cs:90 
  at androidTest.MainActivity.PsiTest () [0x00081] in C:\PSITest\androidTest\androidTest\androidTest\MainActivity.cs:71 

I fixed it by changing https://github.com/microsoft/psi/blob/419d35f89a5f8f110d10ca4e28f78bf2275d1f89/Sources/Runtime/Microsoft.Psi/Persistence/InfiniteFileReader.cs#L31-L33 into the following:

InfiniteFileWriter.PulseEventName(path, fileName);
this.writePulse = new Mutex(false);

Again, I don't exactly know whether not using System.Threading.Mutex.TryOpenExisting has any unwanted side effects, but for now it makes the app work and I get a stream of "Sin:" values printed to the console.

I'm currently stuck on the playing back of a different type of stream: store.OpenStream<(PsiHand, PsiHand)>("Hands");

This results in the following exception:

Message: 
Exception has been thrown by the target of an invocation.
InnerException: 
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.TypeInitializationException: The type initializer for 'Microsoft.Psi.Serialization.SimpleArraySerializer`1' threw an exception. ---> System.InvalidProgramException: Invalid IL code in (wrapper dynamic-method) Microsoft.Psi.Serializer:Deserialize (Microsoft.Psi.Common.BufferReader,double[]&,Microsoft.Psi.Serialization.SerializationContext): IL_0006: ldelema   0x00000001
  at (wrapper managed-to-native) System.Delegate.CreateDelegate_internal(System.Type,object,System.Reflection.MethodInfo,bool)
  at System.Delegate.CreateDelegate (System.Type type, System.Object firstArgument, System.Reflection.MethodInfo method, System.Boolean throwOnBindFailure, System.Boolean allowClosed) [0x002f0] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/corlib/System/Delegate.cs:286 
  at System.Delegate.CreateDelegate (System.Type type, System.Object firstArgument, System.Reflection.MethodInfo method) [0x00000] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/corlib/System/Delegate.cs:296 
  at System.Reflection.Emit.DynamicMethod.CreateDelegate (System.Type delegateType) [0x00029] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/mcs/class/corlib/System.Reflection.Emit/DynamicMethod.cs:178 
  at Microsoft.Psi.Serialization.Generator.GenerateMethodFromPrototype (System.Reflection.MethodInfo prototype, System.Type delegateType, System.Action`1[T] emit) [0x0005a] in C:\PSITest\psi\Sources\Runtime\Microsoft.Psi\Serialization\Generator.cs:74 
  at Microsoft.Psi.Serialization.Generator.GenerateDeserializeMethod[T] (System.Action`1[T] emit) [0x00016] in C:\PSITest\psi\Sources\Runtime\Microsoft.Psi\Serialization\Generator.cs:48 
  at Microsoft.Psi.Serialization.SimpleArraySerializer`1[T]..cctor () [0x00000] in C:\PSITest\psi\Sources\Runtime\Microsoft.Psi\Serialization\SimpleArraySerializer.cs:24 
   --- End of inner exception stack trace ---

Looking into this, I found the following description above the lines that throw the exception https://github.com/microsoft/psi/blob/419d35f89a5f8f110d10ca4e28f78bf2275d1f89/Sources/Runtime/Microsoft.Psi/Serialization/SimpleArraySerializer.cs#L20-L22 Something seems to go wrong when creating a delegate to IL code for the array (de)serializers, although I don't know how to circumvent this (by maybe sacrificing a bit of performance?) Any suggestions?

[chitsaw]

Thanks for sharing your efforts on getting this to work! To answer your questions:

1. Type.GetType: The reason we are using a custom assembly resolver is mainly to ensure that any type that the runtime attempts to load by name has already been loaded by the application. This is for security purposes, to prevent any arbitrary assembly from being loaded into the process by name. Additionally, the custom assembly resolver also attempts to resolve an older version of an assembly to a newer version that is currently loaded. This could matter if you are reading from a store containing streams of a type whose assembly version has changed. For your purposes, it should be fine to just modify those calls to Type.GetType as you have done.

2. Mutex.TryOpenExisting: This is used for synchronization purposes when a store is being concurrently written to and read from. The mutex is pulsed when new data is written to the store, such that any concurrent reader(s) may be notified immediately when new data is available. If the named mutex cannot be created (as is the case on some platforms), then the reader will just poll for new data. So what you did here is fine too.

3. Invalid IL exception: I'm not too sure about this one. Initially I thought that maybe the platform you're running on does not support IL generation (we have run into some platforms which do not support System.Reflection.Emit). However, in this case it appears that IL generation is supported, since the exception message mentions the "Invalid IL code" ldelema 0x00000001. I'm not sure why this is happening, since Ldelema is a valid opcode. If this is indeed a limitation with the .NET framework on Android, the only option I can think of would be to use the non-optimized ArraySerializer instead. You should be able to accomplish this by changing the following line in KnownSerializers: https://github.com/microsoft/psi/blob/419d35f89a5f8f110d10ca4e28f78bf2275d1f89/Sources/Runtime/Microsoft.Psi/Serialization/KnownSerializers.cs#L601 However, the serialization subsystem in \psi makes extensive use of IL generation, so it is possible that you will run into similar issues. If so, then this might require creating non-optimized serializers that you could then use in place of the IL-generated ones.

Hope this helps.