MediaPipeUnityPlugin
MediaPipeUnityPlugin copied to clipboard
homuler
matrix classification via tflite model
- example usage of tflite graph from Unity
- accepts inputs other than image
- returns a list of floats representing tflite classification results
- introduces new
- input packet class: MatrixFramePacket
- output packet class: FloatVectorFramePacket -> the name "Frame" should resemble "ImageFramePacket" and should help to avoid confusions with similar sounding classes like
- matrix.cc
- FloatArrayPacket.cs
Missing:
- not tested on Android
- no tests written
- unsure function naming in "matrix_frame.cc" and "float_vector_frame.cc"
- when compiling particular solutions "TfliteConverterCalculator" seems to be missing
just testing around a bit (didn't excessively test after rebasing onto current master). I guess I somehow lost the
"//mediapipe/calculators/tflite:tflite_converter_calculator",
"//mediapipe/calculators/tflite:tflite_inference_calculator",
"//mediapipe/calculators/tflite:tflite_tensors_to_floats_calculator",
dependency needed for the example graph somewhere along the way. Will look into this...
ok so the problem was related to building only the pose solution
python build.py build --desktop cpu --opencv=cmake -v --verbose -c dbg --solution pose
it works when I build the full solution ( omit the using --solution pose option)
So I added some tests for "FloatVectorPacket". Unfortunately the test "At_ShouldReturnNewPacketWithTimestamp()" fails. I'm not exactly sure why.
I assume data gets deleted after calling packet.Get(), so I'm not sure how creating another packet from the same with another timestamp should work.
I took over the Test functions from FloatArrayPacketTest.cs
ok so I also pushed some MatrixPacket tests. However everything involving Get() is not yet working. Will look into that later...
Get was not needed as the Packet is only used as input to the graph and not as output, so one might also just omit those tests...
I just tested the "Matrix Classification" example scene on a mobile device and got a "ResourceManager not found" error. I will look into that.
Here the logcat:
08-11 12:36:29.586 21361 21449 D Unity : UnloadTime: 0.357708 ms
08-11 12:36:29.588 21361 21449 I IL2CPP : Locale de-DE
08-11 12:36:29.591 21361 21449 I Unity : MatrixClassificationToyExample: Loading dependent assets...
08-11 12:36:29.591 21361 21449 I Unity : UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
08-11 12:36:29.591 21361 21449 I Unity : UnityEngine.Logger:Log(LogType, String, Object)
08-11 12:36:29.591 21361 21449 I Unity : Mediapipe.Unity.LoggerWrapper:Log(LogLevel, String, Object) (at C:\Users\garbade\Git\mpup_v14_220715_snglsltn_mtrxclssfr\Packages\com.github.homuler.mediapipe\Runtime\Scripts\Unity\Logger.cs:262)
08-11 12:36:29.591 21361 21449 I Unity : Mediapipe.Unity.Logger:Log(LogLevel, String, Object) (at C:\Users\garbade\Git\mpup_v14_220715_snglsltn_mtrxclssfr\Packages\com.github.homuler.mediapipe\Runtime\Scripts\Unity\Logger.cs:138)
08-11 12:36:29.591 21361 21449 I Unity : Mediapipe.Unity.Logger:LogInfo(String, Object) (at C:\Users\garbade\Git\mpup_v14_220715_snglsltn_mtrxclssfr\Packages\com.github.homuler.mediapipe\Runtime\Scripts\Unity\Logger.cs:181)
08-11 12:36:29.591 21361 21449 I Unity : Mediapipe.Unity.<Init>d__8:MoveNext() (at C:\Users\garbade\Git\mpup_v14_220715_snglsltn_mtrxclssfr\Assets\MediaPipeUnity\Tutorial\Matrix Classification\MatrixClassification.cs:180)
08-11 12:36:29.591 21361 21449 I Unity : UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr) (at \Users\bokken\buildslave\u
08-11 12:36:29.654 21361 21449 E Unity : InvalidOperationException: ResourceManager is not provided
08-11 12:36:29.654 21361 21449 E Unity : at Mediapipe.Unity.AssetLoader.PrepareAssetAsync (System.String name, System.String uniqueKey, System.Boolean overwrite) [0x00013] in C:\Users\garbade\Git\mpup_v14_220715_snglsltn_mtrxclssfr\Assets\MediaPipeUnity\Samples\Common\Scripts\AssetLoader.cs:28
08-11 12:36:29.654 21361 21449 E Unity : at Mediapipe.Unity.MatrixClassification.WaitForAsset (System.String assetName, System.String uniqueKey, System.Boolean overwrite) [0x00005] in C:\Users\garbade\Git\mpup_v14_220715_snglsltn_mtrxclssfr\Assets\MediaPipeUnity\Tutorial\Matrix Classification\MatrixClassification.cs:175
08-11 12:36:29.654 21361 21449 E Unity : at Mediapipe.Unity.MatrixClassification.WaitForAsset (System.String assetName, System.Boolean overwrite) [0x00005] in C:\Users\garbade\Git\mpup_v14_220715_snglsltn_mtrxclssfr\Assets\MediaPipeUnity\Tutorial\Matrix Classification\MatrixClassification.cs:170
08-11 12:36:29.654 21361 21449 E Unity : at Mediapipe.Unity.MatrixClassification.RequestDependentAssets () [0x0000e] in C:\Users\garbade\Git\mpup_v14_220715_snglsltn_mtrxclssfr\Assets\MediaPipeUnity\Tuto
08-11 12:36:29.661 1144 7075 W ServiceManager: Permission failure: android.permission.HIGH_SAMPLING_RATE_SENSORS from uid=10347 pid=21361
08-11 12:36:29.662 1144 7075 W ServiceManager: Permission failure: android.permission.HIGH_SAMPLING_RATE_SENSORS from uid=10347 pid=21361
08-11 12:36:29.663 21361 21449 D Unity : Sensor : Accelerometer ( 1) ; 0.002393 / 0.00s ; LSM6DSO Accelerometer / STMicro
08-11 12:36:29.670 1144 4006 V WindowManager: Relayout hash=190914e, pid=21361: mAttrs={(0,0)(fillxfill) sim={adjust=pan} layoutInDisplayCutoutMode=shortEdges ty=BASE_APPLICATION fmt=TRANSLUCENT wanim=0x1030303
08-11 12:36:29.674 21361 21361 I ViewRootImpl@3a333d9[UnityPlayerActivity]: Relayout returned: old=(0,0,2560,1600) new=(0,0,2560,1600) req=(2560,1600)0 dur=15 res=0x1 s={true -5476376646377316352} ch=false fn=4
08-11 12:36:29.675 21361 21361 I ViewRootImpl@3a333d9[UnityPlayerActivity]: updateBoundsLayer: t = android.view.SurfaceControl$Transaction@97b2314 sc = Surface(name=Bounds for - com.DefaultCompany.MediaPipeUnityPlugin/com.unity3d.player.UnityPlayerActivity@0)/@0x39637bd frame = 4
08-11 12:36:29.677 21361 21449 D CompatibilityChangeReporter: Compat change id reported: 147600208; UID 10347; state: ENABLED
08-11 12:36:29.681 21361 21449 D Unity : Choreographer available: Enabling VSYNC timing
08-11 12:36:29.718 21361 21449 I Unity : Setup Protobuf Logging
08-11 12:36:29.718 21361 21449 I Unity : UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
08-11 12:36:29.718 21361 21449 I Unity : UnityEngine.Logger:Log(LogType, Object)
08-11 12:36:29.718 21361 21449 I Unity : UnityEngine.Debug:Log(Object)
08-11 12:36:29.718 21361 21449 I Unity : Mediapipe.Unity.MatrixClassification:Start() (at C:\Users\garbade\Git\mpup_v14_220715_snglsltn_mtrxclssfr\Assets\MediaPipeUnity\Tutorial\Matrix Classification\MatrixClassification.cs:58)
08-11 12:36:29.718 21361 21449 I Unity :
08-11 12:36:30.028 21361 21449 I Unity : Start
08-11 12:36:30.028 21361 21449 I Unity : UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
08-11 12:36:30.028 21361 21449 I Unity : UnityEngine.Logger:Log(LogType, Object)
08-11 12:36:30.028 21361 21449 I Unity : UnityEngine.Debug:Log(Object)
08-11 12:36:30.028 21361 21449 I Unity : Mediapipe.Unity.MatrixClassification:Start() (at C:\Users\garbade\Git\mpup_v14_220715_snglsltn_mtrxclssfr\Assets\MediaPipeUnity\Tutorial\Matrix Classification\MatrixClassification.cs:61)
08-11 12:36:30.028 21361 21449 I Unity :
08-11 12:36:30.054 21361 21449 I Unity : StartRun
08-11 12:36:30.054 21361 21449 I Unity : UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
08-11 12:36:30.054 21361 21449 I Unity : UnityEngine.Logger:Log(LogType, Object)
08-11 12:36:30.054 21361 21449 I Unity : UnityEngine.Debug:Log(Object)
08-11 12:36:30.054 21361 21449 I Unity : Mediapipe.Unity.MatrixClassification:Start() (at C:\Users\garbade\Git\mpup_v14_220715_snglsltn_mtrxclssfr\Assets\MediaPipeUnity\Tutorial\Matrix Classification\MatrixClassification.cs:106)
08-11 12:36:30.054 21361 21449 I Unity :
08-11 12:36:30.059 21361 22697 I tflite : Initialized TensorFlow Lite runtime.
08-11 12:36:30.062 21361 21449 I Unity : Poll output
08-11 12:36:30.062 21361 21449 I Unity : UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
08-11 12:36:30.062 21361 21449 I Unity : UnityEngine.Logger:Log(LogType, Object)
08-11 12:36:30.062 21361 21449 I Unity : UnityEngine.Debug:Log(Object)
08-11 12:36:30.062 21361 21449 I Unity : Mediapipe.Unity.MatrixClassification:Start() (at C:\Users\garbade\Git\mpup_v14_220715_snglsltn_mtrxclssfr\Assets\MediaPipeUnity\Tutorial\Matrix Classification\MatrixClassification.cs:118)
08-11 12:36:30.062 21361 21449 I Unity :
08-11 12:36:30.065 21361 21449 E Unity : MediaPipeException: INVALID_ARGUMENT: CalculatorGraph::Run() failed in Run:
08-11 12:36:30.065 21361 21449 E Unity : Calculator::Open() for node "TfLiteInferenceCalculator" failed: ; Can't find file: mediapipe/models/adder_model_single_input_2x3.tflite
08-11 12:36:30.065 21361 21449 E Unity : at Mediapipe.Status.AssertOk () [0x00015] in C:\Users\garbade\Git\mpup_v14_220715_snglsltn_mtrxclssfr\Packages\com.github.homuler.mediapipe\Runtime\Scripts\Framework\Port\Status.cs:147
08-11 12:36:30.065 21361 21449 E Unity : at Mediapipe.Unity.MatrixClassification.Start () [0x00140] in C:\Users\garbade\Git\mpup_v14_220715_snglsltn_mtrxclssfr\Assets\MediaPipeUnity\Tutorial\Matrix Classification\MatrixClassification.cs:134
08-11 12:36:30.065 21361 21449 E Unity :
08-11 13:05:44.044 1022 1022 I SurfaceFlinger: SFWD update time=21361391032162
08-11 13:08:34.286 21361 21377 W PipeUnityPlugi: Reducing the number of considered missed Gc histogram windows from 209 to 100
08-11 13:08:34.287 21361 21378 D InputTransport: Input channel destroyed: 'ClientS', fd=114
08-11 13:30:00.022 21361 21361 D CompatibilityChangeReporter: Compat change id reported: 150939131; UID 10347; state: ENABLED
Nevermind. Got it working on Android as well. I had to add
AssetLoader.Provide(mew StreamingAssetsResourceManager());
and place the .bytes file into
Assets\StreamingAssets
I'm not totally getting this github review process. I tried to accept everything, but I'm not sure whether it was actually all correct. I'll get back when I actually figured out if the suggested version compiles...
@mgarbade Thank you for your contributions! If there are any comments that are not clear to you, you can leave the related code as it is. I will merge this PR into another branch first, and then I will fix it.
Thx. I'm still working on those last mentioned points:
- deleting MatrixClassification in "Assets/../Tutorial Scene"
- FloatVectorPacket still assumes the vector length to be set by Unity via constructor
- run and test the suggested version
- and 2. are done. While testing I found bugs, so it takes a while until I have them fixed. It takes longer as Unity crashes on Windows for every error that occurs in mediapipe.
As for 3. I already found 1 bug, caused by refactoring.
moving on ...
- deleting MatrixClassification in "Assets/../Tutorial Scene"
- FloatVectorPacket still assumes the vector length to be set by Unity via constructor
- run and test the suggested version
I force pushed some more changes:
- is done
- is done
- at least the matrix classification tutorial scene was compiling and running on my machine again
all your requested changes should still be inside the current HEAD of "matrix_packet_rebased_v3"
@mgarbade Will you fix the failed tests? If they all pass, I'll merge this PR.
I'm sorry to bother you. But I cannot figure out from the reports what is the problem with the tests. For example the test "run-tests-on-linux / test (ubuntu-22.04, 2021.3.3f1) failed 13 days ago in 6m 9s"
has the following log output at the end:
[ALLOC_PROFILER_EDITOR] Peak usage frame count: [0-1.0 KB]: 56 frames Requested Block Size 1.0 MB Peak Block count 0 Peak Allocated memory 0 B Peak Large allocation bytes 0 B [ALLOC_PROFILER_BUCKET] Large Block size 32.0 MB Used Block count 1 Peak Allocated bytes 2.0 KB ##utp:{"type":"MemoryLeaks","version":2,"phase":"Immediate","time":1664285327254,"processId":4274,"allocatedMemory":4018038,"memoryLabels":[{"Default":8247},{"Permanent":16120},{"Thread":1083180},{"Manager":18061},{"VertexData":12},{"Geometry":280},{"Texture":160},{"Shader":69203},{"Material":24},{"GfxDevice":35520},{"Animation":312},{"Audio":3976},{"Physics":289},{"Serialization":672},{"Input":26792},{"JobScheduler":200},{"Mono":40},{"ScriptingNativeRuntime":43520},{"BaseObject":1620252},{"Resource":944},{"Renderer":1936},{"Transform":48},{"File":271412},{"WebCam":24},{"Culling":40},{"Terrain":1025},{"Wind":24},{"String":30299},{"DynamicArray":93028},{"HashMap":77574},{"Utility":8682},{"Curl":3751},{"PoolAlloc":32192},{"AI":40},{"TypeTree":2064},{"ScriptManager":368},{"RuntimeInitializeOnLoadManager":72},{"SpriteAtlas":112},{"GI":3784},{"Unet":16},{"Director":7816},{"WebRequest":720},{"VR":45529},{"SceneManager":624},{"Video":72},{"LazyScriptCache":32},{"NativeArray":12},{"Camera":25},{"Secure":6609},{"SerializationCache":624},{"APIUpdating":5872},{"Subsystems":384},{"VirtualTexturing":57560},{"StaticSafetyDebugInfo":327752},{"EditorGui":40},{"EditorUtility":37559},{"VersionControl":4},{"Undo":450},{"AssetDatabase":10586},{"RestService":1518},{"EditorGi":368},{"License":1056},{"UnityConnect":29688},{"Collab":785},{"Upm":1504},{"DrivenProperties":80},{"HubClient":32},{"LocalIPC":101},{"ProfilerEditor":20133},{"CoreBusinessMetrics":1402},{"TLS":2958},{"Licensing":1816},{"AssetReference":32}]} Error: Process completed with exit code 2.
This tells me nothing. I don't know what "##utp" means and any of the words following that keyword seem not to relate to my changes. "type":"MemoryLeaks" might hint to the problem, however without any more context, I'm not sure what to look for.
I'll try to run the tests on my local machine and see if there is something failing either...
See https://github.com/homuler/MediaPipeUnityPlugin/actions/runs/3135556925/jobs/5092743904. I'm rerunning the CI, and once it finishes, you can download the results from the summary page (retained only for a week and the last time results are already removed).
On Linux or macOS, it seems that the result.xml is not printed to the console, so I'll fix it.
Ok. I found some failing tests in "MatrixPacketTest" on my local windows machine.
On the other hand, all tests related to "FloatVectorPacketTest" were successful.
I cannot relate these failed tests to those encountered by github actions. But I will fix them anyway.
As before the output of the github action run test is not very explicit about the cause of the test failure (taken from the link you pasted above):
name="TryReadPixelData_ShouldReturnTrue_When_TheFormatIsSrgba64" fullname="Mediapipe.Unity.Tests.ImageFrameExtensionTest.TryReadPixelData_ShouldReturnTrue_When_TheFormatIsSrgba64" methodname="TryReadPixelData_ShouldReturnTrue_When_TheFormatIsSrgba64" classname="Mediapipe.Unity.Tests.ImageFrameExtensionTest" runstate="Runnable" seed="1791896299" result="Passed" start-time="2022-09-27 14:04:29Z" end-time="2022-09-27 14:04:29Z" duration="0.000819" asserts="0"> <properties /> </test-case> <test-case id="2440" name="TryReadPixelData_ShouldReturnTrue_When_TheFormatIsVec32f1" fullname="Mediapipe.Unity.Tests.ImageFrameExtensionTest.TryReadPixelData_ShouldReturnTrue_When_TheFormatIsVec32f1" methodname="TryReadPixelData_ShouldReturnTrue_When_TheFormatIsVec32f1" classname="Mediapipe.Unity.Tests.ImageFrameExtensionTest" runstate="Runnable" seed="1147385770" result="Passed" start-time="2022-09-27 14:04:29Z" end-time="2022-09-27 14:04:29Z" duration="0.000764" asserts="0"> <properties /> </test-case> </test-suite> </test-suite> </test-suite> </test-suite> </test-suite> </test-suite> </test-run> Error: Process completed with exit code 1.
As before the output of the github action run test is not very explicit about the cause of the test failure (taken from the link you pasted above):
I agree that the output can be improved, but at least you can know which test cases are failing.
xmllint --xpath "//test-case[@result='Failed']" results.xml
<test-case id="2864" name="At_ShouldReturnNewPacketWithTimestamp" fullname="Mediapipe.Tests.MatrixPacketTest.At_ShouldReturnNewPacketWithTimestamp" methodname="At_ShouldReturnNewPacketWithTimestamp" classname="Mediapipe.Tests.MatrixPacketTest" runstate="Runnable" seed="132040465" result="Failed" label="Error" start-time="2022-10-10 10:22:45Z" end-time="2022-10-10 10:22:45Z" duration="0.010864" asserts="0">
<properties/>
<failure>
<message><![CDATA[System.NotImplementedException : The method or operation is not implemented.]]></message>
<stack-trace><![CDATA[ at Mediapipe.MatrixPacket.Get () [0x00001] in /home/runner/work/MediaPipeUnityPlugin/MediaPipeUnityPlugin/DummyProject/Packages/com.github.homuler.mediapipe/Runtime/Scripts/Framework/Packet/MatrixPacket.cs:65
at Mediapipe.Tests.MatrixPacketTest.At_ShouldReturnNewPacketWithTimestamp () [0x0001d] in /home/runner/work/MediaPipeUnityPlugin/MediaPipeUnityPlugin/DummyProject/Packages/com.github.homuler.mediapipe/Tests/EditMode/Framework/Packet/MatrixPacketTest.cs:57
at (wrapper managed-to-native) System.Reflection.RuntimeMethodInfo.InternalInvoke(System.Reflection.RuntimeMethodInfo,object,object[],System.Exception&)
at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <8516445456524a6cb20c4b3b7de67982>:0 ]]></stack-trace>
</failure>
</test-case>
<test-case id="2865" name="Consume_ShouldThrowNotSupportedException" fullname="Mediapipe.Tests.MatrixPacketTest.Consume_ShouldThrowNotSupportedException" methodname="Consume_ShouldThrowNotSupportedException" classname="Mediapipe.Tests.MatrixPacketTest" runstate="Runnable" seed="847120028" result="Failed" start-time="2022-10-10 10:22:45Z" end-time="2022-10-10 10:22:45Z" duration="0.003217" asserts="0">
<properties/>
<failure>
<message><![CDATA[ Expected: <System.NotSupportedException>
But was: <System.NotImplementedException: The method or operation is not implemented.
at Mediapipe.MatrixPacket.Consume () [0x00001] in /home/runner/work/MediaPipeUnityPlugin/MediaPipeUnityPlugin/DummyProject/Packages/com.github.homuler.mediapipe/Runtime/Scripts/Framework/Packet/MatrixPacket.cs:70
at Mediapipe.Tests.MatrixPacketTest+<>c__DisplayClass4_0.<Consume_ShouldThrowNotSupportedException>b__0 () [0x00001] in /home/runner/work/MediaPipeUnityPlugin/MediaPipeUnityPlugin/DummyProject/Packages/com.github.homuler.mediapipe/Tests/EditMode/Framework/Packet/MatrixPacketTest.cs:79
at NUnit.Framework.Assert.Throws (NUnit.Framework.Constraints.IResolveConstraint expression, NUnit.Framework.TestDelegate code, System.String message, System.Object[] args) [0x00004] in <10b6135e63434fdba4fc6c109928ab3b>:0 >
]]></message>
<stack-trace><![CDATA[at Mediapipe.Tests.MatrixPacketTest.Consume_ShouldThrowNotSupportedException () [0x00013] in /home/runner/work/MediaPipeUnityPlugin/MediaPipeUnityPlugin/DummyProject/Packages/com.github.homuler.mediapipe/Tests/EditMode/Framework/Packet/MatrixPacketTest.cs:79
]]></stack-trace>
</failure>
</test-case>
<test-case id="2861" name="Ctor_ShouldInstantiatePacket_When_CalledWithValue" fullname="Mediapipe.Tests.MatrixPacketTest.Ctor_ShouldInstantiatePacket_When_CalledWithValue" methodname="Ctor_ShouldInstantiatePacket_When_CalledWithValue" classname="Mediapipe.Tests.MatrixPacketTest" runstate="Runnable" seed="697797710" result="Failed" label="Error" start-time="2022-10-10 10:22:45Z" end-time="2022-10-10 10:22:45Z" duration="0.000617" asserts="0">
<properties/>
<failure>
<message><![CDATA[System.NotImplementedException : The method or operation is not implemented.]]></message>
<stack-trace><![CDATA[ at Mediapipe.Packet`1[TValue].ValidateAsType () [0x00001] in /home/runner/work/MediaPipeUnityPlugin/MediaPipeUnityPlugin/DummyProject/Packages/com.github.homuler.mediapipe/Runtime/Scripts/Framework/Packet/Packet.cs:74
at Mediapipe.Tests.MatrixPacketTest.Ctor_ShouldInstantiatePacket_When_CalledWithValue () [0x0000f] in /home/runner/work/MediaPipeUnityPlugin/MediaPipeUnityPlugin/DummyProject/Packages/com.github.homuler.mediapipe/Tests/EditMode/Framework/Packet/MatrixPacketTest.cs:22
at (wrapper managed-to-native) System.Reflection.RuntimeMethodInfo.InternalInvoke(System.Reflection.RuntimeMethodInfo,object,object[],System.Exception&)
at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <8516445456524a6cb20c4b3b7de67982>:0 ]]></stack-trace>
</failure>
</test-case>
Thanks. Looks like the same tests that where failing in my local windows build. (not sure where to download the "results.xml" file you mentioned above)
Anyway, I'll go ahead and try to fix the tests...
I'm not sure how to write the "packet__Get()" function for MatrixPacket.cs which is needed for one of the test functions
- in C++ the packet holds an Eigen::Matrix
- in Unity the same data is of class "MatrixData" which is a protobuf format --> I assume I have to convert the EigenMatrix to MatrixData inside the C++ part of the code before sending it back to Unity
However I'm not sure how to do that conversion. This is my first draft:
MP_CAPI(MpReturnCode) mp_Packet__GetMatrix(mediapipe::Packet* packet, mp_api::SerializedProto* value_out) {
TRY
// Get Eigen::Matrix from packet
mediapipe::Matrix matrix;
matrix = packet->Get<mediapipe::Matrix>();
// Convert to format that can be send to Unity
mediapipe::MatrixData* matrix_data;
mediapipe::MatrixDataProtoFromMatrix(matrix, matrix_data);
*value_out = matrix_data; // <-- THIS IS WHERE THE CODE IS CRASHING
// *value_out = packet->Get<bool>();
RETURN_CODE(MpReturnCode::Success);
CATCH_EXCEPTION
}
the compilation fails saying
ERROR: C:/users/garbade/git/homuler/mpup_pr/mediapipe_api/framework/formats/BUILD:67:11: Compiling mediapipe_api/framework/formats/matrix_data.cc failed: (Exit 2): cl.exe failed: error executing command D:\Programs\MicrosoftVisualStudio\2022\Community\VC\Tools\MSVC\14.30.30704\bin\HostX64\x64\cl.exe @bazel-out/x64_windows-dbg/bin/mediapipe_api/framework/formats/_objs/matrix_data/matrix_data.obj.params
mediapipe_api/framework/formats/matrix_data.cc(60): error C2679: binary '=': no operator found which takes a right-hand operand of type 'mediapipe::MatrixData *' (or there is no acceptable conversion)
C:\_bzl\p7phwaas\execroot\mediapipe_api\mediapipe_api/external/protobuf.h(22): note: could be 'mp_api::SerializedProto &mp_api::SerializedProto::operator =(mp_api::SerializedProto &&)'
C:\_bzl\p7phwaas\execroot\mediapipe_api\mediapipe_api/external/protobuf.h(22): note: or 'mp_api::SerializedProto &mp_api::SerializedProto::operator =(const mp_api::SerializedProto &)'
mediapipe_api/framework/formats/matrix_data.cc(60): note: while trying to match the argument list '(mp_api::SerializedProto, mediapipe::MatrixData *)'
Target //mediapipe_api:mediapipe_desktop failed to build
however I'm not sure how to convert the protobuf to serialized proto
moving forward. This code seems more reasonable but still crashes:
MP_CAPI(MpReturnCode) mp_Packet__GetMatrix(mediapipe::Packet* packet, mp_api::SerializedProto* value_out) {
TRY
// Get Eigen::Matrix from packet
mediapipe::Matrix matrix;
matrix = packet->Get<mediapipe::Matrix>();
// Convert to format that can be send to Unity
mediapipe::MatrixData* matrix_data;
mediapipe::MatrixDataProtoFromMatrix(matrix, matrix_data); // <-- CRASHES HERE
// auto matrix_data_serialized = new mp_api::SerializedProto();
SerializeProto(*matrix_data, value_out);
RETURN_CODE(MpReturnCode::Success);
CATCH_EXCEPTION
}
The exact line where it crashes:
void MatrixDataProtoFromMatrix(const Matrix& matrix, MatrixData* matrix_data) {
const int rows = matrix.rows();
const int cols = matrix.cols();
matrix_data->set_rows(rows);
matrix_data->set_cols(cols);
matrix_data->clear_layout();
proto_ns::RepeatedField<float>(matrix.data(), matrix.data() + rows * cols) // <-- CRASHES HERE
.Swap(matrix_data->mutable_packed_data());
}
It throws an exception in mediapipe which I'm unable to read...
this is what the console says:
The thread 0xaf910 has exited with code 0 (0x0).
The thread 0x325b8 has exited with code 0 (0x0).
The thread 0xa7b68 has exited with code 0 (0x0).
The thread 0x10d28 has exited with code 0 (0x0).
The thread 0x9a708 has exited with code 0 (0x0).
The thread 0xcb4a4 has exited with code 0 (0x0).
Exception thrown at 0x00007FFA8D454FD9 in Unity.exe: Microsoft C++ exception: google::protobuf::FatalException at memory location 0x0000004DEC7EDB90.
The thread 0x4d68 has exited with code 0 (0x0).
The thread 0x8dd88 has exited with code 0 (0x0).
the "_mp_return_code" is -327236016 (which I'm unable to google) will look further into this. Hints are welcome...
nevermind. I found the problem! (some astrix to many ...)
MP_CAPI(MpReturnCode) mp_Packet__GetMatrix(mediapipe::Packet* packet, mp_api::SerializedProto* value_out) {
TRY
// Get Eigen::Matrix from packet
mediapipe::Matrix matrix;
matrix = packet->Get<mediapipe::Matrix>();
// Convert to format that can be send to Unity
mediapipe::MatrixData matrix_data;
mediapipe::MatrixDataProtoFromMatrix(matrix, &matrix_data);
// auto matrix_data_serialized = new mp_api::SerializedProto();
SerializeProto(matrix_data, value_out);
RETURN_CODE(MpReturnCode::Success);
CATCH_EXCEPTION
}
all tests concerning the two new classes I introduced are running successfully on my Windows 10 Pro machine in Unity Editor mode:
I'm not 100% positive how useful these tests are, but they roughly follow the pattern of similar classes. Especially questionable is the test "Consume_ShouldThrowNotSupportedException" which just makes sure that the Consume function is not implemented. (I'm not sure what the Consume function is used for anyway)