Realtime Spatial Mapping & Texturing crashing Unity

[Rosthouse]

Hello everyone

I'm trying to do Spatial Mapping with texturing in real time. However, every time I call the camera.ApplyTexture(...) method, Unity will crash.

My basic update loop is as follows:

public void FetchMesh(ref ProgressiveMesh progressiveMesh)
{
    camera.RequestMesh();
    UpdateMesh(ref progressiveMesh); // This waits for camera.GetMeshRequestStatus() == sl.ERROR_CODE.SUCCESS
    
    ResizeMesh(ref progressiveMesh);
    RetrieveMesh(ref progressiveMesh, false);
    FilterMesh(ref progMesh);
    ApplyTexture(ref progressiveMesh);
    RetrieveMesh(ref progressiveMesh, true);
    ResizeMesh(ref progressiveMesh);
    RetrieveMesh(ref progressiveMesh);
}

public void ResizeMesh(ref ProgressiveMesh progressiveMesh)
{
    progressiveMesh.vertices = new Vector3[progressiveMesh.numVertices]; //Allocation is faster than resizing.
    progressiveMesh.triangles = new int[3 * progressiveMesh.numTriangles];
    progressiveMesh.uvs = new Vector2[progressiveMesh.numVertices];
}

private void RetrieveMesh(ref ProgressiveMesh progressiveMesh, bool updateTexture = false)
{
    var errorCode = ERROR_CODE.FAILURE;
    errorCode = camera.RetrieveMesh(
        progressiveMesh.vertices,
        progressiveMesh.triangles,
        progressiveMesh.MAX_SUBMESH,
        updateTexture ? progressiveMesh.uvs : null,
        updateTexture ? progressiveMesh.texture.GetNativeTexturePtr() : IntPtr.Zero
    );
    if (errorCode != ERROR_CODE.SUCCESS)
    {
        throw new Exception("Error while retrieving mesh; {errorCode}");
    }
}

private void FilterMesh(ref ProgressiveMesh progressiveMesh)
{
    this.camera.FilterMesh(
        FILTER.MEDIUM,
        progressiveMesh.numVerticesInSubmesh,
        progressiveMesh.numTrianglesInSubmesh,
        ref progressiveMesh.numUpdatedMeshes,
        progressiveMesh.updatedIndices,
        ref progressiveMesh.numVertices,
        ref progressiveMesh.numTriangles,
        progressiveMesh.MAX_SUBMESH);
    this.ResizeMesh(ref progressiveMesh);
    this.RetrieveMesh(ref progressiveMesh, false);
}

private bool ApplyTexture(ref ProgressiveMesh progressiveMesh)
{
    var textureSize = new int[2];
    var result = this.camera.ApplyTexture(
        progressiveMesh.numVerticesInSubmesh,
        progressiveMesh.numTrianglesInSubmesh,
        ref progressiveMesh.numUpdatedMeshes,
        progressiveMesh.updatedIndices,
        ref progressiveMesh.numVertices,
        ref progressiveMesh.numTriangles,
        textureSize,
        progressiveMesh.MAX_SUBMESH
    );
    progressiveMesh.textureSize = textureSize;
    return result;
}

This is not the full code, but should contain everything needed for this issue (full code here. It is basically an adaption of the ZEDSpatialMapper, trimmed for real-time spatial mapping.

Embedded in a MonoBehaviour (and running from a coroutine, since creating Textures can only be done in the unity main thread), this code runs fine when I'm using it without texturing. However, as soon as I introduce the ApplyTexture(...) method, Unity will crash and give me the following crash report:

sl_core64.dll caused an Access Violation (0xc0000005)
  in module sl_core64.dll at 0033:11545ca2.

Looking at the stack trace, it seems the problem is in the sl_core64 library:

Stack Trace of Crashed Thread 8792:
0x00007FFC108A5CA2 (sl_core64) slutils::cv::Mat::ptr
0x00007FFC108A567B (sl_core64) slutils::cv::Mat::ptr
0x00007FFC1089C295 (sl_core64) slutils::cv::Mat::ptr
0x00007FFC108A05CA (sl_core64) slutils::cv::Mat::ptr
0x00007FFC1089F71E (sl_core64) slutils::cv::Mat::ptr
0x00007FFC108E6902 (sl_core64) slutils::cv::Mat::ptr
0x00007FFC10ABDB3A (sl_core64) slutils::cv::Mat::ptr
0x00007FFC76E8728F (sl_unitywrapper) dllz_update_mesh
0x00007FFC76E7F88C (sl_unitywrapper) dllz_apply_texture
0x00000000587784FA (ZED) sl.ZEDCamera.dllz_apply_texture()
0x00000000587782B3 (ZED) sl.ZEDCamera.ApplyTexture()
0x0000000058778113 (MainAssembly) ProgressiveMapper.ApplyTexture()
0x00000000587766BB (MainAssembly) ProgressiveMapper.FetchMesh()
0x00000000587762C3 (MainAssembly) ProgressiveMapper.RetrieveMeshFromCamera()
0x0000000058775B53 (MainAssembly) <UpdateMesh>d__18.MoveNext()
0x0000000048FD2190 (UnityEngine.CoreModule) UnityEngine.SetupCoroutine.InvokeMoveNext()
0x0000000048FD2380 (UnityEngine.CoreModule) <Module>.runtime_invoke_void_object_intptr()
0x00007FFC1636BE7B (mono-2.0-bdwgc) mono_get_runtime_build_info
0x00007FFC162F1E32 (mono-2.0-bdwgc) mono_perfcounters_init
0x00007FFC162FAE3F (mono-2.0-bdwgc) mono_runtime_invoke
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 0000000140C0055A)
0x0000000140C0055A (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 0000000140BF08AA)
0x0000000140BF08AA (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 0000000140BC3F15)
0x0000000140BC3F15 (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 00000001406F2B0E)
0x00000001406F2B0E (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 000000014095C6FC)
0x000000014095C6FC (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 000000014095B2D7)
0x000000014095B2D7 (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 000000014095B3A3)
0x000000014095B3A3 (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 000000014095E661)
0x000000014095E661 (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 000000014133BCCF)
0x000000014133BCCF (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 000000014133A280)
0x000000014133A280 (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 000000014149663B)
0x000000014149663B (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 00000001414982D6)
0x00000001414982D6 (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 000000014247F6DA)
0x000000014247F6DA (Unity) (function-name not available)
0x00007FFC81D63DC4 (KERNEL32) BaseThreadInitThunk
0x00007FFC84913691 (ntdll) RtlUserThreadStart

I tried this code in several variations:

• Fixed in the unity main thread
• In a coroutine
• In a separate thread

But the same error appeared in every variation.

Right now I'm a bit at a loss. I don't know what I could be doing wrong, altough I couldn't find any examples of people doing something similar.

I'd greatly appreciate some input on this matter. Should any more information be needed, I'd quickly add that of course.

Disclaimer: I have read the recommondations not to do texturing in real time, since that could impede performance. However, I am currently not interested in performance, since this will not be a major issue in the application I'm building.

[ckorris]

One thing that strikes me as odd is that between the call to dllz_apply_texture and the crash is a call to dllz_update_mesh. From your code, it looks like Retrieve, Resize and then Retrieve should all be called before another Update. I'm not positive it's related, but it's suspect.

Could you try these two things?

First, can you open ZEDManager, and at the top, change wrapperVerbose to true? Then when you run the app, you should get messages saved in C:/ProgramData/stereolabs/SL_Unity_wrapper.txt from the wrapper itself. There's a few log entries in the wrapper side of UpdateMesh, so that might shed light into what's going wrong.

Second, can you log how many times FetchMesh loops before you get the crash? I'm pretty sure it's a shared memory issue, but I'd like to know if it breaks the first time, the second time, or if it's inconsistent about how long it runs before breaking.

[Rosthouse]

I logged and debugged the code several times now, it always fails on the second iteration. The first time, the.

As you suggested, I also changed the wrapperVerbose variable to true. This gives me some output when I run the code without texturing, and the following output when I'm using texturing:

2019-04-16.10:14:33:    :       ENTER init
2019-04-16.10:14:33:    :       FUNC init : start Open Camera
2019-04-16.10:14:33:    :       *****************************************************

2019-04-16.10:14:35:    :       ENTER getCalibrationParameters
2019-04-16.10:14:35:    :       EXIT getCalibrationParameters
2019-04-16.10:14:35:    :       ENTER getCalibrationParameters
2019-04-16.10:14:35:    :       EXIT getCalibrationParameters
2019-04-16.10:14:35:    :       ENTER enableTracking
2019-04-16.10:14:35:    :       EXIT enableTracking
2019-04-16.10:14:36:    :       ENTER getCalibrationParameters
2019-04-16.10:14:36:    :       EXIT getCalibrationParameters
2019-04-16.10:14:36:    :       ENTER getCalibrationParameters
2019-04-16.10:14:36:    :       EXIT getCalibrationParameters
2019-04-16.10:14:43:    :       ENTER enableTracking
2019-04-16.10:14:43:    :       EXIT enableTracking
2019-04-16.10:14:43:    :       ENTER enableSpatialMapping
2019-04-16.10:14:44:    :       EXIT  enableSpatialMapping ErrCode:0
2019-04-16.10:14:44:    :       ENTER requestMeshAsync
2019-04-16.10:14:44:    :       EXIT requestMeshAsync
2019-04-16.10:14:44:    :       ENTER updateMesh
2019-04-16.10:14:44:    :       FUNC updateMesh : call getMeshRequestStatusAsync success
2019-04-16.10:14:44:    :       FUNC updateMesh : ca

The write call seems to crash right while writing the last line. However, I only get that output if I start an untextured spatial mapping first, which is even weirder.

Below I also included a bit more of a crash log. I think a shared memory issue is likely, as the first line of that log suggests:

Read from location FFFFFFFFFFFFFFF8 caused an access violation.

Context:
RDI:    0x00000000005fcb18  RSI: 0x0000000000000000  RAX:   0x0000000000000000
RBX:    0x0000000000000000  RCX: 0x0000000000000000  RDX:   0x0000000000000000
RIP:    0x00007ffb25a45ca2  RBP: 0x00000000005fc8c0  SegCs: 0x0000000000000033
EFlags: 0x0000000000010202  RSP: 0x00000000005fc7c0  SegSs: 0x000000000000002b
R8:     0x0000000000000000  R9:  0x00000000005fc8a0  R10:   0x0000000000000000
R11:    0x000000008b3507ac  R12: 0x0000000000000000  R13:   0x00000000005fcc08
R14:    0x000000008b350780  R15: 0x000000008b3507a0


Bytes at CS:EIP:
48 8b 4c c8 f8 48 8d 14 49 48 8b 07 48 8d 1c d0 

Mono DLL loaded successfully at 'D:\Tools\Unity\Editors\2018.3.11f1\Editor\Data\MonoBleedingEdge\EmbedRuntime\mono-2.0-bdwgc.dll'.


Stack Trace of Crashed Thread 8872:
0x00007FFB25A45CA2 (sl_core64) slutils::cv::Mat::ptr
0x00007FFB25A4567B (sl_core64) slutils::cv::Mat::ptr
0x00007FFB25A3C295 (sl_core64) slutils::cv::Mat::ptr
0x00007FFB25A405CA (sl_core64) slutils::cv::Mat::ptr
0x00007FFB25A3F71E (sl_core64) slutils::cv::Mat::ptr
0x00007FFB25A86902 (sl_core64) slutils::cv::Mat::ptr
0x00007FFB25C5DB3A (sl_core64) slutils::cv::Mat::ptr
0x00007FFB59C4728F (sl_unitywrapper) dllz_update_mesh
0x00007FFB59C3F88C (sl_unitywrapper) dllz_apply_texture
0x00000000470835DA (ZED) sl.ZEDCamera.dllz_apply_texture()
0x0000000047083393 (ZED) sl.ZEDCamera.ApplyTexture()
0x00000000470831F3 (MainAssembly) ProgressiveMapper.ApplyTexture()
0x00000000470817EB (MainAssembly) ProgressiveMapper.FetchMesh()
0x0000000047081373 (MainAssembly) ProgressiveMapper.RetrieveMeshFromCamera()
0x0000000047080C03 (MainAssembly) <UpdateMesh>d__18.MoveNext()
0x0000000046CEC470 (UnityEngine.CoreModule) UnityEngine.SetupCoroutine.InvokeMoveNext()
0x0000000046CEC660 (UnityEngine.CoreModule) <Module>.runtime_invoke_void_object_intptr()
0x00007FFB2C9FBE7B (mono-2.0-bdwgc) mono_get_runtime_build_info
0x00007FFB2C981E32 (mono-2.0-bdwgc) mono_perfcounters_init
0x00007FFB2C98AE3F (mono-2.0-bdwgc) mono_runtime_invoke
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 0000000140C0055A)
0x0000000140C0055A (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 0000000140BF08AA)
0x0000000140BF08AA (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 0000000140BC3F15)
0x0000000140BC3F15 (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 00000001406F2B0E)
0x00000001406F2B0E (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 000000014095C6FC)
0x000000014095C6FC (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 000000014095B2D7)
0x000000014095B2D7 (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 000000014095B3A3)
0x000000014095B3A3 (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 000000014095E661)
0x000000014095E661 (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 000000014133BCCF)
0x000000014133BCCF (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 000000014133A280)
0x000000014133A280 (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 000000014149663B)
0x000000014149663B (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 00000001414982D6)
0x00000001414982D6 (Unity) (function-name not available)
ERROR: SymGetSymFromAddr64, GetLastError: 'Attempt to access invalid address.' (Address: 000000014247F6DA)
0x000000014247F6DA (Unity) (function-name not available)
0x00007FFBA2293DC4 (KERNEL32) BaseThreadInitThunk
0x00007FFBA23E3691 (ntdll) RtlUserThreadStart

[ckorris]

Those log results are interesting.

I'd like to debug this on my side, if possible. Could you post any other relevant scripts in your project, like the ProgressiveMesh class and whatever you use to start/stop the scan? You can also send them (or the whole project if you want) to [email protected] if you'd prefer not to post them publicly.

[Rosthouse]

I'll send you a mail with the necessary sources. Should we find out what causes this problem, I'll post the solution here afterwards.

Thanks for looking into this by the way, this is a huge help :)

[ckorris]

For others reading this, the issue turned out to be a bug with how filtering and mesh textures are handled in our SDK. It's fixed for SDK v2.8.

[Rosthouse]

Jep, this fixed the issue. Unity does not crash anymore when calling ApplyTexture.

Huge thanks to you and your team @ckorris , That was quite a quick turnaround for this bug :)

[robot-Yang]

Hi, @ckorris

For others reading this, the issue turned out to be a bug with how filtering and mesh textures are handled in our SDK. It's fixed for SDK v2.8.

I think this problem still happens. The Unity crashes when I run spatial mapping with texturing option selected using OpenGL as Graphics API. The problem can be easily reproduced by:

1. Create a new unity project. Import ZED-unity plugin (v3.7.1).
2. Use OpenGL as Graphics API instead of Direct3D11.
3. Drag the ZED_Rig_Mono to Hierarchy. Select the Texturing under Spatial Mapping in ZED Manager.
4. Run the project, start Spatial Mapping, scan the environment, then click stop Spatial Mapping. Then Unity will crash.

Any help? Thanks in advance.

[Bvallon-sl]

Hi,

Are you on Windows ?

The ZED Unity plugin is only compatible with D3D11 on Windows and with OpenGL on Linux for the moment.

Best, Benjamin Vallon

[robot-Yang]

Thank you for the reply.

Yes, I am on Windows.

I understand, but it's a pity, because everything else works fine for me with OpenGL on Windows only if I don't select the Texuring option.

Are you (or others from stereolab) planning to make ZED Unity plugin compatible with OpenGL on Windows in the near future?

Best,

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment otherwise it will be automatically closed in 5 days

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment otherwise it will be automatically closed in 5 days