Open3D icon indicating copy to clipboard operation
Open3D copied to clipboard

Segmentation fault when updating mesh positions many times

Open SamTov opened this issue 3 years ago • 20 comments

Checklist

Describe the issue

I wrote a package for visualising particle simulations/trajectories built on top of Open3D. Code is available here. However, during the visualisation, if many frames (~>1000) or a larger number of particles (~>300) are being visualised, the program will eventually segfault.

It seems that there is some trade-off between number of particles and length of the simulation run and therefore I thought it could be associated with the update of the mesh position at each time step. I do so as follows:

  1. Render all mesh objects to screen (add them to an O3DVisualizer)
  2. Perform translation/rotations on previously instantiated mesh objects to get their state in the next time step
  3. Remove all geometries from the O3DVisualizer screen
  4. Add transformed meshes back to the Visualizer

It seem that for a large number of these updates there are memory issues. I have checked the raw memory usage and found that there is NO proliferation of memory use before the crash, i.e, there is plenty of memory left on the machine before the program segfaults and crashes. The memory usage is in fact very constant after initialisation as no new data is added after this point.

Steps to reproduce the bug

# With the ZnVis package

import numpy as np

import znvis as vis

if __name__ == "__main__":
    """
    Run the simple spheres example.
    """
    # Define the first particle.
    trajectory = np.random.uniform(-10, 10, (5000, 300, 3))
    mesh = vis.Sphere(radius=2.0, colour=np.array([30, 144, 255]) / 255, resolution=10)
    particle = vis.Particle(name="Blue", mesh=mesh, position=trajectory)

    # Define the second particle.
    trajectory_2 = np.random.uniform(-10, 10, (5000, 300, 3))
    mesh_2 = vis.Sphere(radius=1.0, colour=np.array([255, 140, 0]) / 255, resolution=10)
    particle_2 = vis.Particle(name="Orange", mesh=mesh_2, position=trajectory_2)

    # Construct the visualizer and run
    visualizer = vis.Visualizer(particles=[particle, particle_2], frame_rate=20)
    visualizer.run_visualization()

Error message

In the terminal one receives the standard: ``bash zsh: segmentation fault


I will also provide the error report as printed on my Mac. I can also provide the error from a linux machine for additional assistance.
```bash
-------------------------------------
Translated Report (Full Report Below)
-------------------------------------

Process:               python3.8 [22047]
Path:                  /Users/USER/*/python3
Identifier:            python3.8
Version:               ???
Code Type:             X86-64 (Native)
Parent Process:        zsh [17826]
Responsible:           Terminal [1728]
User ID:               501

Date/Time:             2022-01-21 00:07:02.9153 +0100
OS Version:            macOS 12.1 (21C52)
Report Version:        12
Bridge OS Version:     6.1 (19P647)
Anonymous UUID:        F2912C6F-A060-A98E-C043-4F3215C85928

Sleep/Wake UUID:       C98280A7-8872-45AA-B1A7-FB0A5FDB1827

Time Awake Since Boot: 28000 seconds
Time Since Wake:       1980 seconds

System Integrity Protection: enabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       UNKNOWN_0xD at 0x0000000000000000
Exception Codes:       0x000000000000000d, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Termination Reason:    Namespace SIGNAL, Code 11 Segmentation fault: 11
Terminating Process:   exc handler [22047]

VM Region Info: 0 is not in any region.  Bytes before following region: 4364251136
      REGION TYPE                    START - END         [ VSIZE] PRT/MAX SHRMOD  REGION DETAIL
      UNUSED SPACE AT START
--->  
      __TEXT                      104213000-104545000    [ 3272K] r-x/rwx SM=COW  ...R/*/python3.8

Thread 0 Crashed::  Dispatch queue: com.apple.main-thread
0   pybind.cpython-38-darwin.so   	       0x179da1b46 0x173a19000 + 104368966

Thread 1:
0   libsystem_kernel.dylib        	    0x7ff81af973da __semwait_signal + 10
1   libsystem_c.dylib             	    0x7ff81aeac8a9 nanosleep + 196
2   libc++.1.dylib                	    0x7ff81af3ac11 std::__1::this_thread::sleep_for(std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > const&) + 73
3   pybind.cpython-38-darwin.so   	       0x176efbc48 0x173a19000 + 55454792
4   pybind.cpython-38-darwin.so   	       0x176f0b09e 0x173a19000 + 55517342
5   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
6   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 2:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   libtensorflow_framework.2.dylib	       0x11d810c85 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   libtensorflow_framework.2.dylib	       0x11d810979 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   libtensorflow_framework.2.dylib	       0x11d81046c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   libtensorflow_framework.2.dylib	       0x11d80fd72 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 3:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   libtensorflow_framework.2.dylib	       0x11d810c85 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   libtensorflow_framework.2.dylib	       0x11d810979 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   libtensorflow_framework.2.dylib	       0x11d81046c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   libtensorflow_framework.2.dylib	       0x11d80fd72 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 4:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   libtensorflow_framework.2.dylib	       0x11d810c85 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   libtensorflow_framework.2.dylib	       0x11d810979 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   libtensorflow_framework.2.dylib	       0x11d81046c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   libtensorflow_framework.2.dylib	       0x11d80fd72 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 5:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   libtensorflow_framework.2.dylib	       0x11d810c85 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   libtensorflow_framework.2.dylib	       0x11d810979 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   libtensorflow_framework.2.dylib	       0x11d81046c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   libtensorflow_framework.2.dylib	       0x11d80fd72 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 6:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   libtensorflow_framework.2.dylib	       0x11d810c85 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   libtensorflow_framework.2.dylib	       0x11d810979 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   libtensorflow_framework.2.dylib	       0x11d81046c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   libtensorflow_framework.2.dylib	       0x11d80fd72 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 7:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   libtensorflow_framework.2.dylib	       0x11d810c85 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   libtensorflow_framework.2.dylib	       0x11d810979 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   libtensorflow_framework.2.dylib	       0x11d81046c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   libtensorflow_framework.2.dylib	       0x11d80fd72 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 8:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   libtensorflow_framework.2.dylib	       0x11d810c85 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   libtensorflow_framework.2.dylib	       0x11d810979 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   libtensorflow_framework.2.dylib	       0x11d81046c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   libtensorflow_framework.2.dylib	       0x11d80fd72 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 9:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   libtensorflow_framework.2.dylib	       0x11d810c85 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   libtensorflow_framework.2.dylib	       0x11d810979 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   libtensorflow_framework.2.dylib	       0x11d81046c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   libtensorflow_framework.2.dylib	       0x11d80fd72 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 10:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   libtensorflow_framework.2.dylib	       0x11d810c85 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   libtensorflow_framework.2.dylib	       0x11d810979 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   libtensorflow_framework.2.dylib	       0x11d81046c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   libtensorflow_framework.2.dylib	       0x11d80fd72 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 11:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   libtensorflow_framework.2.dylib	       0x11d810c85 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   libtensorflow_framework.2.dylib	       0x11d810979 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   libtensorflow_framework.2.dylib	       0x11d81046c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   libtensorflow_framework.2.dylib	       0x11d80fd72 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 12:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   libtensorflow_framework.2.dylib	       0x11d810c85 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   libtensorflow_framework.2.dylib	       0x11d810979 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   libtensorflow_framework.2.dylib	       0x11d81046c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   libtensorflow_framework.2.dylib	       0x11d80fd72 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 13:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   libtensorflow_framework.2.dylib	       0x11d810c85 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   libtensorflow_framework.2.dylib	       0x11d810979 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   libtensorflow_framework.2.dylib	       0x11d81046c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   libtensorflow_framework.2.dylib	       0x11d80fd72 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 14:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   libtensorflow_framework.2.dylib	       0x11d810c85 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   libtensorflow_framework.2.dylib	       0x11d810979 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   libtensorflow_framework.2.dylib	       0x11d81046c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   libtensorflow_framework.2.dylib	       0x11d80fd72 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 15:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   libtensorflow_framework.2.dylib	       0x11d810c85 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   libtensorflow_framework.2.dylib	       0x11d810979 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   libtensorflow_framework.2.dylib	       0x11d81046c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   libtensorflow_framework.2.dylib	       0x11d80fd72 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 16:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   libtensorflow_framework.2.dylib	       0x11d810c85 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   libtensorflow_framework.2.dylib	       0x11d810979 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   libtensorflow_framework.2.dylib	       0x11d81046c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   libtensorflow_framework.2.dylib	       0x11d80fd72 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 17:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   libtensorflow_framework.2.dylib	       0x11d810c85 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   libtensorflow_framework.2.dylib	       0x11d810979 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   libtensorflow_framework.2.dylib	       0x11d81046c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   libtensorflow_framework.2.dylib	       0x11d80fd72 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 18:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   _pywrap_tensorflow_internal.so	       0x13b3b3555 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   _pywrap_tensorflow_internal.so	       0x13b3b3249 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   _pywrap_tensorflow_internal.so	       0x13b3b2d3c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   _pywrap_tensorflow_internal.so	       0x13b3b2642 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 19:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   _pywrap_tensorflow_internal.so	       0x13b3b3555 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   _pywrap_tensorflow_internal.so	       0x13b3b3249 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   _pywrap_tensorflow_internal.so	       0x13b3b2d3c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   _pywrap_tensorflow_internal.so	       0x13b3b2642 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 20:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   _pywrap_tensorflow_internal.so	       0x13b3b3555 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   _pywrap_tensorflow_internal.so	       0x13b3b3249 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   _pywrap_tensorflow_internal.so	       0x13b3b2d3c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   _pywrap_tensorflow_internal.so	       0x13b3b2642 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 21:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   _pywrap_tensorflow_internal.so	       0x13b3b3555 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   _pywrap_tensorflow_internal.so	       0x13b3b3249 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   _pywrap_tensorflow_internal.so	       0x13b3b2d3c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   _pywrap_tensorflow_internal.so	       0x13b3b2642 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 22:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   _pywrap_tensorflow_internal.so	       0x13b3b3555 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   _pywrap_tensorflow_internal.so	       0x13b3b3249 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   _pywrap_tensorflow_internal.so	       0x13b3b2d3c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   _pywrap_tensorflow_internal.so	       0x13b3b2642 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 23:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   _pywrap_tensorflow_internal.so	       0x13b3b3555 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   _pywrap_tensorflow_internal.so	       0x13b3b3249 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   _pywrap_tensorflow_internal.so	       0x13b3b2d3c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   _pywrap_tensorflow_internal.so	       0x13b3b2642 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 24:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   _pywrap_tensorflow_internal.so	       0x13b3b3555 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   _pywrap_tensorflow_internal.so	       0x13b3b3249 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   _pywrap_tensorflow_internal.so	       0x13b3b2d3c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   _pywrap_tensorflow_internal.so	       0x13b3b2642 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 25:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   _pywrap_tensorflow_internal.so	       0x13b3b3555 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   _pywrap_tensorflow_internal.so	       0x13b3b3249 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   _pywrap_tensorflow_internal.so	       0x13b3b2d3c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   _pywrap_tensorflow_internal.so	       0x13b3b2642 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 26:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   _pywrap_tensorflow_internal.so	       0x13b3b3555 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   _pywrap_tensorflow_internal.so	       0x13b3b3249 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   _pywrap_tensorflow_internal.so	       0x13b3b2d3c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   _pywrap_tensorflow_internal.so	       0x13b3b2642 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 27:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   _pywrap_tensorflow_internal.so	       0x13b3b3555 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   _pywrap_tensorflow_internal.so	       0x13b3b3249 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   _pywrap_tensorflow_internal.so	       0x13b3b2d3c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   _pywrap_tensorflow_internal.so	       0x13b3b2642 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 28:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   _pywrap_tensorflow_internal.so	       0x13b3b3555 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   _pywrap_tensorflow_internal.so	       0x13b3b3249 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   _pywrap_tensorflow_internal.so	       0x13b3b2d3c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   _pywrap_tensorflow_internal.so	       0x13b3b2642 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 29:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   _pywrap_tensorflow_internal.so	       0x13b3b3555 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   _pywrap_tensorflow_internal.so	       0x13b3b3249 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   _pywrap_tensorflow_internal.so	       0x13b3b2d3c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   _pywrap_tensorflow_internal.so	       0x13b3b2642 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 30:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   _pywrap_tensorflow_internal.so	       0x13b3b3555 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   _pywrap_tensorflow_internal.so	       0x13b3b3249 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   _pywrap_tensorflow_internal.so	       0x13b3b2d3c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   _pywrap_tensorflow_internal.so	       0x13b3b2642 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 31:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   _pywrap_tensorflow_internal.so	       0x13b3b3555 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   _pywrap_tensorflow_internal.so	       0x13b3b3249 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   _pywrap_tensorflow_internal.so	       0x13b3b2d3c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   _pywrap_tensorflow_internal.so	       0x13b3b2642 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 32:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   _pywrap_tensorflow_internal.so	       0x13b3b3555 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   _pywrap_tensorflow_internal.so	       0x13b3b3249 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   _pywrap_tensorflow_internal.so	       0x13b3b2d3c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   _pywrap_tensorflow_internal.so	       0x13b3b2642 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 33:
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   _pywrap_tensorflow_internal.so	       0x13b3b3555 Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*) + 229
4   _pywrap_tensorflow_internal.so	       0x13b3b3249 Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) + 889
5   _pywrap_tensorflow_internal.so	       0x13b3b2d3c Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1532
6   _pywrap_tensorflow_internal.so	       0x13b3b2642 tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const + 66
7   libtensorflow_framework.2.dylib	       0x11d7fb9b8 tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) + 104
8   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
9   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 34:
0   libsystem_pthread.dylib       	    0x7ff81afccfec start_wqthread + 0

Thread 35:: com.apple.NSEventThread
0   libsystem_kernel.dylib        	    0x7ff81af94aba mach_msg_trap + 10
1   libsystem_kernel.dylib        	    0x7ff81af94e2b mach_msg + 59
2   CoreFoundation                	    0x7ff81b098af2 __CFRunLoopServiceMachPort + 319
3   CoreFoundation                	    0x7ff81b0971cb __CFRunLoopRun + 1325
4   CoreFoundation                	    0x7ff81b0965dd CFRunLoopRunSpecific + 563
5   AppKit                        	    0x7ff81dc33d98 _NSEventThread + 132
6   libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
7   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 36:: JobSystem::loop
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   pybind.cpython-38-darwin.so   	       0x179e63eec utils::JobSystem::loop(utils::JobSystem::ThreadState*) + 220
4   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 37:: JobSystem::loop
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   pybind.cpython-38-darwin.so   	       0x179e63eec utils::JobSystem::loop(utils::JobSystem::ThreadState*) + 220
4   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 38:: JobSystem::loop
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   pybind.cpython-38-darwin.so   	       0x179e63eec utils::JobSystem::loop(utils::JobSystem::ThreadState*) + 220
4   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 39:: JobSystem::loop
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   pybind.cpython-38-darwin.so   	       0x179e63eec utils::JobSystem::loop(utils::JobSystem::ThreadState*) + 220
4   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 40:: JobSystem::loop
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   pybind.cpython-38-darwin.so   	       0x179e63eec utils::JobSystem::loop(utils::JobSystem::ThreadState*) + 220
4   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 41:: JobSystem::loop
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   pybind.cpython-38-darwin.so   	       0x179e63eec utils::JobSystem::loop(utils::JobSystem::ThreadState*) + 220
4   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 42:: JobSystem::loop
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   pybind.cpython-38-darwin.so   	       0x179e63eec utils::JobSystem::loop(utils::JobSystem::ThreadState*) + 220
4   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 43:: FEngine::loop
0   libsystem_kernel.dylib        	    0x7ff81af97506 __psynch_cvwait + 10
1   libsystem_pthread.dylib       	    0x7ff81afd1a69 _pthread_cond_wait + 1224
2   libc++.1.dylib                	    0x7ff81af30ec2 std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
3   pybind.cpython-38-darwin.so   	       0x179ddf4db 0x173a19000 + 104621275
4   libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15

Thread 44:
0   libsystem_pthread.dylib       	    0x7ff81afccfec start_wqthread + 0

Thread 45:
0   libsystem_kernel.dylib        	    0x7ff81af9ce4a __select + 10
1   python3.8                     	       0x10448a98f time_sleep + 351
2   python3.8                     	       0x1042a04f4 cfunction_vectorcall_O + 100
3   python3.8                     	       0x10438ce32 call_function + 162
4   python3.8                     	       0x1043893e2 _PyEval_EvalFrameDefault + 43522
5   python3.8                     	       0x104242818 _PyFunction_Vectorcall + 248
6   python3.8                     	       0x10424649e method_vectorcall + 494
7   python3.8                     	       0x104241a69 PyVectorcall_Call + 121
8   python3.8                     	       0x10438a994 _PyEval_EvalFrameDefault + 49076
9   python3.8                     	       0x104242818 _PyFunction_Vectorcall + 248
10  python3.8                     	       0x10438ce32 call_function + 162
11  python3.8                     	       0x104389353 _PyEval_EvalFrameDefault + 43379
12  python3.8                     	       0x104242818 _PyFunction_Vectorcall + 248
13  python3.8                     	       0x10438ce32 call_function + 162
14  python3.8                     	       0x104389353 _PyEval_EvalFrameDefault + 43379
15  python3.8                     	       0x104242818 _PyFunction_Vectorcall + 248
16  python3.8                     	       0x10424649e method_vectorcall + 494
17  python3.8                     	       0x104241a69 PyVectorcall_Call + 121
18  python3.8                     	       0x10448db64 t_bootstrap + 212
19  python3.8                     	       0x104418654 pythread_wrapper + 36
20  libsystem_pthread.dylib       	    0x7ff81afd14f4 _pthread_start + 125
21  libsystem_pthread.dylib       	    0x7ff81afcd00f thread_start + 15


Thread 0 crashed with X86 Thread State (64-bit):
  rax: 0x00007fd64853c900  rbx: 0x0000000000000000  rcx: 0x0000000000000001  rdx: 0x00007fd656e6b330
  rdi: 0x0000000000000000  rsi: 0x00007fd64fa00000  rbp: 0x0000000003001102  rsp: 0x00007ff7bbce76b8
   r8: 0x1000000000000000   r9: 0x0000000000000000  r10: 0x000060000126a8e0  r11: 0x0000000000000010
  r12: 0x0000000000000000  r13: 0x000060000126a8f8  r14: 0x0000000000000000  r15: 0x0000000000000000
  rip: 0x0000000179da1b46  rfl: 0x0000000000010246  cr2: 0x00007fd63d613000
  
Logical CPU:     8
Error Code:      0x00000000 
Trap Number:     13

Thread 0 instruction stream:
  04 f3 0f 59 f4 f3 0f 58-f5 f3 0f 58 f0 0f 28 c1  ...Y...X...X..(.
  f3 0f 5c c6 66 41 0f 7e-c4 44 89 f0 83 e0 07 48  ..\.fA.~.D.....H
  c1 e0 32 49 09 c4 48 8b-44 24 c8 8b 04 88 89 44  ..2I..H.D$.....D
  24 c4 41 c1 ee 03 44 23-74 24 b8 41 83 e6 01 49  $.A...D#t$.A...I
  89 c9 44 89 d1 4d 8b 14-1f 4b 8d 44 6d 00 4d 8d  ..D..M...K.Dm.M.
  2c c2 41 89 ff 0f 1f 44-00 00 49 8b 02 4c 8b 00  ,.A....D..I..L..
 [41]8b 98 00 02 00 00 44-89 da 44 0f b6 98 10 01  A......D..D.....	<==
  00 00 41 83 e3 03 83 e5-fc 44 09 dd 41 89 d3 41  ..A......D..A..A
  8b 7a 08 48 89 46 08 89-7e 10 8b 44 24 c4 89 46  .z.H.F..~..D$..F
  14 89 6e 18 66 44 89 4e-1c 44 88 5e 1e c6 46 1f  ..n.fD.N.D.^..F.
  00 41 0f b6 80 08 02 00-00 84 c0 0f 95 c2 3c 03  .A............<.
  0f 95 c0 89 df c1 ef 18-c1 eb 1d 44 21 fb f7 d3  ...........D!...

Binary Images:
       0x173a19000 -        0x17aa84fff pybind.cpython-38-darwin.so (*) <3208e253-2e02-3b91-ae4c-fb160da00117> /Users/USER/*/pybind.cpython-38-darwin.so
    0x7ff81af94000 -     0x7ff81afcafff libsystem_kernel.dylib (*) <5aa1e5be-b5b8-3a02-9885-a8c99e0ca378> /usr/lib/system/libsystem_kernel.dylib
    0x7ff81ae9c000 -     0x7ff81af24fff libsystem_c.dylib (*) <e58814cc-dcb7-35a5-badc-e367ed3ac207> /usr/lib/system/libsystem_c.dylib
    0x7ff81af25000 -     0x7ff81af7dfff libc++.1.dylib (*) <c006ac8c-2c1c-3ca5-9085-9a754386bf42> /usr/lib/libc++.1.dylib
    0x7ff81afcb000 -     0x7ff81afd6fff libsystem_pthread.dylib (*) <6c7561b4-4b92-3f45-921e-abe669299844> /usr/lib/system/libsystem_pthread.dylib
       0x11cdbb000 -        0x11df4efff libtensorflow_framework.2.dylib (*) <6bd6ae58-824c-3789-9db8-0434638315cd> /Users/USER/*/libtensorflow_framework.2.dylib
       0x13a577000 -        0x149938fff _pywrap_tensorflow_internal.so (*) <7e600010-fab1-3f38-a272-e86634fc9896> /Users/USER/*/_pywrap_tensorflow_internal.so
    0x7ff81b018000 -     0x7ff81b51afff com.apple.CoreFoundation (6.9) <eab0e216-2a9e-31ce-9164-fdf3ebcf7bd3> /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
    0x7ff81da87000 -     0x7ff81e915fff com.apple.AppKit (6.9) <d23b9681-3764-3298-a716-fbb511dd5a7c> /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
       0x104213000 -        0x104544fff python3.8 (*) <8ed09e45-5896-34eb-966c-bb60ab3887ac> /Users/USER/*/python3.8

External Modification Summary:
  Calls made by other processes targeting this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by all processes on this machine:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0

VM Region Summary:
ReadOnly portion of Libraries: Total=2.0G resident=0K(0%) swapped_out_or_unallocated=2.0G(100%)
Writable regions: Total=1.7G written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=1.7G(100%)

                                VIRTUAL   REGION 
REGION TYPE                        SIZE    COUNT (non-coalesced) 
===========                     =======  ======= 
Accelerate framework               128K        1 
Activity Tracing                   256K        1 
CG backing stores                 2688K        4 
CG image                            72K        5 
ColorSync                          228K       26 
CoreAnimation                      144K        8 
CoreGraphics                        16K        3 
CoreUI image data                  972K        7 
Foundation                          16K        1 
IOKit                             15.5M        2 
Kernel Alloc Once                    8K        1 
MALLOC                           778.6M      829 
MALLOC guard page                   32K        8 
MALLOC_LARGE (reserved)            904K        4         reserved VM address space (unallocated)
MALLOC_NANO (reserved)           384.0M        1         reserved VM address space (unallocated)
OpenGL GLSL                        384K        5 
STACK GUARD                        184K       46 
Stack                             54.4M       46 
VM_ALLOCATE                      101.1M      415 
VM_ALLOCATE (reserved)           384.0M        4         reserved VM address space (unallocated)
__DATA                            67.9M      719 
__DATA_CONST                      13.3M      169 
__DATA_DIRTY                       541K       99 
__FONT_DATA                          4K        1 
__GLSLBUILTINS                    5176K        1 
__LINKEDIT                         1.0G      339 
__OBJC_RO                         81.8M        1 
__OBJC_RW                         3136K        2 
__TEXT                           985.3M      633 
__UNICODE                          588K        1 
dyld private memory               1280K        3 
mapped file                       89.8M       24 
shared memory                      780K       18 
===========                     =======  ======= 
TOTAL                              3.9G     3427 
TOTAL, minus reserved VM space     3.1G     3427 



-----------
Full Report
-----------

{"app_name":"python3.8","timestamp":"2022-01-21 00:07:03.00 +0100","app_version":"","slice_uuid":"8ed09e45-5896-34eb-966c-bb60ab3887ac","build_version":"","platform":1,"share_with_app_devs":0,"is_first_party":1,"bug_type":"309","os_version":"macOS 12.1 (21C52)","incident_id":"41973686-9679-4523-88A9-3BB1EC6CFEE2","name":"python3.8"}
{
  "uptime" : 28000,
  "procLaunch" : "2022-01-21 00:06:06.1055 +0100",
  "procRole" : "Foreground",
  "version" : 2,
  "userID" : 501,
  "deployVersion" : 210,
  "modelCode" : "MacBookPro16,1",
  "procStartAbsTime" : 28164777310408,
  "coalitionID" : 1355,
  "osVersion" : {
    "train" : "macOS 12.1",
    "build" : "21C52",
    "releaseType" : "User"
  },
  "captureTime" : "2022-01-21 00:07:02.9153 +0100",
  "incident" : "41973686-9679-4523-88A9-3BB1EC6CFEE2",
  "bug_type" : "309",
  "pid" : 22047,
  "procExitAbsTime" : 28221586927086,
  "cpuType" : "X86-64",
  "procName" : "python3.8",
  "procPath" : "\/Users\/USER\/*\/python3",
  "parentProc" : "zsh",
  "parentPid" : 17826,
  "coalitionName" : "com.apple.Terminal",
  "crashReporterKey" : "F2912C6F-A060-A98E-C043-4F3215C85928",
  "responsiblePid" : 1728,
  "responsibleProc" : "Terminal",
  "wakeTime" : 1980,
  "bridgeVersion" : {"build":"19P647","train":"6.1"},
  "sleepWakeUUID" : "C98280A7-8872-45AA-B1A7-FB0A5FDB1827",
  "sip" : "enabled",
  "vmRegionInfo" : "0 is not in any region.  Bytes before following region: 4364251136\n      REGION TYPE                    START - END         [ VSIZE] PRT\/MAX SHRMOD  REGION DETAIL\n      UNUSED SPACE AT START\n--->  \n      __TEXT                      104213000-104545000    [ 3272K] r-x\/rwx SM=COW  ...R\/*\/python3.8",
  "isCorpse" : 1,
  "exception" : {"codes":"0x000000000000000d, 0x0000000000000000","rawCodes":[13,0],"type":"EXC_BAD_ACCESS","signal":"SIGSEGV","subtype":"UNKNOWN_0xD at 0x0000000000000000"},
  "termination" : {"flags":0,"code":11,"namespace":"SIGNAL","indicator":"Segmentation fault: 11","byProc":"exc handler","byPid":22047},
  "vmregioninfo" : "0 is not in any region.  Bytes before following region: 4364251136\n      REGION TYPE                    START - END         [ VSIZE] PRT\/MAX SHRMOD  REGION DETAIL\n      UNUSED SPACE AT START\n--->  \n      __TEXT                      104213000-104545000    [ 3272K] r-x\/rwx SM=COW  ...R\/*\/python3.8",
  "extMods" : {"caller":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"system":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"targeted":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"warnings":0},
  "faultingThread" : 0,
  "threads" : [{"triggered":true,"id":510527,"instructionState":{"instructionStream":{"bytes":[4,243,15,89,244,243,15,88,245,243,15,88,240,15,40,193,243,15,92,198,102,65,15,126,196,68,137,240,131,224,7,72,193,224,50,73,9,196,72,139,68,36,200,139,4,136,137,68,36,196,65,193,238,3,68,35,116,36,184,65,131,230,1,73,137,201,68,137,209,77,139,20,31,75,141,68,109,0,77,141,44,194,65,137,255,15,31,68,0,0,73,139,2,76,139,0,65,139,152,0,2,0,0,68,137,218,68,15,182,152,16,1,0,0,65,131,227,3,131,229,252,68,9,221,65,137,211,65,139,122,8,72,137,70,8,137,126,16,139,68,36,196,137,70,20,137,110,24,102,68,137,78,28,68,136,94,30,198,70,31,0,65,15,182,128,8,2,0,0,132,192,15,149,194,60,3,15,149,192,137,223,193,239,24,193,235,29,68,33,251,247,211],"offset":96}},"threadState":{"r13":{"value":105553135577336},"rax":{"value":140558313179392},"rflags":{"value":66118},"cpu":{"value":8},"r14":{"value":0},"rsi":{"value":140558435614720},"r8":{"value":1152921504606846976},"cr2":{"value":140558129508352},"rdx":{"value":140558557688624},"r10":{"value":105553135577312},"r9":{"value":0},"r15":{"value":0},"rbx":{"value":0},"trap":{"value":13},"err":{"value":0},"r11":{"value":16},"rip":{"value":6339304262,"matchesCrashFrame":1},"rbp":{"value":50336002},"rsp":{"value":140701984519864},"r12":{"value":0},"rcx":{"value":1},"flavor":"x86_THREAD_STATE","rdi":{"value":0}},"queue":"com.apple.main-thread","frames":[{"imageOffset":104368966,"imageIndex":0}]},{"id":510675,"frames":[{"imageOffset":13274,"symbol":"__semwait_signal","symbolLocation":10,"imageIndex":1},{"imageOffset":67753,"symbol":"nanosleep","symbolLocation":196,"imageIndex":2},{"imageOffset":89105,"symbol":"std::__1::this_thread::sleep_for(std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > const&)","symbolLocation":73,"imageIndex":3},{"imageOffset":55454792,"imageIndex":0},{"imageOffset":55517342,"imageIndex":0},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510679,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":10837125,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":5},{"imageOffset":10836345,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":5},{"imageOffset":10835052,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":5},{"imageOffset":10833266,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":5},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510680,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":10837125,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":5},{"imageOffset":10836345,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":5},{"imageOffset":10835052,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":5},{"imageOffset":10833266,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":5},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510681,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":10837125,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":5},{"imageOffset":10836345,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":5},{"imageOffset":10835052,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":5},{"imageOffset":10833266,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":5},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510682,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":10837125,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":5},{"imageOffset":10836345,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":5},{"imageOffset":10835052,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":5},{"imageOffset":10833266,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":5},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510683,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":10837125,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":5},{"imageOffset":10836345,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":5},{"imageOffset":10835052,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":5},{"imageOffset":10833266,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":5},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510684,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":10837125,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":5},{"imageOffset":10836345,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":5},{"imageOffset":10835052,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":5},{"imageOffset":10833266,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":5},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510685,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":10837125,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":5},{"imageOffset":10836345,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":5},{"imageOffset":10835052,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":5},{"imageOffset":10833266,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":5},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510686,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":10837125,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":5},{"imageOffset":10836345,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":5},{"imageOffset":10835052,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":5},{"imageOffset":10833266,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":5},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510687,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":10837125,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":5},{"imageOffset":10836345,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":5},{"imageOffset":10835052,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":5},{"imageOffset":10833266,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":5},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510688,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":10837125,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":5},{"imageOffset":10836345,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":5},{"imageOffset":10835052,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":5},{"imageOffset":10833266,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":5},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510689,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":10837125,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":5},{"imageOffset":10836345,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":5},{"imageOffset":10835052,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":5},{"imageOffset":10833266,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":5},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510690,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":10837125,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":5},{"imageOffset":10836345,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":5},{"imageOffset":10835052,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":5},{"imageOffset":10833266,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":5},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510691,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":10837125,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":5},{"imageOffset":10836345,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":5},{"imageOffset":10835052,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":5},{"imageOffset":10833266,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":5},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510692,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":10837125,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":5},{"imageOffset":10836345,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":5},{"imageOffset":10835052,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":5},{"imageOffset":10833266,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":5},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510693,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":10837125,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":5},{"imageOffset":10836345,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":5},{"imageOffset":10835052,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":5},{"imageOffset":10833266,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":5},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510694,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":10837125,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":5},{"imageOffset":10836345,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":5},{"imageOffset":10835052,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":5},{"imageOffset":10833266,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":5},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510695,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":14927189,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":6},{"imageOffset":14926409,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":6},{"imageOffset":14925116,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":6},{"imageOffset":14923330,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":6},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510696,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":14927189,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":6},{"imageOffset":14926409,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":6},{"imageOffset":14925116,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":6},{"imageOffset":14923330,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":6},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510697,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":14927189,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":6},{"imageOffset":14926409,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":6},{"imageOffset":14925116,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":6},{"imageOffset":14923330,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":6},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510698,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":14927189,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":6},{"imageOffset":14926409,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":6},{"imageOffset":14925116,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":6},{"imageOffset":14923330,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":6},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510699,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":14927189,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":6},{"imageOffset":14926409,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":6},{"imageOffset":14925116,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":6},{"imageOffset":14923330,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":6},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510700,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":14927189,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":6},{"imageOffset":14926409,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":6},{"imageOffset":14925116,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":6},{"imageOffset":14923330,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":6},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510701,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":14927189,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":6},{"imageOffset":14926409,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":6},{"imageOffset":14925116,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":6},{"imageOffset":14923330,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":6},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510702,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":14927189,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":6},{"imageOffset":14926409,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":6},{"imageOffset":14925116,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":6},{"imageOffset":14923330,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":6},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510703,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":14927189,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":6},{"imageOffset":14926409,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":6},{"imageOffset":14925116,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":6},{"imageOffset":14923330,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":6},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510704,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":14927189,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":6},{"imageOffset":14926409,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":6},{"imageOffset":14925116,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":6},{"imageOffset":14923330,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":6},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510705,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":14927189,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":6},{"imageOffset":14926409,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":6},{"imageOffset":14925116,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":6},{"imageOffset":14923330,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":6},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510706,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":14927189,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":6},{"imageOffset":14926409,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":6},{"imageOffset":14925116,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":6},{"imageOffset":14923330,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":6},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510707,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":14927189,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":6},{"imageOffset":14926409,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":6},{"imageOffset":14925116,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":6},{"imageOffset":14923330,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":6},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510708,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":14927189,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":6},{"imageOffset":14926409,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":6},{"imageOffset":14925116,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":6},{"imageOffset":14923330,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":6},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510709,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":14927189,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":6},{"imageOffset":14926409,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":6},{"imageOffset":14925116,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":6},{"imageOffset":14923330,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":6},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510710,"frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":14927189,"symbol":"Eigen::EventCount::CommitWait(Eigen::EventCount::Waiter*)","symbolLocation":229,"imageIndex":6},{"imageOffset":14926409,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*)","symbolLocation":889,"imageIndex":6},{"imageOffset":14925116,"symbol":"Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)","symbolLocation":1532,"imageIndex":6},{"imageOffset":14923330,"symbol":"tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()::operator()() const","symbolLocation":66,"imageIndex":6},{"imageOffset":10750392,"symbol":"tensorflow::(anonymous namespace)::PThread::ThreadFn(void*)","symbolLocation":104,"imageIndex":5},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510711,"frames":[{"imageOffset":8172,"symbol":"start_wqthread","symbolLocation":0,"imageIndex":4}]},{"id":510717,"name":"com.apple.NSEventThread","frames":[{"imageOffset":2746,"symbol":"mach_msg_trap","symbolLocation":10,"imageIndex":1},{"imageOffset":3627,"symbol":"mach_msg","symbolLocation":59,"imageIndex":1},{"imageOffset":527090,"symbol":"__CFRunLoopServiceMachPort","symbolLocation":319,"imageIndex":7},{"imageOffset":520651,"symbol":"__CFRunLoopRun","symbolLocation":1325,"imageIndex":7},{"imageOffset":517597,"symbol":"CFRunLoopRunSpecific","symbolLocation":563,"imageIndex":7},{"imageOffset":1756568,"symbol":"_NSEventThread","symbolLocation":132,"imageIndex":8},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510771,"name":"JobSystem::loop","frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":105164524,"symbol":"utils::JobSystem::loop(utils::JobSystem::ThreadState*)","symbolLocation":220,"imageIndex":0},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510772,"name":"JobSystem::loop","frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":105164524,"symbol":"utils::JobSystem::loop(utils::JobSystem::ThreadState*)","symbolLocation":220,"imageIndex":0},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510773,"name":"JobSystem::loop","frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":105164524,"symbol":"utils::JobSystem::loop(utils::JobSystem::ThreadState*)","symbolLocation":220,"imageIndex":0},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510774,"name":"JobSystem::loop","frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":105164524,"symbol":"utils::JobSystem::loop(utils::JobSystem::ThreadState*)","symbolLocation":220,"imageIndex":0},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510775,"name":"JobSystem::loop","frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":105164524,"symbol":"utils::JobSystem::loop(utils::JobSystem::ThreadState*)","symbolLocation":220,"imageIndex":0},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510776,"name":"JobSystem::loop","frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":105164524,"symbol":"utils::JobSystem::loop(utils::JobSystem::ThreadState*)","symbolLocation":220,"imageIndex":0},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510777,"name":"JobSystem::loop","frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":105164524,"symbol":"utils::JobSystem::loop(utils::JobSystem::ThreadState*)","symbolLocation":220,"imageIndex":0},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510778,"name":"FEngine::loop","frames":[{"imageOffset":13574,"symbol":"__psynch_cvwait","symbolLocation":10,"imageIndex":1},{"imageOffset":27241,"symbol":"_pthread_cond_wait","symbolLocation":1224,"imageIndex":4},{"imageOffset":48834,"symbol":"std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)","symbolLocation":18,"imageIndex":3},{"imageOffset":104621275,"imageIndex":0},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]},{"id":510815,"frames":[{"imageOffset":8172,"symbol":"start_wqthread","symbolLocation":0,"imageIndex":4}]},{"id":510884,"frames":[{"imageOffset":36426,"symbol":"__select","symbolLocation":10,"imageIndex":1},{"imageOffset":2587023,"symbol":"time_sleep","symbolLocation":351,"imageIndex":9},{"imageOffset":578804,"symbol":"cfunction_vectorcall_O","symbolLocation":100,"imageIndex":9},{"imageOffset":1547826,"symbol":"call_function","symbolLocation":162,"imageIndex":9},{"imageOffset":1532898,"symbol":"_PyEval_EvalFrameDefault","symbolLocation":43522,"imageIndex":9},{"imageOffset":194584,"symbol":"_PyFunction_Vectorcall","symbolLocation":248,"imageIndex":9},{"imageOffset":210078,"symbol":"method_vectorcall","symbolLocation":494,"imageIndex":9},{"imageOffset":191081,"symbol":"PyVectorcall_Call","symbolLocation":121,"imageIndex":9},{"imageOffset":1538452,"symbol":"_PyEval_EvalFrameDefault","symbolLocation":49076,"imageIndex":9},{"imageOffset":194584,"symbol":"_PyFunction_Vectorcall","symbolLocation":248,"imageIndex":9},{"imageOffset":1547826,"symbol":"call_function","symbolLocation":162,"imageIndex":9},{"imageOffset":1532755,"symbol":"_PyEval_EvalFrameDefault","symbolLocation":43379,"imageIndex":9},{"imageOffset":194584,"symbol":"_PyFunction_Vectorcall","symbolLocation":248,"imageIndex":9},{"imageOffset":1547826,"symbol":"call_function","symbolLocation":162,"imageIndex":9},{"imageOffset":1532755,"symbol":"_PyEval_EvalFrameDefault","symbolLocation":43379,"imageIndex":9},{"imageOffset":194584,"symbol":"_PyFunction_Vectorcall","symbolLocation":248,"imageIndex":9},{"imageOffset":210078,"symbol":"method_vectorcall","symbolLocation":494,"imageIndex":9},{"imageOffset":191081,"symbol":"PyVectorcall_Call","symbolLocation":121,"imageIndex":9},{"imageOffset":2599780,"symbol":"t_bootstrap","symbolLocation":212,"imageIndex":9},{"imageOffset":2119252,"symbol":"pythread_wrapper","symbolLocation":36,"imageIndex":9},{"imageOffset":25844,"symbol":"_pthread_start","symbolLocation":125,"imageIndex":4},{"imageOffset":8207,"symbol":"thread_start","symbolLocation":15,"imageIndex":4}]}],
  "usedImages" : [
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 6234935296,
    "size" : 117882880,
    "uuid" : "3208e253-2e02-3b91-ae4c-fb160da00117",
    "path" : "\/Users\/USER\/*\/pybind.cpython-38-darwin.so",
    "name" : "pybind.cpython-38-darwin.so"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 140703581159424,
    "size" : 225280,
    "uuid" : "5aa1e5be-b5b8-3a02-9885-a8c99e0ca378",
    "path" : "\/usr\/lib\/system\/libsystem_kernel.dylib",
    "name" : "libsystem_kernel.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 140703580143616,
    "size" : 561152,
    "uuid" : "e58814cc-dcb7-35a5-badc-e367ed3ac207",
    "path" : "\/usr\/lib\/system\/libsystem_c.dylib",
    "name" : "libsystem_c.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 140703580704768,
    "size" : 364544,
    "uuid" : "c006ac8c-2c1c-3ca5-9085-9a754386bf42",
    "path" : "\/usr\/lib\/libc++.1.dylib",
    "name" : "libc++.1.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 140703581384704,
    "size" : 49152,
    "uuid" : "6c7561b4-4b92-3f45-921e-abe669299844",
    "path" : "\/usr\/lib\/system\/libsystem_pthread.dylib",
    "name" : "libsystem_pthread.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4779126784,
    "size" : 18432000,
    "uuid" : "6bd6ae58-824c-3789-9db8-0434638315cd",
    "path" : "\/Users\/USER\/*\/libtensorflow_framework.2.dylib",
    "name" : "libtensorflow_framework.2.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 5273776128,
    "size" : 255598592,
    "uuid" : "7e600010-fab1-3f38-a272-e86634fc9896",
    "path" : "\/Users\/USER\/*\/_pywrap_tensorflow_internal.so",
    "name" : "_pywrap_tensorflow_internal.so"
  },
  {
    "source" : "P",
    "arch" : "x86_64h",
    "base" : 140703581700096,
    "CFBundleShortVersionString" : "6.9",
    "CFBundleIdentifier" : "com.apple.CoreFoundation",
    "size" : 5255168,
    "uuid" : "eab0e216-2a9e-31ce-9164-fdf3ebcf7bd3",
    "path" : "\/System\/Library\/Frameworks\/CoreFoundation.framework\/Versions\/A\/CoreFoundation",
    "name" : "CoreFoundation",
    "CFBundleVersion" : "1856.105"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 140703626194944,
    "CFBundleShortVersionString" : "6.9",
    "CFBundleIdentifier" : "com.apple.AppKit",
    "size" : 15265792,
    "uuid" : "d23b9681-3764-3298-a716-fbb511dd5a7c",
    "path" : "\/System\/Library\/Frameworks\/AppKit.framework\/Versions\/C\/AppKit",
    "name" : "AppKit",
    "CFBundleVersion" : "2113.20.111"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4364251136,
    "size" : 3350528,
    "uuid" : "8ed09e45-5896-34eb-966c-bb60ab3887ac",
    "path" : "\/Users\/USER\/*\/python3.8",
    "name" : "python3.8"
  }
],
  "sharedCache" : {
  "base" : 140703578157056,
  "size" : 15216738304,
  "uuid" : "40432a03-88d3-305f-9c0c-e7549e71d927"
},
  "vmSummary" : "ReadOnly portion of Libraries: Total=2.0G resident=0K(0%) swapped_out_or_unallocated=2.0G(100%)\nWritable regions: Total=1.7G written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=1.7G(100%)\n\n                                VIRTUAL   REGION \nREGION TYPE                        SIZE    COUNT (non-coalesced) \n===========                     =======  ======= \nAccelerate framework               128K        1 \nActivity Tracing                   256K        1 \nCG backing stores                 2688K        4 \nCG image                            72K        5 \nColorSync                          228K       26 \nCoreAnimation                      144K        8 \nCoreGraphics                        16K        3 \nCoreUI image data                  972K        7 \nFoundation                          16K        1 \nIOKit                             15.5M        2 \nKernel Alloc Once                    8K        1 \nMALLOC                           778.6M      829 \nMALLOC guard page                   32K        8 \nMALLOC_LARGE (reserved)            904K        4         reserved VM address space (unallocated)\nMALLOC_NANO (reserved)           384.0M        1         reserved VM address space (unallocated)\nOpenGL GLSL                        384K        5 \nSTACK GUARD                        184K       46 \nStack                             54.4M       46 \nVM_ALLOCATE                      101.1M      415 \nVM_ALLOCATE (reserved)           384.0M        4         reserved VM address space (unallocated)\n__DATA                            67.9M      719 \n__DATA_CONST                      13.3M      169 \n__DATA_DIRTY                       541K       99 \n__FONT_DATA                          4K        1 \n__GLSLBUILTINS                    5176K        1 \n__LINKEDIT                         1.0G      339 \n__OBJC_RO                         81.8M        1 \n__OBJC_RW                         3136K        2 \n__TEXT                           985.3M      633 \n__UNICODE                          588K        1 \ndyld private memory               1280K        3 \nmapped file                       89.8M       24 \nshared memory                      780K       18 \n===========                     =======  ======= \nTOTAL                              3.9G     3427 \nTOTAL, minus reserved VM space     3.1G     3427 \n",
  "legacyInfo" : {
  "threadTriggered" : {
    "queue" : "com.apple.main-thread"
  }
},
  "trialInfo" : {
  "rollouts" : [
    {
      "rolloutId" : "607844aa04477260f58a8077",
      "factorPackIds" : {
        "SIRI_MORPHUN_ASSETS" : "6103050cbfe6dc472e1c982a"
      },
      "deploymentId" : 240000066
    },
    {
      "rolloutId" : "60da5e84ab0ca017dace9abf",
      "factorPackIds" : {

      },
      "deploymentId" : 240000008
    },
    {
      "rolloutId" : "602ad4dac86151000cf27e46",
      "factorPackIds" : {
        "SIRI_DICTATION_ASSETS" : "6193d03f2171a2330e561dfc"
      },
      "deploymentId" : 240000290
    },
    {
      "rolloutId" : "5fc94383418129005b4e9ae0",
      "factorPackIds" : {

      },
      "deploymentId" : 240000207
    },
    {
      "rolloutId" : "601d9415f79519000ccd4b69",
      "factorPackIds" : {
        "SIRI_TEXT_TO_SPEECH" : "61c11dcd2cb6041dc630dc63"
      },
      "deploymentId" : 240000362
    },
    {
      "rolloutId" : "5ffde50ce2aacd000d47a95f",
      "factorPackIds" : {

      },
      "deploymentId" : 240000094
    }
  ],
  "experiments" : [

  ]
}
}

Model: MacBookPro16,1, BootROM 1715.60.5.0.0 (iBridge: 19.16.10647.0.0,0), 8 processors, 8-Core Intel Core i9, 2,3 GHz, 16 GB, SMC 
Graphics: Intel UHD Graphics 630, Intel UHD Graphics 630, Built-In
Display: Color LCD, 3072 x 1920 Retina, Main, MirrorOff, Online
Graphics: AMD Radeon Pro 5500M, AMD Radeon Pro 5500M, PCIe, 4 GB
Memory Module: BANK 0/ChannelA-DIMM0, 8 GB, DDR4, 2667 MHz, SK Hynix, HMA81GS6CJR8N-VK
Memory Module: BANK 2/ChannelB-DIMM0, 8 GB, DDR4, 2667 MHz, SK Hynix, HMA81GS6CJR8N-VK
AirPort: spairport_wireless_card_type_wifi (0x14E4, 0x7BF), wl0: Jul 12 2021 18:02:56 version 9.30.464.0.32.5.76 FWID 01-c081cfed
Bluetooth: Version (null), 0 services, 0 devices, 0 incoming serial ports
Network Service: Wi-Fi, AirPort, en0
USB Device: USB31Bus
USB Device: T2Bus
USB Device: composite_device
USB Device: Touch Bar Backlight
USB Device: Touch Bar Display
USB Device: Apple Internal Keyboard / Trackpad
USB Device: Headset
USB Device: Ambient Light Sensor
USB Device: FaceTime HD Camera (Built-in)
USB Device: Apple T2 Controller
Thunderbolt Bus: MacBook Pro, Apple Inc., 63.5
Thunderbolt Bus: MacBook Pro, Apple Inc., 63.5

Expected behavior

It should be able to run for as many meshes and as many time steps as required. Seeing as the memory use in constant, I do not see why it should fail at any point.

Open3D, Python and System information

- Operating system: (e.g. OSX 10.15, Ubuntu 18.04, Windows 10 64-bit)
macOS 12.1 (21C52) and Ubuntu 18.04
- Python version: (e.g. Python 3.8 / output from `import sys  print(sys.version)`)
3.7, 3.8, 3.9
- Open3D version: (output from python: `print(open3d.__version__)`)
0.14.1
- System type: (x84 / arm64 / apple-silicon / jetson / rpi)
x86
- Is this remote workstation?: yes or no
no
- How did you install Open3D?: (e.g. pip, conda, build from source)
pip
- Compiler version (if built from source): (e.g. gcc 7.5, clang 7.0)

Additional information

No response

SamTov avatar Jan 20 '22 23:01 SamTov

We fixed a couple of memory leak issues after 0.14.1. I want to confirm, have you tried the latest development version of Open3D available here?

errissa avatar Jan 20 '22 23:01 errissa

Yep, it still failed using '0.14.1+19635ef' on MacOS with Python 3.8

SamTov avatar Jan 21 '22 10:01 SamTov

The code snippet is high level and hard to track the issue. Could you please dig into your code base and see what might be the minimal reproducible example that is directly related to Open3D?

theNded avatar Jan 21 '22 13:01 theNded

Yes I can do this, sorry I did not provide it previously. I have printed below how Open3D is used in the visualizer.

Here is the code for updating of a particle state, i.e. position and rotation

def _draw_particles(self, visualizer=None, initial: bool = False):
        """
        Draw the particles on the visualizer.
        Parameters
        ----------
        initial : bool (default = True)
                If true, no particles are removed.
        Returns
        -------
        updates the information in the visualizer.
        Notes
        -----
        TODO: Use of initial is a dirty fix. It can be removed when support for
              transforming multiple geometry objects is added to open3d.
        """
        if visualizer is None:
            visualizer = self.vis
        if initial:
            for item in self.particles:
                for particle in item.mesh_dict:
                    visualizer.add_geometry(particle, item.mesh_dict[particle])
        else:
            for item in self.particles:
                for particle in item.mesh_dict:
                    visualizer.remove_geometry(particle)
                    visualizer.add_geometry(particle, item.mesh_dict[particle])

The mesh dict in this case is a dictionary full of mesh objects, e.g. spheres. These are Open3D TriangleMesh objects.

The following method updates the particles in the visualizer window:

def _update_particles(self, visualizer=None, step: int = None):
        """
        Update the positions of the particles.
        Parameters
        ----------
        step : int
                Step to update to.
        Returns
        -------
        Updates the positions of the particles in the box.
        """
        if visualizer is None:
            visualizer = self.vis
        if step is None:
            if self.counter == self.number_of_steps - 1:
                self.counter = 0
            else:
                self.counter += 1
            step = self.counter
        for particle in self.particles:
            particle.update_position_data(step)

        self._draw_particles(visualizer=visualizer)  # draw the particles.
        visualizer.post_redraw()  # re-draw the window.

Finally, the following two methods (one is just a thread opening linked to a button) uses a callback to run the simulation is the visualizer window:

def _continuous_trajectory(self, vis):
        """
        Button command for running the simulation in the visualizer.
        Parameters
        ----------
        vis : visualizer
                Object passed during the callback.
        """
        threading.Thread(target=self._run_trajectory).start()

    def _run_trajectory(self):
        """
        Callback method for running the trajectory smoothly.
        Returns
        -------
        Runs through the trajectory.
        """
        while self.counter < self.number_of_steps:
            time.sleep(1 / self.frame_rate)
            o3d.visualization.gui.Application.instance.post_to_main_thread(
                self.vis, self._update_particles
            )
            if self.interrupt == 1:
                break

        self.interrupt = 0

In terms of updating the position and rotation data of a mesh it is done in the following code:

def update_position_data(self, step: int):
        """
        Update the positions of each particle.
        Parameters
        ----------
        step : int
                Step to update to.
        Returns
        -------
        Updates the position of the mesh in the mesh_dict
        Notes
        -----
        TODO: Allow for no position data.
        """
        for i, item in enumerate(self.mesh_dict):
            self.mesh_dict[item].translate(self.position[step][i], relative=False)
            if self.director is not None:
                if step == 0:
                    current = self.director[-1][i]
                    matrix = rotation_matrix(current, self.director[step][i])
                    self.mesh_dict[item].rotate(matrix)
                else:
                    current = self.director[step - 1][i]
                    matrix = rotation_matrix(current, self.director[step][i])
                    self.mesh_dict[item].rotate(matrix)

Please let me know if I can provide more information about any part of the code.

SamTov avatar Jan 21 '22 14:01 SamTov

@theNded and @errissa have you have any ideas on this recently?

SamTov avatar Feb 04 '22 13:02 SamTov

I encountered the same problem as you. Finally, I found that the system would crash after calling the function post_redraw many times. The post_redraw function may have some vulnerabilities. It may crash or not respond to mouse operations after multiple calls.

LuckyLee2022 avatar Jun 21 '22 09:06 LuckyLee2022

I encountered the same problem as you. Finally, I found that the system would crash after calling the function post_redraw many times. The post_redraw function may have some vulnerabilities. It may crash or not respond to mouse operations after multiple calls.

I am having similar issues with the post_redraw, is there anything on the issue or how to bypass the crashes? currently trying to debug and find the issue with gdb.

The code here will recreate the crashes: https://github.com/isl-org/Open3D/issues/5243

MeyerBuaharon avatar Jun 23 '22 13:06 MeyerBuaharon

Unfortunately, I have not solved this problem. Two weeks have been delayed on this issue, resulting in the inability to realize animation. The asynchronous method can be used to refresh in a short time by using the python threading.Thread, and problems will occur again after several times. Or use the regular visualizer as open3d.visualization.Visualizer, as you reply in the issue 5245.

LuckyLee2022 avatar Jun 24 '22 00:06 LuckyLee2022

Hey, after how many calls does it crash approximately? The code I liked above can be called thousands of times until It crashes currently. The repo is https://github.com/zincware/ZnVis

SamTov avatar Jun 24 '22 09:06 SamTov

open3d 0.16 is out and it seems like the crashes stopped accuring for me. try and check it out if it solves this issue @SamTov

MeyerBuaharon avatar Nov 06 '22 15:11 MeyerBuaharon

I still get a segmentation fault with the new version. The issue persists when I use a smaller number of spheres for a larger number of updates or more spheres for less updates so I guess it is really the post_redraw operation being called many times still.

SamTov avatar Nov 07 '22 09:11 SamTov

I can confirm that the issue still persists. It crashes on exactly the 65520th call to add geometry. Basic example code to reproduce the crash:

import open3d as o3d
import open3d.visualization.gui as gui
import open3d.visualization.rendering as rendering
import time
import threading

class Window3D:

	def __init__(self):
		self.window = gui.Application.instance.create_window("Test", 1800, 700)
		self.window.set_on_close(self._on_close)

		self.widget3d = gui.SceneWidget()
		self.widget3d.scene = rendering.Open3DScene(self.window.renderer)
		self.window.add_child(self.widget3d)

		self.lit = rendering.MaterialRecord()
		self.lit.shader = "defaultUnlit"

		
		self.elems = []
		for i in range(20):
			wall = o3d.geometry.TriangleMesh.create_box(width=0.01, height=0.01, depth=0.01, create_uv_map=False, map_texture_to_each_face=False)
			wall.translate([i*0.1, 0, 0])
			self.elems.append(wall)
			self.widget3d.scene.add_geometry("wall" + str(i), wall, self.lit)
		bounds = self.widget3d.scene.bounding_box
		self.widget3d.setup_camera(60.0, bounds, bounds.get_center())
		
		self.counter = 0;

		self.is_done = False
		threading.Thread(target=self._update_thread).start()	


	def _on_close(self):
		self.is_done = True
		return True  
		
	def _update_thread(self):
		
		while not self.is_done:
			time.sleep(0.01)

			def update():
				self.counter +=1
				print(self.counter)
				print("remove_geometry")
				for i in range(len(self.elems)):
					# print("remove_geometry")
					self.widget3d.scene.remove_geometry('wall' + str(i))

				print("add_geometry")
				for i in range(len(self.elems)):
					self.widget3d.scene.add_geometry('wall' + str(i), self.elems[i], self.lit)
				   
			if not self.is_done:
				gui.Application.instance.post_to_main_thread(self.window, update)	
		

def main():
	app = o3d.visualization.gui.Application.instance
	app.initialize()
	win = Window3D()
	app.run()


if __name__ == "__main__":
	main()

jen-co avatar Nov 07 '22 10:11 jen-co

Having the same issue as @jen-co . I am trying to implement SLAM pipeline with 1.16.0 and drawing camera frustum (line sets) at video frame rate but always fails with this segmentation fault at some point.

#5243 #4992 #4881 are also similar issues. Did anyone manage to solve this?

muskie82 avatar Nov 14 '22 16:11 muskie82

@muskie82 I don' think there is a simple solution to this particular bug on the Python level as it is usually a Segmentation fault. This would suggest one needs to look into the c++ side to resolve it.

SamTov avatar Nov 21 '22 07:11 SamTov

Same issue here. Interesting thing is that my application running in an event loop by iteratively calling instance.run_one_tick() crashes as well with a segfault after a certain amount of calls. post_redraw() is not actively called. Maybe run_one_tick() calls this function or parts of it as well.

SensorTom avatar Jan 29 '23 10:01 SensorTom

did the issue got fixed in 0.17?

MeyerBuaharon avatar Apr 02 '23 11:04 MeyerBuaharon

@MeyerBuaharonAssuming the 0.17 version is on PyPi not from my side. I have however moved my software over to combining all meshes into one large mesh which means I delay this fault for many time steps which is a temporary workable solution.

SamTov avatar Apr 03 '23 09:04 SamTov

I'm using the v0.17 release in C++ and I seem to be having the same problem. Does anyone know if the issue is fixed in the current latest commit on the master branch?

xorespesp avatar Sep 12 '23 04:09 xorespesp

+1, similar issue here with the latest 0.17 release too.

suddhu avatar Nov 19 '23 21:11 suddhu

The issue persists in v0.18

canoneod avatar Feb 07 '24 14:02 canoneod