mesmerize-core icon indicating copy to clipboard operation
mesmerize-core copied to clipboard

Windows CI runs are segfaulting

Open ethanbb opened this issue 3 months ago • 0 comments

Not sure exactly what's going on but it seems to happen when tensorflow is being initialized. So maybe easier to just wait until tensorflow is no longer part of caiman, which is getting pretty close

Stack trace

Testing cnmf cnmf input_movie_path: 417d0c0b-f2bb-41df-8ded-ce51b96ee45e\417d0c0b-f2bb-41df-8ded-ce51b96ee45e-mcorr_els__d1_60_d2_80_d3_1_order_F_frames_2000.mmap Running a3d6d1f0-b0d6-404c-a3fc-9adca9e0e346 with local backend


Starting CNMF item: algo cnmf item_name test-cnmf input_movie_path 417d0c0b-f2bb-41df-8ded-ce51b96ee45e\417d0c0b-... params {'main': {'p': 2, 'nb': 1, 'rf': None, 'stride... outputs None added_time 2025-09-05T01:39:11 ran_time None algo_duration None comments None uuid a3d6d1f0-b0d6-404c-a3fc-9adca9e0e346 Name: 1, dtype: object With params:{'main': {'p': 2, 'nb': 1, 'rf': None, 'stride': 30, 'K': 10, 'gSig': [5, 5], 'ssub': 1, 'tsub': 1, 'method_init': 'greedy_roi', 'min_SNR': 2.5, 'rval_thr': 0.8, 'use_cnn': True, 'min_cnn_thr': 0.8, 'cnn_lowest': 0.1, 'decay_time': 1}, 'refit': True} making memmap C:\Miniconda\envs\mescore\lib\site-packages\caiman_init_.py:3: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. import pkg_resources C:\Miniconda\envs\mescore\lib\site-packages\caiman_init_.py:3: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. import pkg_resources C:\Miniconda\envs\mescore\lib\site-packages\caiman_init_.py:3: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. import pkg_resources C:\Miniconda\envs\mescore\lib\site-packages\caiman\source_extraction\cnmf\deconvolution.py:1004: FutureWarning: Beginning in SciPy 1.17, multidimensional input will be treated as a batch, not raveled. To preserve the existing behavior and silence this warning, ravel arguments before passing them to toeplitz. A = scipy.linalg.toeplitz(xc[lags + np.arange(lags)], C:\Miniconda\envs\mescore\lib\site-packages\caiman\source_extraction\cnmf\deconvolution.py:1004: FutureWarning: Beginning in SciPy 1.17, multidimensional input will be treated as a batch, not raveled. To preserve the existing behavior and silence this warning, ravel arguments before passing them to toeplitz. A = scipy.linalg.toeplitz(xc[lags + np.arange(lags)], C:\Miniconda\envs\mescore\lib\site-packages\caiman\source_extraction\cnmf\deconvolution.py:1004: FutureWarning: Beginning in SciPy 1.17, multidimensional input will be treated as a batch, not raveled. To preserve the existing behavior and silence this warning, ravel arguments before passing them to toeplitz. A = scipy.linalg.toeplitz(xc[lags + np.arange(lags)], 2025-09-05 01:39:27.018030: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. Windows fatal exception: access violation

Thread 0x00000a00 (most recent call first): File "C:\Miniconda\envs\mescore\lib\multiprocessing\connection.py", line 310 in _recv_bytes File "C:\Miniconda\envs\mescore\lib\multiprocessing\connection.py", line 255 in recv File "C:\Miniconda\envs\mescore\lib\multiprocessing\pool.py", line 579 in _handle_results File "C:\Miniconda\envs\mescore\lib\threading.py", line 953 in run File "C:\Miniconda\envs\mescore\lib\threading.py", line 1016 in _bootstrap_inner File "C:\Miniconda\envs\mescore\lib\threading.py", line 973 in _bootstrap

Thread 0x00001764 (most recent call first): File "C:\Miniconda\envs\mescore\lib\multiprocessing\pool.py", line 531 in _handle_tasks File "C:\Miniconda\envs\mescore\lib\threading.py", line 953 in run File "C:\Miniconda\envs\mescore\lib\threading.py", line 1016 in _bootstrap_inner File "C:\Miniconda\envs\mescore\lib\threading.py", line 973 in _bootstrap

Thread 0x0000105c (most recent call first): File "C:\Miniconda\envs\mescore\lib\multiprocessing\connection.py", line 816 in _exhaustive_wait File "C:\Miniconda\envs\mescore\lib\multiprocessing\connection.py", line 884 in wait File "C:\Miniconda\envs\mescore\lib\multiprocessing\pool.py", line 502 in _wait_for_updates File "C:\Miniconda\envs\mescore\lib\multiprocessing\pool.py", line 522 in _handle_workers File "C:\Miniconda\envs\mescore\lib\threading.py", line 953 in run File "C:\Miniconda\envs\mescore\lib\threading.py", line 1016 in _bootstrap_inner File "C:\Miniconda\envs\mescore\lib\threading.py", line 973 in _bootstrap

Thread 0x00001064 (most recent call first): File "C:\Miniconda\envs\mescore\lib\threading.py", line 324 in wait File "C:\Miniconda\envs\mescore\lib\threading.py", line 607 in wait File "C:\Miniconda\envs\mescore\lib\site-packages\tqdm_monitor.py", line 60 in run File "C:\Miniconda\envs\mescore\lib\threading.py", line 1016 in _bootstrap_inner File "C:\Miniconda\envs\mescore\lib\threading.py", line 973 in _bootstrap

Current thread 0x00001dd4 (most recent call first): File "C:\Miniconda\envs\mescore\lib\site-packages\tensorflow\python\eager\context.py", line 622 in ensure_initialized File "C:\Miniconda\envs\mescore\lib\site-packages\tensorflow\python\framework\constant_op.py", line 101 in convert_to_eager_tensor File "C:\Miniconda\envs\mescore\lib\site-packages\tensorflow\python\framework\constant_op.py", line 304 in _constant_eager_impl File "C:\Miniconda\envs\mescore\lib\site-packages\tensorflow\python\framework\constant_op.py", line 279 in _constant_impl File "C:\Miniconda\envs\mescore\lib\site-packages\tensorflow\python\framework\constant_op.py", line 267 in constant File "C:\Miniconda\envs\mescore\lib\site-packages\tensorflow\python\framework\tensor_conversion_registry.py", line 48 in _default_conversion_function File "C:\Miniconda\envs\mescore\lib\site-packages\tensorflow\python\framework\ops.py", line 1638 in convert_to_tensor File "C:\Miniconda\envs\mescore\lib\site-packages\tensorflow\python\profiler\trace.py", line 183 in wrapped File "C:\Miniconda\envs\mescore\lib\site-packages\tensorflow\python\ops\resource_variable_ops.py", line 1817 in _init_from_args File "C:\Miniconda\envs\mescore\lib\site-packages\tensorflow\python\ops\resource_variable_ops.py", line 1659 in init File "C:\Miniconda\envs\mescore\lib\site-packages\tensorflow\python\ops\variables.py", line 268 in call File "C:\Miniconda\envs\mescore\lib\site-packages\tensorflow\python\util\traceback_utils.py", line 150 in error_handler File "C:\Miniconda\envs\mescore\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 2755 in default_variable_creator_v2 File "C:\Miniconda\envs\mescore\lib\site-packages\tensorflow\python\ops\variables.py", line 240 in File "C:\Miniconda\envs\mescore\lib\site-packages\tensorflow\python\ops\variables.py", line 247 in _variable_v2_call File "C:\Miniconda\envs\mescore\lib\site-packages\tensorflow\python\ops\variables.py", line 266 in call File "C:\Miniconda\envs\mescore\lib\site-packages\tensorflow\python\util\traceback_utils.py", line 150 in error_handler File "C:\Miniconda\envs\mescore\lib\site-packages\keras\engine\training.py", line 334 in _init_batch_counters File "C:\Miniconda\envs\mescore\lib\site-packages\tensorflow\python\trackable\base.py", line 205 in _method_wrapper File "C:\Miniconda\envs\mescore\lib\site-packages\keras\engine\training.py", line 319 in init File "C:\Miniconda\envs\mescore\lib\site-packages\keras\utils\traceback_utils.py", line 65 in error_handler File "C:\Miniconda\envs\mescore\lib\site-packages\tensorflow\python\trackable\base.py", line 205 in _method_wrapper File "C:\Miniconda\envs\mescore\lib\site-packages\keras\engine\sequential.py", line 114 in init File "C:\Miniconda\envs\mescore\lib\site-packages\keras\utils\traceback_utils.py", line 65 in error_handler File "C:\Miniconda\envs\mescore\lib\site-packages\tensorflow\python\trackable\base.py", line 205 in _method_wrapper File "C:\Miniconda\envs\mescore\lib\site-packages\keras\engine\sequential.py", line 470 in from_config File "C:\Miniconda\envs\mescore\lib\site-packages\keras\utils\generic_utils.py", line 734 in deserialize_keras_object File "C:\Miniconda\envs\mescore\lib\site-packages\keras\layers\serialization.py", line 249 in deserialize File "C:\Miniconda\envs\mescore\lib\site-packages\keras\layers\serialization.py", line 272 in deserialize_from_json File "C:\Miniconda\envs\mescore\lib\site-packages\keras\saving\model_config.py", line 109 in model_from_json File "C:\Miniconda\envs\mescore\lib\site-packages\caiman\components_evaluation.py", line 299 in evaluate_components_CNN File "C:\Miniconda\envs\mescore\lib\site-packages\caiman\components_evaluation.py", line 658 in select_components_from_metrics File "C:\Miniconda\envs\mescore\lib\site-packages\caiman\components_evaluation.py", line 613 in estimate_components_quality_auto File "C:\Miniconda\envs\mescore\lib\site-packages\caiman\source_extraction\cnmf\estimates.py", line 1046 in evaluate_components File "Z:\mesmerize_core\algorithms\cnmf.py", line 109 in run_algo File "Z:\mesmerize_core\caiman_extensions\common.py", line 485 in _run_local File "Z:\mesmerize_core\caiman_extensions\common.py", line 585 in run File "Z:\mesmerize_core\caiman_extensions\cache.py", line 214 in __invalidate File "Z:\tests\test_core.py", line 452 in test_cnmf File "C:\Miniconda\envs\mescore\lib\site-packages_pytest\python.py", line 157 in pytest_pyfunc_call File "C:\Miniconda\envs\mescore\lib\site-packages\pluggy_callers.py", line 121 in _multicall File "C:\Miniconda\envs\mescore\lib\site-packages\pluggy_manager.py", line 120 in _hookexec File "C:\Miniconda\envs\mescore\lib\site-packages\pluggy_hooks.py", line 512 in call File "C:\Miniconda\envs\mescore\lib\site-packages_pytest\python.py", line 1671 in runtest File "C:\Miniconda\envs\mescore\lib\site-packages_pytest\runner.py", line 178 in pytest_runtest_call File "C:\Miniconda\envs\mescore\lib\site-packages\pluggy_callers.py", line 121 in _multicall File "C:\Miniconda\envs\mescore\lib\site-packages\pluggy_manager.py", line 120 in _hookexec File "C:\Miniconda\envs\mescore\lib\site-packages\pluggy_hooks.py", line 512 in call File "C:\Miniconda\envs\mescore\lib\site-packages_pytest\runner.py", line 246 in File "C:\Miniconda\envs\mescore\lib\site-packages_pytest\runner.py", line 344 in from_call File "C:\Miniconda\envs\mescore\lib\site-packages_pytest\runner.py", line 245 in call_and_report File "C:\Miniconda\envs\mescore\lib\site-packages_pytest\runner.py", line 136 in runtestprotocol File "C:\Miniconda\envs\mescore\lib\site-packages_pytest\runner.py", line 117 in pytest_runtest_protocol File "C:\Miniconda\envs\mescore\lib\site-packages\pluggy_callers.py", line 121 in _multicall File "C:\Miniconda\envs\mescore\lib\site-packages\pluggy_manager.py", line 120 in _hookexec File "C:\Miniconda\envs\mescore\lib\site-packages\pluggy_hooks.py", line 512 in call File "C:\Miniconda\envs\mescore\lib\site-packages_pytest\main.py", line 367 in pytest_runtestloop File "C:\Miniconda\envs\mescore\lib\site-packages\pluggy_callers.py", line 121 in _multicall File "C:\Miniconda\envs\mescore\lib\site-packages\pluggy_manager.py", line 120 in _hookexec File "C:\Miniconda\envs\mescore\lib\site-packages\pluggy_hooks.py", line 512 in call File "C:\Miniconda\envs\mescore\lib\site-packages_pytest\main.py", line 343 in _main File "C:\Miniconda\envs\mescore\lib\site-packages_pytest\main.py", line 289 in wrap_session File "C:\Miniconda\envs\mescore\lib\site-packages_pytest\main.py", line 336 in pytest_cmdline_main File "C:\Miniconda\envs\mescore\lib\site-packages\pluggy_callers.py", line 121 in multicall File "C:\Miniconda\envs\mescore\lib\site-packages\pluggy_manager.py", line 120 in hookexec File "C:\Miniconda\envs\mescore\lib\site-packages\pluggy_hooks.py", line 512 in call File "C:\Miniconda\envs\mescore\lib\site-packages_pytest\config_init.py", line 175 in main File "C:\Miniconda\envs\mescore\lib\site-packages_pytest\config_init.py", line 201 in console_main File "C:\Miniconda\envs\mescore\Scripts\pytest-script.py", line 10 in D:\a_temp\b0813710-eb9d-4c39-8826-4708cda59f9a.sh: line 8: 2063 Segmentation fault DOWNLOAD_GROUND_TRUTHS=1 pytest -s .

ethanbb avatar Sep 06 '25 04:09 ethanbb