pytest-xdist
pytest-xdist copied to clipboard
Published
20 hours ago •
pytest-dev
pytest-dev
Index error in handling of crashed worker
I am using pytest-xdist to test code which uses compiled modules. Occasionally these modules fail with errors such as seg faults. Previously this would lead to an E in the output (instead of . or F), however it now leads to the following error:
Fatal Python error: Floating point exception
Thread 0x0000700001cbf000 (most recent call first):
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/execnet/gateway_base.py", line 400 in read
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/execnet/gateway_base.py", line 432 in from_io
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/execnet/gateway_base.py", line 967 in _thread_receiver
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/execnet/gateway_base.py", line 220 in run
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/execnet/gateway_base.py", line 285 in _perform_spawn
Current thread 0x0000000110d51600 (most recent call first):
File "/Users/runner/work/pyccel/pyccel/tests/epyccel/recognised_functions/test_numpy_funcs.py", line 1045 in test_shape_indexed
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/_pytest/python.py", line 1799 in runtest
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/_pytest/runner.py", line 262 in <lambda>
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/_pytest/runner.py", line 341 in from_call
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/_pytest/runner.py", line 222 in call_and_report
....................................[gw2] node down: Not properly terminated
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/_pytest/runner.py", line 133 in runtestprotocol
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/xdist/remote.py", line 174 in run_one_test
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/xdist/remote.py", line 157 in pytest_runtestloop
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/_pytest/main.py", line 323 in _main
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/_pytest/main.py", line 269 in wrap_session
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/xdist/remote.py", line 355 in <module>
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/execnet/gateway_base.py", line 1084 in executetask
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/execnet/gateway_base.py", line 220 in run
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/execnet/gateway_base.py", line 285 in _perform_spawn
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/execnet/gateway_base.py", line 267 in integrate_as_primary_thread
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/execnet/gateway_base.py", line 1060 in serve
File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/execnet/gateway_base.py", line 1554 in serve
File "<string>", line 8 in <module>
File "<string>", line 1 in <module>
Extension modules: mpi4py.MPI, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg._cythonized_array_utils, scipy.linalg._flinalg, scipy.linalg._solve_toeplitz, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_lapack, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._isolve._iterative, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, mod_8lxxeigkm7y5_8lxxeigkm7y5, mod_n352e0p4apjy_n352e0p4apjy, mod_mct957s9nps2_mct957s9nps2, mod_z8spg507gp6l_z8spg507gp6l, mod_39ea2o56eaqr_39ea2o56eaqr, mod_adahzxncq555_adahzxncq555, mod_znm50px88q3x_znm50px88q3x, mod_h2iotszh1aep_h2iotszh1aep, mod_jwppijyysou6_jwppijyysou6, mod_a4kho2a4mmix_a4kho2a4mmix, mod_yl6hkmo5iab5_yl6hkmo5iab5, mod_a3o251n61g2t_a3o251n61g2t, mod_f01gdwlvxbge_f01gdwlvxbge, mod_4fd4p3j1mzv1_4fd4p3j1mzv1, mod_s71kwcw8y37w_s71kwcw8y37w, mod_kgqrxd93r5b7_kgqrxd93r5b7, mod_xklfteujj0bo_xklfteujj0bo, mod_is4h9nry1e4p_is4h9nry1e4p, mod_ceb3p9ygo0fe_ceb3p9ygo0fe, mod_nnpc9w025fa1_nnpc9w025fa1, mod_apfakoy0s13w_apfakoy0s13w, mod_kvaa7hvepgrr_kvaa7hvepgrr, mod_uxgn5rc9c31d_uxgn5rc9c31d, mod_ncrtht6pnuq6_ncrtht6pnuq6, numpy.linalg.lapack_lite, mod_wgyea5rit7zo_wgyea5rit7zo, mod_9que8e2p4s83_9que8e2p4s83, mod_ebr1mkfyxteu_ebr1mkfyxteu, mod_79r2y0251q4m_79r2y0251q4m, mod_fm5nbktv8wmc_fm5nbktv8wmc, mod_gpboc4k6zsq3_gpboc4k6zsq3, mod_cbq8husuk598_cbq8husuk598, mod_weh0qrn4gd6v_weh0qrn4gd6v, mod_z68kj5so2fi4_z68kj5so2fi4, mod_bm1ll04q5eap_bm1ll04q5eap, mod_y06k15d5jube_y06k15d5jube, mod_wy0y5sdzsbdg_wy0y5sdzsbdg, mod_e28467cj9pcl_e28467cj9pcl, mod_e19pv2yo6kas_e19pv2yo6kas, mod_5lpsi7fo7pzp_5lpsi7fo7pzp, mod_n72hcyyhkntj_n72hcyyhkntj, mod_zn8l5t7xwh3r_zn8l5t7xwh3r, mod_isy762tgrqt9_isy762tgrqt9, mod_b8gu2kpj3xfk_b8gu2kpj3xfk, mod_jzdqk2v9tkhd_jzdqk2v9tkhd, mod_c75qqaea1t1j_c75qqaea1t1j, mod_1b6yn77od5mw_1b6yn77od5mw, mod_8uxoautcaieh_8uxoautcaieh, mod_j741icfrx8f5_j741icfrx8f5, mod_5s6eag951r48_5s6eag951r48, mod_zvs4ejxykij7_zvs4ejxykij7, mod_h4denrhvov1x_h4denrhvov1x, mod_q27q22f6w45i_q27q22f6w45i, mod_oc2o0yhibdog_oc2o0yhibdog, mod_ixsbpntrllln_ixsbpntrllln, mod_q8rvns0lm4yt_q8rvns0lm4yt, mod_8dh8181ohjsy_8dh8181ohjsy, mod_2uvlwmhmce9i_2uvlwmhmce9i, mod_wpajo7hnr07d_wpajo7hnr07d, mod_3492awgwqava_3492awgwqava, mod_c2p8w2zfc9h8_c2p8w2zfc9h8, mod_40c3v3tyes6n_40c3v3tyes6n, mod_a0uj3gygj6u6_a0uj3gygj6u6, mod_dzlgkww5yadg_dzlgkww5yadg, mod_drmyzpkr52t3_drmyzpkr52t3, mod_o6m6m3enstax_o6m6m3enstax, mod_mghozhazuz6f_mghozhazuz6f, mod_fk96hb8916f2_fk96hb8916f2, mod_5pa6hrk7w0wp_5pa6hrk7w0wp, mod_xew79xtilinm_xew79xtilinm, mod_9yj8m1i6ng8z_9yj8m1i6ng8z, mod_dh4crf5dnyeq_dh4crf5dnyeq, mod_35hdwkvxp9a2_35hdwkvxp9a2, mod_ehsy78cfs3oi_ehsy78cfs3oi, mod_fyw3it1d4362_fyw3it1d4362, mod_pvr4kwsc4x9e_pvr4kwsc4x9e, mod_xxsd2d2m1f2c_xxsd2d2m1f2c, mod_0ehuvze7loxl_0ehuvze7loxl, mod_cprw42xol5en_cprw42xol5en, mod_5n51bong7kwz_5n51bong7kwz, mod_3vhvhlffcr2c_3vhvhlffcr2c, mod_pdwj6zcj8fg9_pdwj6zcj8fg9, mod_puoc7ec2mpsb_puoc7ec2mpsb, mod_qetwy248ex61_qetwy248ex61, mod_llcrv96h71uq_llcrv96h71uq, mod_fia9kgo70q7f_fia9kgo70q7f, mod_astwy78gene0_astwy78gene0, mod_6nglr1owwsjl_6nglr1owwsjl, mod_lw17en312p2y_lw17en312p2y, mod_hdkzmkhjcrvb_hdkzmkhjcrvb, mod_6rfelx2suzx3_6rfelx2suzx3, mod_t0hcg185ob2x_t0hcg185ob2x, mod_7azci22ju71e_7azci22ju71e, mod_fhttlsfiy9k3_fhttlsfiy9k3, mod_bk6mu5yk1q81_bk6mu5yk1q81, mod_6ucxdwreitkt_6ucxdwreitkt, mod_h2j76h3kl0sb_h2j76h3kl0sb, mod_txnq5fh952jv_txnq5fh952jv, mod_qwxw3zbb7fth_qwxw3zbb7fth, mod_y6x6dmqqmt4r_y6x6dmqqmt4r, mod_5nn6irb3xqb7_5nn6irb3xqb7, mod_bkj6kfq1tz0w_bkj6kfq1tz0w, mod_e7h64nudfgz5_e7h64nudfgz5, mod_q4sj4ls2h8jc_q4sj4ls2h8jc, mod_8bxi9haxnt6l_8bxi9haxnt6l, mod_wwhjohe8mlbp_wwhjohe8mlbp, mod_04049kh50t98_04049kh50t98, mod_fdwfw419pvo4_fdwfw419pvo4, mod_ohhd7y1m5c2g_ohhd7y1m5c2g, mod_0fb0rlv8hukk_0fb0rlv8hukk, mod_ytg2m7k0awof_ytg2m7k0awof, mod_8wmn0grphoul_8wmn0grphoul, mod_5lpb5wvoel6g_5lpb5wvoel6g, mod_p4dtnuvye9il_p4dtnuvye9il, mod_6tnc1ugz72n4_6tnc1ugz72n4, mod_hnjj408bbvu9_hnjj408bbvu9, mod_jcu7a8otx6h8_jcu7a8otx6h8, mod_nd90sj16pycz_nd90sj16pycz, mod_737w372paus0_737w372paus0, mod_47jur3v4z2pt_47jur3v4z2pt, mod_9ki4okjuk28f_9ki4okjuk28f, mod_jwnq3iqvoqt7_jwnq3iqvoqt7, mod_1fn0t76dvpxx_1fn0t76dvpxx, mod_3phz71m2cbrj_3phz71m2cbrj, mod_nhys03n7p1v2_nhys03n7p1v2, mod_pwno7luf4itp_pwno7luf4itp, mod_7vbaad0sl5a4_7vbaad0sl5a4, mod_gqyrm4a4fl67_gqyrm4a4fl67, mod_m93caxn1cazh_m93caxn1cazh, mod_6aeuq9ifdqvs_6aeuq9ifdqvs, mod_cmhcqy6yucct_cmhcqy6yucct, mod_uvvvzin3qqmd_uvvvzin3qqmd, mod_wudeqao0fqge_wudeqao0fqge, mod_80rqian8vp4f_80rqian8vp4f, mod_8jeiniry9f00_8jeiniry9f00, mod_aturiqr1bp44_aturiqr1bp44, mod_lhb8h6oikl5t_lhb8h6oikl5t, generic_functions_xh78z41amzve_xh78z41amzve, mod_xihgbve9yvq8_xihgbve9yvq8, mod_ktrsztrgtwqs_ktrsztrgtwqs, mod_twbh3xmsogbu_twbh3xmsogbu, mod_gfybrx1mlqfy_gfybrx1mlqfy, mod_kkhrys76mtbu_kkhrys76mtbu, mod_m0c3luro6z71_m0c3luro6z71, mod_wj1fmt39hbst_wj1fmt39hbst, mod_teqpkq6me4sd_teqpkq6me4sd, mod_6k2kdpk3xcf9_6k2kdpk3xcf9, mod_bia8rabadfx5_bia8rabadfx5, mod_mffwzzl9y3fa_mffwzzl9y3fa, mod_tpdpxxt9ihdn_tpdpxxt9ihdn, mod_hmnnmaxezwb8_hmnnmaxezwb8, mod_gtgr38x98csc_gtgr38x98csc, mod_327vs1ram4pb_327vs1ram4pb, mod_it52q6vuul9r_it52q6vuul9r, mod_19thzjcf3ppk_19thzjcf3ppk, mod_2yia2og3g48y_2yia2og3g48y, mod_3c8kzhjt8d1v_3c8kzhjt8d1v, mod_a3msyjihd8mr_a3msyjihd8mr, mod_ea1mk3efrgxa_ea1mk3efrgxa, mod_ojiyeers2e40_ojiyeers2e40, mod_nzdgcb8czjyu_nzdgcb8czjyu, mod_5dtxubik1jfk_5dtxubik1jfk, mod_ah0wg00uklmp_ah0wg00uklmp, mod_wa7oe3qcnb4v_wa7oe3qcnb4v, highorder_functions_27plwt85fx9f_27plwt85fx9f, mod_h2ufjadtaoxp_h2ufjadtaoxp, mod_3d8yrlav6xjh_3d8yrlav6xjh, mod_6ca581yni4xw_6ca581yni4xw, mod_ma93wpc3opjr_ma93wpc3opjr, mod_kbtseebastt6_kbtseebastt6, mod_5lt4p7079aco_5lt4p7079aco, mod_q3lry7ooopt9_q3lry7ooopt9, mod_irs0urejoe2e_irs0urejoe2e, mod_wxgndsot4eb8_wxgndsot4eb8, mod_y64q10bfs4se_y64q10bfs4se, mod_dtshv0f6i1yr_dtshv0f6i1yr, mod_2728gijqsrxt_2728gijqsrxt, mod_sc69zbqb9j18_sc69zbqb9j18, mod_emquw1u2epn4_emquw1u2epn4, mod_1zewyyvoyz7t_1zewyyvoyz7t, mod_ujk2rqnh7fgh_ujk2rqnh7fgh, mod_9tbic59mbbia_9tbic59mbbia, mod_2l1jhzslkmhj_2l1jhzslkmhj, mod_p6zrcxiux638_p6zrcxiux638, mod_q77p21qvnuys_q77p21qvnuys, mod_0epc5x3uz4nq_0epc5x3uz4nq, mod_6vn923kb07p2_6vn923kb07p2, mod_5ggim0hmyngn_5ggim0hmyngn, mod_elbtcuql61ff_elbtcuql61ff, mod_ojzt1bnw5pdv_ojzt1bnw5pdv, mod_y1k2iw8ih8ma_y1k2iw8ih8ma, mod_3pwl07omuddm_3pwl07omuddm, mod_0rra00txrcoc_0rra00txrcoc, mod_5lpchg7td5nq_5lpchg7td5nq, mod_h1rdd3uzwhjr_h1rdd3uzwhjr, mod_xx084pavffoq_xx084pavffoq, mod_eo685ir20xff_eo685ir20xff, mod_b4hkmufhg8et_b4hkmufhg8et, mod_r6l7s6930y90_r6l7s6930y90, mod_d47dh4s8l4dl_d47dh4s8l4dl, mod_7wql7da5ynep_7wql7da5ynep, mod_2yftwsbhnz1h_2yftwsbhnz1h, mod_1nl8l0oqsrao_1nl8l0oqsrao, mod_6syfdnes6jq6_6syfdnes6jq6, mod_qc5smt5befx8_qc5smt5befx8, mod_07bp5sauejiy_07bp5sauejiy, mod_2ovkh01nzvi0_2ovkh01nzvi0, mod_8crkra2prshu_8crkra2prshu, mod_09uff1ckwsco_09uff1ckwsco, mod_vfvqk379pp3s_vfvqk379pp3s, mod_g75a39twnwfw_g75a39twnwfw, mod_0h7xfdbha6xo_0h7xfdbha6xo, mod_jssaw05dqemq_jssaw05dqemq, mod_0cb3ljrg7vck_0cb3ljrg7vck, mod_j4jejyyya19f_j4jejyyya19f, mod_cly8n3e8lo5l_cly8n3e8lo5l, mod_e9dyou96iq9k_e9dyou96iq9k, mod_8zz5d7u3i28n_8zz5d7u3i28n, mod_0majrypk4fxe_0majrypk4fxe, mod_7a65qramgily_7a65qramgily, mod_jt8oevy23ny5_jt8oevy23ny5, mod_7ionjz3nqnof_7ionjz3nqnof, mod_mmkgokvju4wc_mmkgokvju4wc, mod_2c5g8o1kj1yu_2c5g8o1kj1yu, mod_iy97fwcsnpmi_iy97fwcsnpmi, mod_i46z2aakn1p4_i46z2aakn1p4, mod_t8f1uf1vd3zj_t8f1uf1vd3zj, mod_lzr653pckcdw_lzr653pckcdw, mod_3305jp1d7uyc_3305jp1d7uyc, mod_f8mi700fhd2f_f8mi700fhd2f, mod_j9ykn84nd9q5_j9ykn84nd9q5, mod_qlfuat5s1ph2_qlfuat5s1ph2, mod_2ghwnccek02e_2ghwnccek02e, mod_2uwnikmd6vih_2uwnikmd6vih, mod_iscr30jckgjq_iscr30jckgjq, mod_sjxshm8hzfdj_sjxshm8hzfdj, mod_m0wmiz10zwff_m0wmiz10zwff, mod_zvx4sjroz31h_zvx4sjroz31h, mod_lk45p36igy8v_lk45p36igy8v, mod_wx8japqybway_wx8japqybway, mod_g5tknn62w9eg_g5tknn62w9eg, mod_wcivx7tyboq5_wcivx7tyboq5, mod_xlwuyhk8hnox_xlwuyhk8hnox, mod_se38wryoumo6_se38wryoumo6, mod_pyngjxoeyysn_pyngjxoeyysn, mod_78ijyid3he03_78ijyid3he03, mod_ztnqsdif18vb_ztnqsdif18vb, mod_lh14834i3bgw_lh14834i3bgw, mod_4uyvchcsma1h_4uyvchcsma1h, mod_if4xqvnw0qoz_if4xqvnw0qoz, mod_ast3j83rkpjm_ast3j83rkpjm, mod_otil0y7rs5uc_otil0y7rs5uc, mod_esw6ffkc8r9p_esw6ffkc8r9p, mod_tkgfckyh0iam_tkgfckyh0iam, mod_v7lbrvgah54w_v7lbrvgah54w, mod_wamb1o0b3vzr_wamb1o0b3vzr, mod_m104ckvfvtw1_m104ckvfvtw1, mod_142smfd1feys_142smfd1feys, mod_1rxcw76szppu_1rxcw76szppu, mod_nokp24gghl4x_nokp24gghl4x, mod_9ezv1rz348js_9ezv1rz348js, mod_0lx7sctc0wv5_0lx7sctc0wv5, mod_aofw5i1ualpz_aofw5i1ualpz, mod_numgac7xi8w7_numgac7xi8w7, mod_zkak2tn2mocb_zkak2tn2mocb, mod_ymkmvha9nuxd_ymkmvha9nuxd, mod_vb3pkpfk0b0r_vb3pkpfk0b0r, mod_utdudgmse8ou_utdudgmse8ou, mod_lyatq0tg47n7_lyatq0tg47n7, mod_26axrz18hfad_26axrz18hfad, mod_zw7fez0xkskv_zw7fez0xkskv, mod_3wzk5d3hjvy6_3wzk5d3hjvy6, mod_x5h19rdn8uq1_x5h19rdn8uq1, mod_va0tigtvme67_va0tigtvme67, mod_irg967zl18zo_irg967zl18zo, mod_wtt8d8zeykvz_wtt8d8zeykvz, mod_x1qvmxwxc58s_x1qvmxwxc58s, mod_qt38ipqenr9l_qt38ipqenr9l, mod_hfpu1c6nvn0g_hfpu1c6nvn0g, mod_d9lhynrphwsp_d9lhynrphwsp, mod_rwi1r1r5zswc_rwi1r1r5zswc, mod_seoylfyg0ofe_seoylfyg0ofe, mod_b5z2by3tm8gj_b5z2by3tm8gj, mod_291abik46k25_291abik46k25, mod_uj26igdjp1if_uj26igdjp1if, mod_28q5688w8fr1_28q5688w8fr1, mod_ge4u789eznf8_ge4u789eznf8, mod_higgtgc1yivx_higgtgc1yivx, mod_li5n30vl03f1_li5n30vl03f1, mod_e8pgtt2lzh67_e8pgtt2lzh67, mod_ev0g752e3yh9_ev0g752e3yh9, mod_uftz74oif0yt_uftz74oif0yt, mod_t7ix79rxfl0x_t7ix79rxfl0x, mod_h0jxa68ew2z5_h0jxa68ew2z5, mod_hxzqf89oilix_hxzqf89oilix, mod_5am0hz4yy7m8_5am0hz4yy7m8, mod_sr56t0xj1wdt_sr56t0xj1wdt, mod_4lrekhv8ym29_4lrekhv8ym29, mod_b7h2dip8606x_b7h2dip8606x, mod_tvfcb0ibjkgl_tvfcb0ibjkgl, mod_fylwtcbjv13m_fylwtcbjv13m, mod_ypakxoqovzqi_ypakxoqovzqi, mod_tqh7nd7fl01g_tqh7nd7fl01g (total: 341)
F
replacing crashed worker gw2
....INTERNALERROR> def worker_internal_error(self, node, formatted_error):
INTERNALERROR> """
INTERNALERROR> pytest_internalerror() was called on the worker.
INTERNALERROR>
INTERNALERROR> pytest_internalerror() arguments are an excinfo and an excrepr, which can't
INTERNALERROR> be serialized, so we go with a poor man's solution of raising an exception
INTERNALERROR> here ourselves using the formatted message.
INTERNALERROR> """
INTERNALERROR> self._active_nodes.remove(node)
INTERNALERROR> try:
INTERNALERROR> > assert False, formatted_error
INTERNALERROR> E AssertionError: Traceback (most recent call last):
INTERNALERROR> E File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/_pytest/main.py", line 269, in wrap_session
INTERNALERROR> E session.exitstatus = doit(config, session) or 0
INTERNALERROR> E File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/_pytest/main.py", line 323, in _main
INTERNALERROR> E config.hook.pytest_runtestloop(session=session)
INTERNALERROR> E File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR> E return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR> E File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR> E return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR> E return outcome.get_result()
INTERNALERROR> E File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR> E raise ex[1].with_traceback(ex[2])
INTERNALERROR> E File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR> E res = hook_impl.function(*args)
INTERNALERROR> E File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/xdist/remote.py", line 157, in pytest_runtestloop
INTERNALERROR> E self.run_one_test()
INTERNALERROR> E File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/xdist/remote.py", line 165, in run_one_test
INTERNALERROR> E item = items[self.item_index]
INTERNALERROR> E IndexError: list index out of range
INTERNALERROR> E assert False
INTERNALERROR>
INTERNALERROR> ../../../../hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/xdist/dsession.py:197: AssertionError
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/_pytest/main.py", line 269, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
INTERNALERROR> File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/_pytest/main.py", line 323, in _main
INTERNALERROR> config.hook.pytest_runtestloop(session=session)
INTERNALERROR> File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR> return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR> File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR> raise ex[1].with_traceback(ex[2])
INTERNALERROR> File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/xdist/dsession.py", line 122, in pytest_runtestloop
INTERNALERROR> self.loop_once()
INTERNALERROR> File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/xdist/dsession.py", line 145, in loop_once
INTERNALERROR> call(**kwargs)
INTERNALERROR> File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/xdist/dsession.py", line 184, in worker_workerfinished
INTERNALERROR> assert not crashitem, (crashitem, node)
INTERNALERROR> AssertionError: ('tests/epyccel/test_epyccel_openmp.py::test_omp_matmul_single[fortran]', <WorkerController gw3>)
INTERNALERROR> assert not 'tests/epyccel/test_epyccel_openmp.py::test_omp_matmul_single[fortran]'
I think this is related to xdist due to these lines:
INTERNALERROR> E File "/Users/runner/hostedtoolcache/Python/3.10.11/x64/lib/python3.10/site-packages/xdist/remote.py", line 165, in run_one_test
INTERNALERROR> E item = items[self.item_index]
INTERNALERROR> E IndexError: list index out of range
We have to verify if the crashitem handling has a issue
But it's also clear that something absolutely breaks your python interpreters with float state, that's on you
Yes, my code is definitely broken too :sweat_smile: . It's just a lot harder to read which test I broke like this. I have broken code similarly in the past, but I only started seeing these IndexErrors today so I just wanted to let you know
@EmilyBourne you say "with
3.3.0" when compared to3.2.1?
@nicoddemus I've gone through my PRs, but I haven't found one where I triggered this behaviour yet. I have seen this problem on our macosx and windows runners. I've just tried to reproduce it on my linux machine, but I am still getting the old expected behaviour even with the new version:
platform linux -- Python 3.10.6, pytest-7.2.1, pluggy-1.0.0
rootdir: /home/emily/Code/pyccel, configfile: pytest.ini
plugins: xdist-3.2.0
gw0 [3] / gw1 [3]
.Fatal Python error: Floating point exception
Thread 0x00007f1acf5b5640 (most recent call first):
File "/home/emily/.local/lib/python3.10/site-packages/execnet/gateway_base.py", line 400 in read
File "/home/emily/.local/lib/python3.10/site-packages/execnet/gateway_base.py", line 432 in from_io
File "/home/emily/.local/lib/python3.10/site-packages/execnet/gateway_base.py", line 967 in _thread_receiver
File "/home/emily/.local/lib/python3.10/site-packages/execnet/gateway_base.py", line 220 in run
File "/home/emily/.local/lib/python3.10/site-packages/execnet/gateway_base.py", line 285 in _perform_spawn
Current thread 0x00007f1ad1550000 (most recent call first):
File "/home/emily/Code/pyccel/tests/epyccel/recognised_functions/test_numpy_funcs.py", line 1065 in test_shape_property
File "/home/emily/.local/lib/python3.10/site-packages/_pytest/python.py", line 195 in pytest_pyfunc_call
File "/home/emily/.local/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
File "/home/emily/.local/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
File "/home/emily/.local/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
File "/home/emily/.local/lib/python3.10/site-packages/_pytest/python.py", line 1789 in runtest
File "/home/emily/.local/lib/python3.10/site-packages/_pytest/runner.py", line 167 in pytest_runtest_call
File "/home/emily/.local/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
File "/home/emily/.local/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
File "/home/emily/.local/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
File "/home/emily/.local/lib/python3.10/site-packages/_pytest/runner.py", line 260 in <lambda>
File "/home/emily/.local/lib/python3.10/site-packages/_pytest/runner.py", line 339 in from_call
File "/home/emily/.local/lib/python3.10/site-packages/_pytest/runner.py", line 259 in call_runtest_hook
File "/home/emily/.local/lib/python3.10/site-packages/_pytest/runner.py", line 220 in call_and_report
File "/home/emily/.local/lib/python3.10/site-packages/_pytest/runner.py", line 131 in runtestprotocol
File "/home/emily/.local/lib/python3.10/site-packages/_pytest/runner.py", line 112 in pytest_runtest_protocol
File "/home/emily/.local/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
File "/home/emily/.local/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
File "/home/emily/.local/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
File "/home/emily/.local/lib/python3.10/site-packages/xdist/remote.py", line 161 in run_one_test
File "/home/emily/.local/lib/python3.10/site-packages/xdist/remote.py", line 146 in pytest_runtestloop
File "/home/emily/.local/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
File "/home/emily/.local/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
File "/home/emily/.local/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
File "/home/emily/.local/lib/python3.10/site-packages/_pytest/main.py", line 324 in _main
File "/home/emily/.local/lib/python3.10/site-packages/_pytest/main.py", line 270 in wrap_session
File "/home/emily/.local/lib/python3.10/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main
File "/home/emily/.local/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
File "/home/emily/.local/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
File "/home/emily/.local/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
File "/home/emily/.local/lib/python3.10/site-packages/xdist/remote.py", line 342 in <module>
File "/home/emily/.local/lib/python3.10/site-packages/execnet/gateway_base.py", line 1084 in executetask
File "/home/emily/.local/lib/python3.10/site-packages/execnet/gateway_base.py", line 220 in run
File "/home/emily/.local/lib/python3.10/site-packages/execnet/gateway_base.py", line 285 in _perform_spawn
File "/home/emily/.local/lib/python3.10/site-packages/execnet/gateway_base.py", line 267 in integrate_as_primary_thread
File "/home/emily/.local/lib/python3.10/site-packages/execnet/gateway_base.py", line 1060 in serve
File "/home/emily/.local/lib/python3.10/site-packages/execnet/gateway_base.py", line 1554 in serve
File "<string>", line 8 in <module>
File "<string>", line 1 in <module>
Extension modules: mpi4py.MPI, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, mod_3wekuf8378w1_3wekuf8378w1, mod_9bzhnkfu6e60_9bzhnkfu6e60 (total: 16)
[gw1] node down: Not properly terminated
F
replacing crashed worker gw1
gw0 [3] / gw2 [3]
I will investigate further and get back to you
Also please investigate if it works with today's release, if the not in the main thread thing affects float modes of the libraries then it would be a horrible candidate for a problem origin
Strangely after having triggered this error 4 times in a row on the main repo. I then struggled to trigger it on the same code in a clone repo. After some digging I now have a test which reproduces this. It seems that this error is not in fact related to 3.3.0 I can also reproduce it with 3.2.1. Another interesting thing to note is that it only occurs when a large number of tests are run. Failing among 1454 tests triggers it, but not among 170 tests. I also can only trigger it on mac or windows. On linux it is better behaved
if the not in the main thread thing affects float modes of the libraries then it would be a horrible candidate for a problem origin
Sorry I'm not sure what you mean by this.
The test case involves a module which takes a numpy array as argument and indexes it incorrectly (with a stride of 0). It therefore crashes taking the worker with it. In a small test suite pytest-xdist creates a new worker. In a bigger test suite it raises the internal error shown above