macs3-project
Bug: Runtime error pybind11::handle::inc_ref() PyGILState_Check() failure.
Error message below (bottom): INFO @ 21 Oct 2024 16:38:48: [107 MB]
Command line: hmmratac -i BED_files_sorted/9.bedpe -f BEDPE -n macs3_output/9
Random seed selected as: 10151
Use --hmm-type to select a Gaussian ('gaussian') or Poisson ('poisson') model for the hidden markov model in HMMRATAC. Default: 'gaussian'.
INFO @ 21 Oct 2024 16:38:48: [107 MB] #1 Read fragments from BEDPE file...
INFO @ 21 Oct 2024 16:38:49: [197 MB] 1000000 fragments parsed
INFO @ 21 Oct 2024 16:38:50: [213 MB] 2000000 fragments parsed
INFO @ 21 Oct 2024 16:38:51: [222 MB] 3000000 fragments parsed
INFO @ 21 Oct 2024 16:38:52: [226 MB] 4000000 fragments parsed
INFO @ 21 Oct 2024 16:38:54: [226 MB] 5000000 fragments parsed
INFO @ 21 Oct 2024 16:38:55: [227 MB] 6000000 fragments parsed
INFO @ 21 Oct 2024 16:38:56: [228 MB] 7000000 fragments parsed
INFO @ 21 Oct 2024 16:38:57: [229 MB] 8000000 fragments parsed
INFO @ 21 Oct 2024 16:38:58: [239 MB] 9000000 fragments parsed
INFO @ 21 Oct 2024 16:39:00: [245 MB] 10000000 fragments parsed
INFO @ 21 Oct 2024 16:39:01: [259 MB] 11000000 fragments parsed
INFO @ 21 Oct 2024 16:39:02: [263 MB] 12000000 fragments parsed
INFO @ 21 Oct 2024 16:39:03: [263 MB] 13000000 fragments parsed
INFO @ 21 Oct 2024 16:39:04: [273 MB] 14000000 fragments parsed
INFO @ 21 Oct 2024 16:39:06: [273 MB] 15000000 fragments parsed
INFO @ 21 Oct 2024 16:39:07: [279 MB] 16000000 fragments parsed
INFO @ 21 Oct 2024 16:39:08: [279 MB] 17000000 fragments parsed
INFO @ 21 Oct 2024 16:39:09: [289 MB] 18000000 fragments parsed
INFO @ 21 Oct 2024 16:39:10: [305 MB] 19000000 fragments parsed
INFO @ 21 Oct 2024 16:39:24: [358 MB] #2 Use EM algorithm to estimate means and stddevs of fragment lengths
INFO @ 21 Oct 2024 16:39:24: [358 MB] # for mono-, di-, and tri-nucleosomal signals...
INFO @ 21 Oct 2024 16:39:24: [359 MB] # A random seed 10151 has been used in the sampling function
INFO @ 21 Oct 2024 16:39:26: [428 MB] # Downsampled 1904083 fragments will be used for EM training...
INFO @ 21 Oct 2024 16:45:35: [428 MB] # Reached convergence after 14 iterations
INFO @ 21 Oct 2024 16:45:35: [428 MB] # The means and stddevs after EM:
INFO @ 21 Oct 2024 16:45:35: [428 MB] # short mono di tri
INFO @ 21 Oct 2024 16:45:35: [428 MB] # means: 50 210.1 382.4 611.7
INFO @ 21 Oct 2024 16:45:35: [428 MB] # stddevs: 20 45.8 45.7 101.8
INFO @ 21 Oct 2024 16:45:35: [428 MB] # Compute the weights for each fragment length for each of the four signal types
INFO @ 21 Oct 2024 16:45:42: [428 MB] # Generate short, mono-, di-, and tri-nucleosomal signals
INFO @ 21 Oct 2024 16:56:22: [928 MB] # Pile up all fragments
INFO @ 21 Oct 2024 16:56:43: [928 MB] # Convert pileup to fold-change over average signal
INFO @ 21 Oct 2024 16:56:49: [928 MB] #3 Look for training set from 19146841 fragments
INFO @ 21 Oct 2024 16:56:49: [928 MB] # Call peak above within fold-change range of 10 and 20.
INFO @ 21 Oct 2024 16:56:49: [928 MB] # The minimum length of the region is set as the average template/fragment length in the dataset: 592
INFO @ 21 Oct 2024 16:56:49: [928 MB] # The maximum gap to merge nearby significant regions is set as the flanking size to extend training regions: 1000
INFO @ 21 Oct 2024 16:56:53: [928 MB] # Total training regions called after applying the lower cutoff 10: 12930
INFO @ 21 Oct 2024 16:56:53: [928 MB] # Total training regions after filtering with upper cutoff 20: 3820
INFO @ 21 Oct 2024 16:56:53: [928 MB] # IMPORTANT
INFO @ 21 Oct 2024 16:56:53: [928 MB] # Please review the cutoff analysis result in macs3_output/9_cutoff_analysis.tsv to verify
INFO @ 21 Oct 2024 16:56:53: [928 MB] # if the choices of lower, upper and prescanning cutoff are appropriate.
INFO @ 21 Oct 2024 16:56:53: [928 MB] # Please read the message in the section 'Choices of cutoff values' by running
INFO @ 21 Oct 2024 16:56:53: [928 MB] # macs3 hmmratac -h for detail.
INFO @ 21 Oct 2024 16:56:53: [928 MB] # ****
INFO @ 21 Oct 2024 16:57:39: [928 MB] # We randomly pick 1000 regions for training
INFO @ 21 Oct 2024 16:57:39: [928 MB] # We expand the training regions with 1000 basepairs and merge overlap
INFO @ 21 Oct 2024 16:57:39: [928 MB] #4 Train Hidden Markov Model with Multivariate Gaussian Emission
INFO @ 21 Oct 2024 16:57:39: [928 MB] # Extract signals in training regions with bin size of 10
INFO @ 21 Oct 2024 16:57:50: [1116 MB] # Use Baum-Welch algorithm to train the HMM
pybind11::handle::inc_ref() is being called while the GIL is either not held or invalid. Please see https://pybind11.readthedocs.io/en/stable/advanced/misc.html#common-sources-of-global-interpreter-lock-errors for debugging advice.
If you are convinced there is no bug in your code, you can #define PYBIND11_NO_ASSERT_GIL_HELD_INCREF_DECREF to disable this check. In that case you have to ensure this #define is consistently used for all translation units linked into a given pybind11 extension, otherwise there will be ODR violations. The failing pybind11::handle::inc_ref() call was triggered on a numpy.ndarray object.
Traceback (most recent call last):
File "/Users/kiliankleemann/anaconda3/envs/macs3/bin/macs3", line 983, in
System (please complete the following information):
- OS: Mac OSX
- Python version: Python 3.11.4
- MACS Version macs3 3.0.1
@Kiliankleemann It seems to be caused by dependencies of numpy which has a major version update to version 2. Could you try to upgrade macs3 to 3.0.2 and try again? You need to use pip (https://pypi.org/project/MACS3/) to upgrade it. Let me know if the problem still exists. The reason may be that in 3.0.1 version, we only set the version of numpy to be >= 1.24, and the conda system may install the newest numpy 2 which seems to not work well with another dependency -- hmmlearn.
I ran into same issue today with macs3 v3.0.3:
Command:
macs3 hmmratac
-i $(find ./cell_type_atac_bams -type f -iname "Bl_*.bam" -print0 | xargs -0 printf "%s ")
-n Bl_all
-e ~/genome_references/mm10-blacklist.v2.bed
--outdir cell_type_atac_bams/macs_output
Output:
INFO @ 03 Jun 2025 09:53:39: [108 MB] #1 Read fragments from BAMPE file...
INFO @ 03 Jun 2025 09:53:39: [160 MB] 32824 fragments have been read.
INFO @ 03 Jun 2025 09:53:42: [176 MB] 1000000 fragments parsed
INFO @ 03 Jun 2025 09:53:46: [194 MB] 2000000 fragments parsed
INFO @ 03 Jun 2025 09:53:46: [224 MB] 2079561 fragments have been read.
INFO @ 03 Jun 2025 09:53:46: [225 MB] 9298 fragments have been read.
INFO @ 03 Jun 2025 09:53:46: [231 MB] 30880 fragments have been read.
INFO @ 03 Jun 2025 09:53:46: [232 MB] 5311 fragments have been read.
INFO @ 03 Jun 2025 09:53:48: [258 MB] 574120 fragments have been read.
INFO @ 03 Jun 2025 09:53:50: [264 MB] 1000000 fragments parsed
INFO @ 03 Jun 2025 09:53:51: [265 MB] 1087260 fragments have been read.
INFO @ 03 Jun 2025 09:53:51: [266 MB] 124926 fragments have been read.
INFO @ 03 Jun 2025 09:53:51: [267 MB] 122093 fragments have been read.
INFO @ 03 Jun 2025 09:53:51: [267 MB] 12788 fragments have been read.
INFO @ 03 Jun 2025 09:53:51: [267 MB] 17580 fragments have been read.
INFO @ 03 Jun 2025 09:53:52: [268 MB] 48981 fragments have been read.
INFO @ 03 Jun 2025 09:53:52: [277 MB] 286489 fragments have been read.
INFO @ 03 Jun 2025 09:53:56: [303 MB] 1000000 fragments parsed
INFO @ 03 Jun 2025 09:53:59: [308 MB] 2000000 fragments parsed
INFO @ 03 Jun 2025 09:54:00: [319 MB] 2172636 fragments have been read.
INFO @ 03 Jun 2025 09:54:00: [319 MB] 118370 fragments have been read.
INFO @ 03 Jun 2025 09:54:02: [339 MB] 531277 fragments have been read.
INFO @ 03 Jun 2025 09:54:02: [340 MB] 177647 fragments have been read.
INFO @ 03 Jun 2025 09:54:03: [340 MB] 9869 fragments have been read.
INFO @ 03 Jun 2025 09:54:06: [360 MB] 1000000 fragments parsed
INFO @ 03 Jun 2025 09:54:07: [367 MB] 1371423 fragments have been read.
INFO @ 03 Jun 2025 09:54:07: [368 MB] 218438 fragments have been read.
INFO @ 03 Jun 2025 09:54:07: [368 MB] 45278 fragments have been read.
INFO @ 03 Jun 2025 09:54:12: [369 MB] # Read 9077049 fragments.
INFO @ 03 Jun 2025 09:54:12: [369 MB] # Read blacklist file...
INFO @ 03 Jun 2025 09:54:23: [426 MB] # We removed 394801 fragments overlapping with blacklisted regions.
INFO @ 03 Jun 2025 09:54:23: [426 MB] # There are 8682248 fragments left.
INFO @ 03 Jun 2025 09:54:23: [426 MB] #2 Use EM algorithm to estimate means and stddevs of fragment lengths
INFO @ 03 Jun 2025 09:54:23: [426 MB] # for mono-, di-, and tri-nucleosomal signals...
INFO @ 03 Jun 2025 09:54:23: [426 MB] # A random seed 10151 has been used in the sampling function
INFO @ 03 Jun 2025 09:54:23: [458 MB] # Downsampled 439771 fragments will be used for EM training...
INFO @ 03 Jun 2025 09:55:22: [458 MB] # Reached maximum number (20) of iterations
INFO @ 03 Jun 2025 09:55:22: [458 MB] # The means and stddevs after EM:
INFO @ 03 Jun 2025 09:55:22: [458 MB] # short mono di tri
INFO @ 03 Jun 2025 09:55:22: [458 MB] # means: 50 156.5 305.5 515.5
INFO @ 03 Jun 2025 09:55:22: [458 MB] # stddevs: 20 42.5 37.6 102.1
INFO @ 03 Jun 2025 09:55:22: [458 MB] # Compute the weights for each fragment length for each of the four signal types
INFO @ 03 Jun 2025 09:55:23: [465 MB] # Generate short, mono-, di-, and tri-nucleosomal signals
INFO @ 03 Jun 2025 09:57:24: [735 MB] # Pile up all fragments
INFO @ 03 Jun 2025 09:57:26: [768 MB] # Convert pileup to fold-change over average signal
INFO @ 03 Jun 2025 09:57:26: [768 MB] #3 Look for training set from 8682248 fragments
INFO @ 03 Jun 2025 09:57:26: [768 MB] # Call peak above within fold-change range of 10 and 20.
INFO @ 03 Jun 2025 09:57:26: [768 MB] # The minimum length of the region is set as the average template/fragment length in the dataset: 125
INFO @ 03 Jun 2025 09:57:26: [768 MB] # The maximum gap to merge nearby significant regions is set as the flanking size to extend training regions: 1000
INFO @ 03 Jun 2025 09:57:27: [785 MB] # Total training regions called after applying the lower cutoff 10: 204259
INFO @ 03 Jun 2025 09:57:27: [785 MB] # Total training regions after filtering with upper cutoff 20: 54674
INFO @ 03 Jun 2025 09:57:27: [785 MB] # IMPORTANT
INFO @ 03 Jun 2025 09:57:27: [785 MB] # Please review the cutoff analysis result in cell_type_atac_bams/macs_output/Bl_all_cutoff_analysis.tsv to verify
INFO @ 03 Jun 2025 09:57:27: [785 MB] # if the choices of lower, upper and prescanning cutoff are appropriate.
INFO @ 03 Jun 2025 09:57:27: [785 MB] # Please read the message in the section 'Choices of cutoff values' by running
INFO @ 03 Jun 2025 09:57:27: [785 MB] # macs3 hmmratac -h for detail.
INFO @ 03 Jun 2025 09:57:27: [785 MB] # ****
INFO @ 03 Jun 2025 09:57:37: [786 MB] # We randomly pick 1000 regions for training
INFO @ 03 Jun 2025 09:57:37: [786 MB] # We expand the training regions with 1000 basepairs and merge overlap
INFO @ 03 Jun 2025 09:57:37: [786 MB] #4 Train Hidden Markov Model with Multivariate Gaussian Emission
INFO @ 03 Jun 2025 09:57:37: [786 MB] # Extract signals in training regions with bin size of 10
INFO @ 03 Jun 2025 09:57:38: [917 MB] # Use Baum-Welch algorithm to train the HMM
pybind11::handle::inc_ref() is being called while the GIL is either not held or invalid. Please see https://pybind11.readthedocs.io/en/stable/advanced/misc.html#common-sources-of-global-interpreter-lock-errors for debugging advice.
If you are convinced there is no bug in your code, you can #define PYBIND11_NO_ASSERT_GIL_HELD_INCREF_DECREF to disable this check. In that case you have to ensure this #define is consistently used for all translation units linked into a given pybind11 extension, otherwise there will be ODR violations. The failing pybind11::handle::inc_ref() call was triggered on a numpy.ndarray object.
Traceback (most recent call last):
File "/Users/zli5/micromamba/envs/macs3/bin/macs3", line 1017, in
System: Python 3.12.7 macs3 3.0.3 hmmlearn 0.3.2 numpy 2.1.2 macOS 14.7
Update: It resolved after downgrading numpy to 1.26.4 and upgrading hmmlearn to 0.3.3.
