Pandas.error and shape mismatch

[partha434]

Dear Wouter, I am getting the following error. The version of methplotlib I am using is 0.13.1. Please help.

methplotlib -m methylation_calls_CpG_sm1.tsv.gz methylation_frequency_sm1.tsv.gz -n calls frequencies -w mtDNA:1-16,626 -g /cluster/projects/nn9337k/as1_nanopolish/all_context/sequence.sorted.gtf.gz --simplify -b /cluster/projects/nn9337k/as1/reference.bed -f /cluster/projects/nn9337k/as1/reference.fasta

Traceback (most recent call last): File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3621, in get_loc return self._engine.get_loc(casted_key) File "pandas/_libs/index.pyx", line 136, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/index.pyx", line 142, in pandas._libs.index.IndexEngine.get_loc TypeError: '18910 False 18973 False 19035 False 19097 False 19159 False ... 22925 False 23000 True 23074 False 23148 False 23224 False Name: log_lik_ratio, Length: 28154, dtype: bool' is an invalid key

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/series.py", line 1085, in setitem self._set_with_engine(key, value) File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/series.py", line 1146, in _set_with_engine loc = self.index.get_loc(key) File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3628, in get_loc self._check_indexing_error(key) File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 5637, in _check_indexing_error raise InvalidIndexError(key) pandas.errors.InvalidIndexError: 18910 False 18973 False 19035 False 19097 False 19159 False ... 22925 False 23000 True 23074 False 23148 False 23224 False Name: log_lik_ratio, Length: 28154, dtype: bool

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/bin/methplotlib", line 8, in sys.exit(main()) File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/methplotlib/methplotlib.py", line 27, in main meth_browser(meth_data=meth_data, File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/methplotlib/methplotlib.py", line 51, in meth_browser meth_traces = plots.methylation(meth_data, dotsize=dotsize) File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/methplotlib/plots.py", line 105, in methylation make_per_read_meth_traces_llr(table=meth.table, File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/methplotlib/plots.py", line 167, in make_per_read_meth_traces_llr table.loc[:, "llr_scaled"] = rescale_log_likelihood_ratio(table["log_lik_ratio"].copy()) File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/methplotlib/plots.py", line 253, in rescale_log_likelihood_ratio llr[llr > 0] = scaler.fit_transform(llr[llr > 0].values.reshape(-1, 1)).tolist() File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/series.py", line 1127, in setitem self._set_values(indexer, value) File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/series.py", line 1185, in _set_values self._mgr = self._mgr.setitem(indexer=key, value=value) File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/internals/managers.py", line 337, in setitem return self.apply("setitem", indexer=indexer, value=value) File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/internals/managers.py", line 304, in apply applied = getattr(b, f)(**kwargs) File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/internals/blocks.py", line 959, in setitem values[indexer] = value ValueError: shape mismatch: value array of shape (5284,1) could not be broadcast to indexing result of shape (5284,)

[wdecoster]

Can you upgrade methplotlib to the latest version? Yours is rather outdated, and it is possible that the issue you are reporting is already solved.

[partha434]

Hello, I tried to upgrade methplotlib, but unfortunately it still shows the old version. I have installed it via conda. On the conda page the version shows 0.20.1, but still I can't install it.

[wdecoster]

I tried to upgrade methplotlib, but unfortunately it still shows the old version.

But the upgrade was successful? Have you earlier installed this with pip? You can try uninstalling with pip and conda first. You can use which methplotlib to see where it is installed.

but still I can't install it.

You have an error?

[partha434]

I successfully installed methplotlib to 0.20.1. But I got this error...

methplotlib -m methylation_calls_CpG_sm1.tsv methylation_frequency_sm1.tsv -n calls frequencies -w mtDNA:1-16626 -g /cluster/projects/nn9337k/as1_nanopolish/all_context/sequence.sorted.gtf.gz --simplify -b /cluster/projects/nn9337k/as1/reference.bed

Reading methylation_calls_CpG_sm1.tsv would be faster with bgzip and tabix. Please index with 'tabix -S1 -s1 -b3 -e4'.

Reading methylation_frequency_sm1.tsv would be faster with bgzip and tabix. Please index with 'tabix -S1 -s1 -b2 -e3'. Traceback (most recent call last): File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3621, in get_loc return self._engine.get_loc(casted_key) File "pandas/_libs/index.pyx", line 136, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/index.pyx", line 142, in pandas._libs.index.IndexEngine.get_loc TypeError: '18971 False 19033 False 19094 False 19156 False 19218 False ... 22888 False 22962 True 23035 False 23109 False 23183 False Name: log_lik_ratio, Length: 28154, dtype: bool' is an invalid key

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/series.py", line 1085, in setitem self._set_with_engine(key, value) File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/series.py", line 1146, in _set_with_engine loc = self.index.get_loc(key) File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3628, in get_loc self._check_indexing_error(key) File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 5637, in _check_indexing_error raise InvalidIndexError(key) pandas.errors.InvalidIndexError: 18971 False 19033 False 19094 False 19156 False 19218 False ... 22888 False 22962 True 23035 False 23109 False 23183 False Name: log_lik_ratio, Length: 28154, dtype: bool

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/bin/methplotlib", line 8, in sys.exit(main()) File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/methplotlib/methplotlib.py", line 26, in main meth_browser(meth_data=meth_data, File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/methplotlib/methplotlib.py", line 53, in meth_browser meth_traces = plots.methylation(meth_data, dotsize=dotsize, binary=binary, minqual=minqual) File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/methplotlib/plots.py", line 106, in methylation make_per_read_meth_traces_llr(table=meth.table, File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/methplotlib/plots.py", line 183, in make_per_read_meth_traces_llr table.loc[:, "llr_scaled"] = rescale_log_likelihood_ratio(table["log_lik_ratio"].copy()) File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/methplotlib/plots.py", line 269, in rescale_log_likelihood_ratio llr[llr > 0] = scaler.fit_transform(llr[llr > 0].values.reshape(-1, 1)).tolist() File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/series.py", line 1127, in setitem self._set_values(indexer, value) File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/series.py", line 1185, in _set_values self._mgr = self._mgr.setitem(indexer=key, value=value) File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/internals/managers.py", line 337, in setitem return self.apply("setitem", indexer=indexer, value=value) File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/internals/managers.py", line 304, in apply applied = getattr(b, f)(**kwargs) File "/cluster/projects/nn9337k/anaconda/envs/methplotlib/lib/python3.10/site-packages/pandas/core/internals/blocks.py", line 959, in setitem values[indexer] = value ValueError: shape mismatch: value array of shape (5284,1) could not be broadcast to indexing result of shape (5284,)

[wdecoster]

Could you please share the log file created by methplotlib?

[partha434]

Hello, I have attached the log file generated from methplotlib. methplotlib_20220905_0847.log

[partha434]

Dear Wouter, Any updates related to the above error? I have checked pip installation with pip check. It also says no broken requirements found. But still I am getting the above error.

[wdecoster]

Would it be possible to share your input files?

[partha434]

Sure. Please find the attached files for calls and frequency. methylation_calls_CpG_sm1.txt methylation_frequency_sm1.txt

[wdecoster]

It works on my system. Can you try upgrading pandas?

[partha434]

I upgraded pandas to latest version 1.4.4 (earlier version was 1.4.3). But I got the same error. Please help.

[wdecoster]

Can you please try to create a conda environment for just methplotlib and see if you have the error if you execute methplotlib there.

conda create -n methplotlib methplotlib
conda activate methplotlib

If you still have the error, please export the environment (conda env export > env.yaml) and share that with me so I can try to reproduce.

[partha434]

Dear Wouter, I created a new conda environment as per your suggestions. But I am still facing the same error. Please find the attached env.yaml file (in text here). Waiting for your kind help. Thank you so much in advance. env.txt

[partha434]

Dear Wouter, I checked my environment with pip check. There is no broken requirements.

[xiaohuli-45]

Dear Wouter, I checked my environment with pip check. There is no broken requirements.

Hello, I have encountered a similar problem. May I ask if you have resolved the issue

[mrmrwinter]

I am also getting a ery similar error. Using methplotlib version 0.20.1

[senukobla]

I am getting the same error . Any solutions ?

[courtnicles]

Same error for me

[wdecoster]

Would it be possible to share your input data and parameters, in order to replicate this problem?