pandarallel
pandarallel copied to clipboard
TypeError: cannot pickle 'sqlite3.Connection' object in pyCharm
General
- Operating System: Ubuntu 22.04
- Python version: 3.9.7
- Pandas version: 1.4.2
- Pandarallel version: 1.6.3
Acknowledgement
- [x] My issue is NOT present when using
pandaswithout alone (withoutpandarallel) - [ ] If I am on Windows, I read the Troubleshooting page before writing a new bug report
Bug description
I observe this when running parallel_apply in pyCharm. #76 sounds similar to me, but none of the tricks suggested there work for me. At this point I am also not sure if it's more of an issue with pyCharm.
Observed behavior
Traceback (most recent call last):
File "~/anaconda3/envs/tor/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3444, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-39-d230d86ff5ef>", line 1, in <module>
iris.groupby("species").parallel_apply(lambda x: np.mean(x))
File "~/anaconda3/envs/tor/lib/python3.9/site-packages/pandarallel/core.py", line 265, in closure
dilled_user_defined_function = dill.dumps(user_defined_function)
File "~/anaconda3/envs/tor/lib/python3.9/site-packages/dill/_dill.py", line 364, in dumps
dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)
File "~/anaconda3/envs/tor/lib/python3.9/site-packages/dill/_dill.py", line 336, in dump
Pickler(file, protocol, **_kwds).dump(obj)
File "~/anaconda3/envs/tor/lib/python3.9/site-packages/dill/_dill.py", line 620, in dump
StockPickler.dump(self, obj)
File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 487, in dump
self.save(obj)
File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "~/anaconda3/envs/tor/lib/python3.9/site-packages/dill/_dill.py", line 1963, in save_function
_save_with_postproc(pickler, (_create_function, (
File "~/anaconda3/envs/tor/lib/python3.9/site-packages/dill/_dill.py", line 1154, in _save_with_postproc
pickler._batch_setitems(iter(source.items()))
File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 997, in _batch_setitems
save(v)
File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 603, in save
self.save_reduce(obj=obj, *rv)
File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 692, in save_reduce
save(args)
File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 886, in save_tuple
save(element)
File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 603, in save
self.save_reduce(obj=obj, *rv)
File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 717, in save_reduce
save(state)
File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "~/anaconda3/envs/tor/lib/python3.9/site-packages/dill/_dill.py", line 1251, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 971, in save_dict
self._batch_setitems(obj.items())
File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 997, in _batch_setitems
save(v)
File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "~/anaconda3/envs/tor/lib/python3.9/site-packages/dill/_dill.py", line 1251, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 971, in save_dict
self._batch_setitems(obj.items())
File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 997, in _batch_setitems
save(v)
File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 603, in save
self.save_reduce(obj=obj, *rv)
File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 717, in save_reduce
save(state)
File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "~/anaconda3/envs/tor/lib/python3.9/site-packages/dill/_dill.py", line 1251, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 971, in save_dict
self._batch_setitems(obj.items())
File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 997, in _batch_setitems
save(v)
File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "~/anaconda3/envs/tor/lib/python3.9/site-packages/dill/_dill.py", line 1251, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 971, in save_dict
self._batch_setitems(obj.items())
File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 997, in _batch_setitems
save(v)
File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 578, in save
rv = reduce(self.proto)
TypeError: cannot pickle 'sqlite3.Connection' object
Minimal but working code sample to ease bug fix for pandarallel team
import pandas as pd
import numpy as np
from pandarallel import pandarallel
pandarallel.initialize(use_memory_fs=False)
iris = pd.read_csv('https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv')
iris.groupby("species").parallel_apply(lambda x: np.mean(x))
I am having same issue
Pandaral·lel is looking for a maintainer! If you are interested, please open an GitHub issue.
The minimal code sample also produces an error for me outside of PyCharm.
Python: 3.10.13 Pandarallel: 1.6.5 Pandas: 2.2.0