"Can't pickle SwigPyObject objects", how to save a HawkesADM4 learner to file?

[yaowang74]

Hello!

First of all thanks to the entire team for developing such a good library for point process learning. I tried to save a fitted HawkesADM4 learner to file through pickling because it took me a long time to train, but I had the following error message:

---

TypeError Traceback (most recent call last) in 1 with open(f'/data1/wyao_workspace/notebooks/{input_path.stem}_adm4_learner.pkl', 'wb') as f: ----> 2 pk.dump(adm4learner, f)

TypeError: can't pickle SwigPyObject objects

Questions: is there an alternative to pickling that I can dump and load the fitted ADM4 learner or how to avoid this error?

PS: This error also prevents me from training model in parallel with multiprocessing ...

Thanks a lot.

Wang

[PhilipDeegan]

I think you need to use "dill"

[yaowang74]

Thank you Dekken, but I got same error when using dill

---

TypeError Traceback (most recent call last) in 2 3 with open(f'/data1/wyao_workspace/notebooks/{input_path.stem}_adm4_learner.pkl', 'wb') as f: ----> 4 dill.dump(learner_dict, f)

~/.conda/envs/LM2/lib/python3.6/site-packages/dill/_dill.py in dump(obj, file, protocol, byref, fmode, recurse, **kwds) 257 _kwds = kwds.copy() 258 _kwds.update(dict(byref=byref, fmode=fmode, recurse=recurse)) --> 259 Pickler(file, protocol, **_kwds).dump(obj) 260 return 261

~/.conda/envs/LM2/lib/python3.6/site-packages/dill/_dill.py in dump(self, obj) 443 raise PicklingError(msg) 444 else: --> 445 StockPickler.dump(self, obj) 446 stack.clear() # clear record of 'recursion-sensitive' pickled objects 447 return

~/.conda/envs/LM2/lib/python3.6/pickle.py in dump(self, obj) 407 if self.proto >= 4: 408 self.framer.start_framing() --> 409 self.save(obj) 410 self.write(STOP) 411 self.framer.end_framing()

~/.conda/envs/LM2/lib/python3.6/pickle.py in save(self, obj, save_persistent_id) 474 f = self.dispatch.get(t) 475 if f is not None: --> 476 f(self, obj) # Call unbound method with explicit self 477 return 478

~/.conda/envs/LM2/lib/python3.6/site-packages/dill/_dill.py in save_module_dict(pickler, obj) 910 # we only care about session the first pass thru 911 pickler._session = False --> 912 StockPickler.save_dict(pickler, obj) 913 log.info("# D2") 914 return

~/.conda/envs/LM2/lib/python3.6/pickle.py in save_dict(self, obj) 819 820 self.memoize(obj) --> 821 self._batch_setitems(obj.items()) 822 823 dispatch[dict] = save_dict

~/.conda/envs/LM2/lib/python3.6/pickle.py in _batch_setitems(self, items) 845 for k, v in tmp: 846 save(k) --> 847 save(v) 848 write(SETITEMS) 849 elif n:

~/.conda/envs/LM2/lib/python3.6/pickle.py in save(self, obj, save_persistent_id) 519 520 # Save the reduce() output and finally memoize the object --> 521 self.save_reduce(obj=obj, *rv) 522 523 def persistent_id(self, obj):

~/.conda/envs/LM2/lib/python3.6/pickle.py in save_reduce(self, func, args, state, listitems, dictitems, obj) 632 633 if state is not None: --> 634 save(state) 635 write(BUILD) 636

~/.conda/envs/LM2/lib/python3.6/pickle.py in save(self, obj, save_persistent_id) 474 f = self.dispatch.get(t) 475 if f is not None: --> 476 f(self, obj) # Call unbound method with explicit self 477 return 478

~/.conda/envs/LM2/lib/python3.6/site-packages/dill/_dill.py in save_module_dict(pickler, obj) 910 # we only care about session the first pass thru 911 pickler._session = False --> 912 StockPickler.save_dict(pickler, obj) 913 log.info("# D2") 914 return

~/.conda/envs/LM2/lib/python3.6/pickle.py in save_dict(self, obj) 819 820 self.memoize(obj) --> 821 self._batch_setitems(obj.items()) 822 823 dispatch[dict] = save_dict

~/.conda/envs/LM2/lib/python3.6/pickle.py in _batch_setitems(self, items) 845 for k, v in tmp: 846 save(k) --> 847 save(v) 848 write(SETITEMS) 849 elif n:

~/.conda/envs/LM2/lib/python3.6/pickle.py in save(self, obj, save_persistent_id) 519 520 # Save the reduce() output and finally memoize the object --> 521 self.save_reduce(obj=obj, *rv) 522 523 def persistent_id(self, obj):

~/.conda/envs/LM2/lib/python3.6/pickle.py in save_reduce(self, func, args, state, listitems, dictitems, obj) 632 633 if state is not None: --> 634 save(state) 635 write(BUILD) 636

~/.conda/envs/LM2/lib/python3.6/pickle.py in save(self, obj, save_persistent_id) 474 f = self.dispatch.get(t) 475 if f is not None: --> 476 f(self, obj) # Call unbound method with explicit self 477 return 478

~/.conda/envs/LM2/lib/python3.6/site-packages/dill/_dill.py in save_module_dict(pickler, obj) 910 # we only care about session the first pass thru 911 pickler._session = False --> 912 StockPickler.save_dict(pickler, obj) 913 log.info("# D2") 914 return

~/.conda/envs/LM2/lib/python3.6/pickle.py in save_dict(self, obj) 819 820 self.memoize(obj) --> 821 self._batch_setitems(obj.items()) 822 823 dispatch[dict] = save_dict

~/.conda/envs/LM2/lib/python3.6/pickle.py in _batch_setitems(self, items) 850 k, v = tmp[0] 851 save(k) --> 852 save(v) 853 write(SETITEM) 854 # else tmp is empty, and we're done

~/.conda/envs/LM2/lib/python3.6/pickle.py in save(self, obj, save_persistent_id) 494 reduce = getattr(obj, "reduce_ex", None) 495 if reduce is not None: --> 496 rv = reduce(self.proto) 497 else: 498 reduce = getattr(obj, "reduce", None)

TypeError: can't pickle SwigPyObject objects

[PhilipDeegan]

I'm not sure if we've configured serialization for this object.

if you'd like to give it a shot PRs are welcome

try something like this

git diff
diff --git a/lib/swig/tick/hawkes/inference/hawkes_adm4.i b/lib/swig/tick/hawkes/inference/hawkes_adm4.i
index 3ff65be..c9cbebf 100644
--- a/lib/swig/tick/hawkes/inference/hawkes_adm4.i
+++ b/lib/swig/tick/hawkes/inference/hawkes_adm4.i
@@ -26,3 +26,5 @@ class HawkesADM4 : public ModelHawkesList {
   void set_rho(const double rho);
 };
 
+
+TICK_MAKE_PICKLABLE(HawkesADM4);

[Mbompr]

Indeed we did not add the code to make this one picklable. Here is a tutorial that could guide you if you want to submit a PR :).

[yaowang74]

Thank you Mbompr for your reply. I am thinking of, for a short term workaround, can I just save all essential parameters of all my ADM4 learners, then is it possible to "recover" those learner by using one of simulator, e.g. SimuHawkesExpKernels ?

Thank you

Wang