Numpy compatibility

[Martin-Rey]

I faced an error today which I am posting here as a reference/documentation.

I (by mistake) generated a database storing numpy arrays with numpy 1.16. When retrieving the properties through both python and the web server on a different computer with numpy 1.15, I received:

Traceback (most recent call last):
  File "/anaconda3/lib/python3.6/site-packages/pyramid-1.9.1-py3.6.egg/pyramid/tweens.py", line 12, in _error_handler
    response = request.invoke_exception_view(exc_info)
  File "/anaconda3/lib/python3.6/site-packages/pyramid-1.9.1-py3.6.egg/pyramid/view.py", line 756, in invoke_exception_view
    raise HTTPNotFound
pyramid.httpexceptions.HTTPNotFound: The resource could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/anaconda3/lib/python3.6/site-packages/waitress/channel.py", line 338, in service
    task.service()
  File "/anaconda3/lib/python3.6/site-packages/waitress/task.py", line 169, in service
    self.execute()
  File "/anaconda3/lib/python3.6/site-packages/waitress/task.py", line 399, in execute
    app_iter = self.channel.server.application(env, start_response)
  File "/anaconda3/lib/python3.6/site-packages/pyramid-1.9.1-py3.6.egg/pyramid/router.py", line 270, in __call__
    response = self.execution_policy(environ, self)
  File "/anaconda3/lib/python3.6/site-packages/pyramid_retry-0.5-py3.6.egg/pyramid_retry/__init__.py", line 134, in retry_policy
    return request.invoke_exception_view(reraise=True)
  File "/anaconda3/lib/python3.6/site-packages/pyramid-1.9.1-py3.6.egg/pyramid/view.py", line 755, in invoke_exception_view
    reraise_(*exc_info)
  File "/anaconda3/lib/python3.6/site-packages/pyramid-1.9.1-py3.6.egg/pyramid/compat.py", line 148, in reraise
    raise value
  File "/anaconda3/lib/python3.6/site-packages/pyramid_retry-0.5-py3.6.egg/pyramid_retry/__init__.py", line 114, in retry_policy
    response = router.invoke_request(request)
  File "/anaconda3/lib/python3.6/site-packages/pyramid-1.9.1-py3.6.egg/pyramid/router.py", line 249, in invoke_request
    response = handle_request(request)
  File "/anaconda3/lib/python3.6/site-packages/pyramid_tm-2.2-py3.6.egg/pyramid_tm/__init__.py", line 171, in tm_tween
    reraise(*exc_info)
  File "/anaconda3/lib/python3.6/site-packages/pyramid_tm-2.2-py3.6.egg/pyramid_tm/compat.py", line 36, in reraise
    raise value
  File "/anaconda3/lib/python3.6/site-packages/pyramid_tm-2.2-py3.6.egg/pyramid_tm/__init__.py", line 136, in tm_tween
    response = handler(request)
  File "/anaconda3/lib/python3.6/site-packages/pyramid-1.9.1-py3.6.egg/pyramid/tweens.py", line 41, in excview_tween
    response = _error_handler(request, exc)
  File "/anaconda3/lib/python3.6/site-packages/pyramid-1.9.1-py3.6.egg/pyramid/tweens.py", line 16, in _error_handler
    reraise(*exc_info)
  File "/anaconda3/lib/python3.6/site-packages/pyramid-1.9.1-py3.6.egg/pyramid/compat.py", line 148, in reraise
    raise value
  File "/anaconda3/lib/python3.6/site-packages/pyramid-1.9.1-py3.6.egg/pyramid/tweens.py", line 39, in excview_tween
    response = handler(request)
  File "/anaconda3/lib/python3.6/site-packages/pyramid-1.9.1-py3.6.egg/pyramid/router.py", line 156, in handle_request
    view_name
  File "/anaconda3/lib/python3.6/site-packages/pyramid-1.9.1-py3.6.egg/pyramid/view.py", line 642, in _call_view
    response = view_callable(context, request)
  File "/anaconda3/lib/python3.6/site-packages/pyramid-1.9.1-py3.6.egg/pyramid/viewderivers.py", line 410, in viewresult_to_response
    result = view(context, request)
  File "/anaconda3/lib/python3.6/site-packages/pyramid-1.9.1-py3.6.egg/pyramid/viewderivers.py", line 148, in _requestonly_view
    response = view(request)
  File "/Users/martinrey/Documents/tangos/tangos/web/views/halo_data.py", line 287, in array_plot
    val, property_info = halo.calculate(name, True)
  File "/Users/martinrey/Documents/tangos/tangos/core/halo.py", line 145, in calculate
    (value,), description = calculation.values_sanitized_and_description([self], Session.object_session(self))
  File "/Users/martinrey/Documents/tangos/tangos/live_calculation/__init__.py", line 131, in values_sanitized_and_description
    values, desc = self.values_and_description(halos)
  File "/Users/martinrey/Documents/tangos/tangos/live_calculation/__init__.py", line 688, in values_and_description
    values = self.values(halos)
  File "/Users/martinrey/Documents/tangos/tangos/live_calculation/__init__.py", line 683, in values
    ret[0, i] = self._extraction_pattern.get_from_cache(h, self._name_id)[0]
  File "/Users/martinrey/Documents/tangos/tangos/core/extraction_patterns.py", line 61, in get_from_cache
    return self.postprocess_data_objects(return_vals)
  File "/Users/martinrey/Documents/tangos/tangos/core/extraction_patterns.py", line 114, in postprocess_data_objects
    return [self._postprocess_one_result(o) for o in outputs]
  File "/Users/martinrey/Documents/tangos/tangos/core/extraction_patterns.py", line 114, in <listcomp>
    return [self._postprocess_one_result(o) for o in outputs]
  File "/Users/martinrey/Documents/tangos/tangos/core/extraction_patterns.py", line 137, in _postprocess_one_result
    return self._mapper.get(property_object)
  File "/Users/martinrey/Documents/tangos/tangos/core/data_attribute_mapper.py", line 112, in get
    return self.unpack(getattr(db_object, self._attribute_name))
  File "/Users/martinrey/Documents/tangos/tangos/core/data_attribute_mapper.py", line 173, in unpack
    return self._unpack_compressed(packed)
  File "/Users/martinrey/Documents/tangos/tangos/core/data_attribute_mapper.py", line 161, in _unpack_compressed
    return pickle_loads(zlib.decompress(packed[2:]))
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

I tracked this thanks to this issue (https://github.com/numpy/numpy/issues/12825) to unpickling a numpy array with a numpy version different to the one it was pickled with. I can confirm that adapting my numpy version to the one which I used to save the array into the database solves this issue.

I suspect this is more of an issue on the side of numpy, to ensure that versions are backward compatible. I am unsure whether there are active actions to be taken on the TANGOS side?

Martin

[apontzen]

I confirm this is too challenging to fix for now as it relates to numpy internals changing between versions.