Out_of memory-error(plot_map)

[Evadzi]

I experienced 'out-of-memory' issue when I attempted to use the "plot_map" operation to generate a map for landcover data for West Africa. Sometimes the screen goes blank. The memory of the file is 1 gb and I was not expecting error as my computer has a memory is 8 gb .

I think the issue of the out_of_memory_error should be looked at in detail if too much data is generated during execution of operations.

The error message is:

Job: Executing operation "plot_map" Message: Out of memory. Code: -32002 Method: set_workspace_resource Exception: MemoryError

Traceback (most recent call last): File "c:\users\prosper\projects\cate\cate\util\web\jsonrpchandler.py", line 209, in send_service_method_result result = future.result() File "C:\Users\prosper\AppData\Local\conda\conda\envs\cate-env\lib\concurrent\futures_base.py", line 425, in result return self.__get_result() File "C:\Users\prosper\AppData\Local\conda\conda\envs\cate-env\lib\concurrent\futures_base.py", line 384, in __get_result raise self._exception File "C:\Users\prosper\AppData\Local\conda\conda\envs\cate-env\lib\concurrent\futures\thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) File "c:\users\prosper\projects\cate\cate\util\web\jsonrpchandler.py", line 306, in call_service_method result = method(*method_params, monitor=monitor) File "c:\users\prosper\projects\cate\cate\webapi\websocket.py", line 289, in set_workspace_resource monitor=monitor) File "c:\users\prosper\projects\cate\cate\core\wsmanag.py", line 317, in set_workspace_resource workspace.execute_workflow(res_name=res_name, monitor=monitor) File "c:\users\prosper\projects\cate\cate\core\workspace.py", line 612, in execute_workflow self.workflow.invoke_steps(steps, context=self._new_context(), monitor=monitor) File "c:\users\prosper\projects\cate\cate\core\workflow.py", line 632, in invoke_steps step.invoke(context=context, monitor=monitor.child(work=1)) File "c:\users\prosper\projects\cate\cate\core\workflow.py", line 318, in invoke self._invoke_impl(_new_context(context, step=self), monitor=monitor) File "c:\users\prosper\projects\cate\cate\core\workflow.py", line 980, in _invoke_impl return_value = self._op(monitor=monitor, **input_values) File "c:\users\prosper\projects\cate\cate\core\op.py", line 216, in call return_value = self._wrapped_op(**input_values) File "c:\users\prosper\projects\cate\cate\ops\plot.py", line 217, in plot_map var_data.plot.pcolormesh(ax=ax, transform=ccrs.PlateCarree(), subplot_kws={'projection': proj}, **properties) File "C:\Users\prosper\AppData\Local\conda\conda\envs\cate-env\lib\site-packages\xarray\plot\plot.py", line 681, in plotmethod return newplotfunc(**allargs) File "C:\Users\prosper\AppData\Local\conda\conda\envs\cate-env\lib\site-packages\xarray\plot\plot.py", line 606, in newplotfunc cmap_params = _determine_cmap_params(**cmap_kwargs) File "C:\Users\prosper\AppData\Local\conda\conda\envs\cate-env\lib\site-packages\xarray\plot\utils.py", line 162, in _determine_cmap_params calc_data = np.ravel(plot_data[~pd.isnull(plot_data)]) MemoryError

Expected behavior:

operation is expected without exception. This is because I concatenated several time-series as well as generated animation map using similar data (only some few cases of out memory issue) .

Steps to reproduce the problem Kindly download and subset it for west Africa(or other area of similar size) generate 'plot-map' and specify the polygon region

Specifications cate 2.0.0.dev12 cate-desktop 2.0.0.dev12 Windows 10

[JanisGailis]

@Evadzi Completely agree, this should work. The good news are that we already have re-implemented the offending _determine_cmap_params to be less naive for animate_map, so this should be fixable. I'll look into it.