mars icon indicating copy to clipboard operation
mars copied to clipboard

[BUG] mars storage fetct data key error

Open chaokunyang opened this issue 2 years ago • 0 comments

Describe the bug A clear and concise description of what the bug is.

To Reproduce To help us reproducing this bug, please provide information below:

  1. Your Python version
  2. The version of Mars you use
  3. Versions of crucial packages, such as numpy, scipy and pandas
  4. Full stack of the error.
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/brain_predict/lib/sql_laundry.py", line 140, in download_marsdata_from_odps
    result_df = mars_o.to_mars_dataframe(tmp_table_name, chunk_size=chunk_size,).execute()
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/core/entity/tileables.py", line 462, in execute
    result = self.data.execute(session=session, **kw)
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/core/entity/executable.py", line 142, in execute
    return execute(self, session=session, **kw)
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/deploy/oscar/session.py", line 1864, in execute
    **kwargs,
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/deploy/oscar/session.py", line 1653, in execute
    timeout=self._isolated_session.timeout
  File "/usr/local/python3/lib/python3.7/concurrent/futures/_base.py", line 435, in result
    return self.__get_result()
  File "/usr/local/python3/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/deploy/oscar/session.py", line 1838, in _execute
    await execution_info
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/deploy/oscar/session.py", line 107, in wait
    return await self._aio_task
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/deploy/oscar/session.py", line 957, in _run_in_background
    raise task_result.error.with_traceback(task_result.traceback)
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/services/task/supervisor/processor.py", line 359, in run
    await self._process_stage_chunk_graph(*stage_args)
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/services/task/supervisor/processor.py", line 241, in _process_stage_chunk_graph
    stage_id, subtask_graph, chunk_graph, tile_context
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/services/task/execution/mars/executor.py", line 250, in execute_subtask_graph
    return await stage_processor.run()
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/services/task/execution/mars/stage.py", line 252, in run
    return await self._get_stage_result()
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/services/task/execution/mars/stage.py", line 136, in _get_stage_result
    await self._update_result_meta(execution_chunk_results)
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/services/task/execution/mars/stage.py", line 302, in _update_result_meta
    worker_metas = await asyncio.gather(*coros)
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/oscar/batch.py", line 153, in _async_batch
    return await self.batch_func(args_list, kwargs_list)
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/services/meta/api/oscar.py", line 182, in batch_get_chunk_meta
    return await self._meta_store.get_meta.batch(*get_chunk_metas)
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/context.py", line 196, in send
    return self._process_result_message(result)
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/context.py", line 76, in _process_result_message
    raise message.as_instanceof_cause()
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/pool.py", line 677, in send
    result = await self._run_coro(message.message_id, coro)
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/pool.py", line 370, in _run_coro
    return await coro
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/oscar/api.py", line 120, in __on_receive__
    return await super().__on_receive__(message)
  File "mars/oscar/core.pyx", line 510, in __on_receive__
  File "mars/oscar/core.pyx", line 490, in mars.oscar.core._BaseActor.__on_receive__
  File "mars/oscar/core.pyx", line 491, in mars.oscar.core._BaseActor.__on_receive__
  File "mars/oscar/core.pyx", line 499, in mars.oscar.core._BaseActor.__on_receive__
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/oscar/batch.py", line 153, in _async_batch
    return await self.batch_func(args_list, kwargs_list)
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/services/meta/store/dictionary.py", line 101, in batch_get_meta
    metas.append(self._get_meta(*args, **kwargs))
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/services/meta/store/dictionary.py", line 80, in _get_meta
    meta = self._store[object_id]
KeyError: [address=ray://ray-cluster-1656468237/4/0, pid=483612] 'f955907f4409be5cb344abc1b1cec01b_0'
  1. Minimized code to reproduce the error.

Expected behavior A clear and concise description of what you expected to happen.

Additional context Add any other context about the problem here.

chaokunyang avatar Jun 29 '22 11:06 chaokunyang