mars
mars copied to clipboard
[BUG] Ray task backend no progress
Describe the bug
Ray task mode doesn't update progress until whole task finished:
To Reproduce
To help us reproducing this bug, please provide information below:
- Your Python version: 3.7.9
- The version of Mars you use: master
- Versions of crucial packages, such as numpy, scipy and pandas
- Full stack of the error.
- Minimized code to reproduce the error.
def test_groupby(n=10):
from datetime import datetime
start = datetime.now()
df = md.DataFrame(
mt.random.rand(n * 500, 4, chunk_size=500),
columns=list('abcd'))
# print(df.sum().execute())
result = df.groupby(['a']).apply(lambda pdf: pdf).execute()
duration = datetime.now() - start
return result, duration
mars.new_session(n_worker=10, n_cpu=10*2, backend="ray")
test_groupby(200)
Expected behavior A clear and concise description of what you expected to happen.
Additional context Add any other context about the problem here.
I'll fix it soon.
There is a status bar and an a continuously updated value as follows:
And i add some log it will print: