mars
mars copied to clipboard
mars-project
Mars is a tensor-based unified framework for large-scale data computation which scales numpy, pandas, scikit-learn and Python functions.
Now, `df.groupby().nunique()` would be delegated to `transform` to perform execution, it will be a shuffle operation which is very time consuming, we can delegate it to `aggregation` which is way...
# Background The current implementation of the group by aggregation API in Mars uses the shuffle implementation for large dataframes. In the shuffle algorithm, the final order of the grouped...
**Describe the bug** When running following code in nodebook cell and cancel in the middle and re-execute it again, mars will throw timeout error:: ``` urldf = df.groupby(["id"])["trd_longitude","trd_latitude","id"].apply(lambda x: x.sum()).reset_index().execute()...
This issue is used to track the progress of mars on ray. The main mars-on-ray related issue and pr should link with this issue. Suggestions and discussions are welcomed too.
**Is your feature request related to a problem? Please describe.** Most supercomputers in the world use of the few schedulers available, like the ones mentioned at the title. Those usually...
There are lots of school and company which use slurm or other cluster management and job scheduling system. It will be great if mars can support it.
**Describe the bug** Groupby shuffle keys for different groups are not well-distributed. In a online case which has 10000_0000 lines and chunk size is 20_0000, some gorups has about 24000...
# Motivation Now, when tileables have large number of chunks, building expr, subtask graph and so forth could become the bottleneck, we need to try to reduce the overhead. #...
## What do these changes do? Closes #2925 ## Related issue number Fixes #xxxx ## Check code requirements - [ ] tests added / passed (if needed) - [ ]...
Metadata
Owner
Metadata
Mars is a tensor-based unified framework for large-scale data computation which scales numpy, pandas, scikit-learn and Python functions.