modin icon indicating copy to clipboard operation
modin copied to clipboard

PERF-#4777: Don't use `copy=True` parameter for `concat` calls inside `to_pandas`

Open anmyachev opened this issue 3 years ago • 3 comments

Signed-off-by: Myachev [email protected]

What do these changes do?

  • [ ] commit message follows format outlined here
  • [ ] passes flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
  • [ ] passes black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
  • [ ] signed commit with git commit -s
  • [x] Resolves #4777
  • [ ] tests added and passing
  • [ ] module layout described at docs/development/architecture.rst is up-to-date
  • [ ] added (Issue Number: PR title (PR Number)) and github username to release notes for next major release

anmyachev avatar Aug 04 '22 20:08 anmyachev

Codecov Report

Merging #4778 (5aae430) into master (eee5f43) will increase coverage by 5.23%. The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #4778      +/-   ##
==========================================
+ Coverage   84.63%   89.86%   +5.23%     
==========================================
  Files         265      266       +1     
  Lines       19626    19909     +283     
==========================================
+ Hits        16610    17892    +1282     
+ Misses       3016     2017     -999     
Impacted Files Coverage Δ
...dataframe/pandas/partitioning/partition_manager.py 89.25% <ø> (+3.88%) :arrow_up:
modin/core/dataframe/pandas/utils.py 100.00% <100.00%> (ø)
modin/logging/config.py 94.59% <0.00%> (-1.30%) :arrow_down:
modin/experimental/batch/test/test_pipeline.py 92.75% <0.00%> (ø)
modin/pandas/groupby.py 93.77% <0.00%> (+0.23%) :arrow_up:
modin/pandas/series.py 94.23% <0.00%> (+0.24%) :arrow_up:
modin/pandas/series_utils.py 99.43% <0.00%> (+0.56%) :arrow_up:
...odin/core/storage_formats/pandas/query_compiler.py 96.59% <0.00%> (+0.56%) :arrow_up:
modin/core/io/text/excel_dispatcher.py 94.01% <0.00%> (+0.85%) :arrow_up:
modin/experimental/xgboost/xgboost.py 87.61% <0.00%> (+0.95%) :arrow_up:
... and 45 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

codecov[bot] avatar Aug 04 '22 20:08 codecov[bot]

Nice!

jbrockmendel avatar Aug 04 '22 23:08 jbrockmendel

LGTM

jbrockmendel avatar Aug 08 '22 21:08 jbrockmendel

LGTM

Happy to merge but there are the following issues in our CI: ValueError: buffer source array is read-only. I don't have fix for this for now.

anmyachev avatar Aug 11 '22 16:08 anmyachev

CI: ValueError: buffer source array is read-only

does this happen on pandas main too?

jbrockmendel avatar Aug 29 '22 22:08 jbrockmendel

CI: ValueError: buffer source array is read-only

does this happen on pandas main too?

I don’t know, I don’t have the right environment for this at hand for now. However, we can definitely try to make at least one copy instead of two (if we can’t get rid of them at all)

anmyachev avatar Aug 30 '22 19:08 anmyachev

Any progress here?

vnlitvinov avatar Oct 05 '22 16:10 vnlitvinov

Any progress here?

Unfortunately no progress

anmyachev avatar Oct 06 '22 19:10 anmyachev

I don't know what else I can do here

anmyachev avatar Mar 01 '23 00:03 anmyachev

~Blocked by https://github.com/modin-project/modin/pull/6706~

anmyachev avatar Nov 05 '23 15:11 anmyachev