pip icon indicating copy to clipboard operation
pip copied to clipboard

When extracting a wheel, do not recheck parent directories

Open morotti opened this issue 1 year ago • 5 comments

Hello,

Can i offer you a patch to make pip install ~5% faster? Counting 5% faster in average. Closer to 10% faster on slow disks or network volumes, closer to 1% faster on fast SSD.

The fix is pretty trivial. When extracting a wheel, do not recheck/recreate the parent directory before extracting each file.

Profiling a run of pip install plotly --prefix ./testdir

PROFILER ON MASTER: image

image

PROFILER WITH THIS FIX: image

image

Notice the 14425 calls to ensure_dir() taking 512ms total or ~5% of the runtime. Notice it's a lot less with the fix.

Total runtime 10.268 -> 9.414 seconds, thanks to all the calls removed + a few hundreds of ms of variance between pip runs.

Regards.

morotti avatar Jun 21 '24 13:06 morotti

FYI, the Windows CI failings are unrelated to your PR

notatallshaw avatar Jun 21 '24 13:06 notatallshaw

Hi, just a heads up that we just cut pip v24.1 so the core team's resources are being focused on making sure that release is handled smoothly. This PR looks simple so I'll try to take look tomorrow, but in the meanwhile, could you add a NEWS entry? Thank you!

ichard26 avatar Jun 22 '24 01:06 ichard26

Thanks, I added a NEWS entry.

morotti avatar Jun 24 '24 09:06 morotti

Builds is failing after rerunning, there are issues with windows builds and 3.13? trying to pickup changes on master for windows but then it's failing even earlier with new ruff rules?

src/pip/_internal/operations/install/wheel.py:418:5: C901 `_install_wheel` is too complex (34 > 33)
src/pip/_internal/operations/install/wheel.py:418:5: PLR0915 Too many statements (139 > 134)

morotti avatar Jun 24 '24 11:06 morotti

All green, should be good to merge now.

morotti avatar Jun 26 '24 17:06 morotti

this PR is ready to merge

morotti avatar Jul 04 '24 12:07 morotti

I'm going to schedule this PR for 24.2 as it seems like a small low-risk performance tweak. Feel free to remove the PR from the milestone if you object.

ichard26 avatar Jul 06 '24 20:07 ichard26

Thanks @morotti! ^.^

pradyunsg avatar Jul 09 '24 23:07 pradyunsg

Thank you very much @pradyunsg

morotti avatar Jul 10 '24 22:07 morotti