Improve import time

[Dreamsorcerer]

See https://github.com/aio-libs/aiohttp-debugtoolbar/issues/394#issuecomment-998280496

[lgtm-com[bot]]

This pull request introduces 1 alert when merging 3f18cfa0c2e7b821d654620782b00cb713d85230 into f78c128f52f4f7e55ebaa8ed0aa764031e98dfe8 - view on LGTM.com

new alerts:

• 1 for Unused local variable

[Dreamsorcerer]

Any idea what this internal error from coverage is about? Seems to have just started happening when I switched it to sys.executable.

[codecov[bot]]

Codecov Report

Merging #6591 (91087d2) into master (d7ebbeb) will decrease coverage by 0.01%. The diff coverage is 89.74%.

@@            Coverage Diff             @@
##           master    #6591      +/-   ##
==========================================
- Coverage   97.50%   97.49%   -0.02%     
==========================================
  Files         103      103              
  Lines       29990    30016      +26     
  Branches     3643     3647       +4     
==========================================
+ Hits        29243    29265      +22     
- Misses        567      569       +2     
- Partials      180      182       +2     

Flag	Coverage Δ
CI-GHA	97.40% <89.74%> (-0.02%)	:arrow_down:
OS-Linux	97.03% <84.61%> (-0.03%)	:arrow_down:
OS-Windows	95.35% <64.10%> (-0.04%)	:arrow_down:
OS-macOS	96.65% <66.66%> (-0.04%)	:arrow_down:
Py-3.10.8	97.15% <89.74%> (-0.02%)	:arrow_down:
Py-3.11.0	96.54% <84.61%> (-0.02%)	:arrow_down:
Py-3.7.15	96.87% <84.61%> (-0.02%)	:arrow_down:
Py-3.7.9	95.24% <64.10%> (-0.04%)	:arrow_down:
Py-3.8.10	95.14% <64.10%> (-0.05%)	:arrow_down:
Py-3.8.14	96.76% <84.61%> (-0.02%)	:arrow_down:
Py-3.9.13	95.13% <64.10%> (-0.04%)	:arrow_down:
Py-3.9.14	96.53% <66.66%> (-0.05%)	:arrow_down:
Py-3.9.15	96.74% <84.61%> (-0.02%)	:arrow_down:
Py-pypy7.3.9	96.40% <66.66%> (-0.05%)	:arrow_down:
VM-macos	96.65% <66.66%> (-0.04%)	:arrow_down:
VM-ubuntu	97.03% <84.61%> (-0.03%)	:arrow_down:
VM-windows	95.35% <64.10%> (-0.04%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
aiohttp/__init__.py	92.85% <86.66%> (-7.15%)	:arrow_down:
tests/test_imports.py	91.66% <91.66%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[Dreamsorcerer]

OK, think we're good now for a final review. Not sure why codecov thinks that the entire test I added is not being run, when it is clearly passing in the logs (in fact, it looks rather like it's only logged codecov for the Windows tests or something...).

In fact, the general codecov report looks a bit weird. For example, it claims that no code in ClientSession has run at all, which is not possible.

[hauntsaninja]

Thanks for fixing this! You may also be interested in https://github.com/benoitc/gunicorn/pull/2539 , which fixes the slowness in importing gunicorn

I've had to have hacky workarounds for this like https://github.com/hauntsaninja/boostedblob/blob/78986043e676806a1e2f8bc46606083b8000f332/boostedblob/init.py#L5 so really glad to see this change :-)

[Dreamsorcerer]

Thanks for fixing this! You may also be interested in benoitc/gunicorn#2539 , which fixes the slowness in importing gunicorn

Shame it's still not been merged.

[Dreamsorcerer]

Hmm, seems the tests can't import aiohttp anymore. Maybe need to do something with PYTHONPATH...

[Dreamsorcerer]

OK, I'm totally lost on why we always get an import error. Anybody got any ideas how to fix this?

[webknjaz]

Maybe because you're patching PYTHONPATH? Old tests didn't do that. And I don't know if pytester changes the directory (it probably does).

[webknjaz]

Also, you modify environ for the whole test process, not just one test. This will have weird side-effects.

[Dreamsorcerer]

Maybe because you're patching PYTHONPATH? Old tests didn't do that. And I don't know if pytester changes the directory (it probably does).

That was me trying to fix the issue, not the cause.

I think it might have been working before merging master, let's see if it works on an older pytest release, maybe some change there broke it..

[Dreamsorcerer]

OK, I've even pushed a separate branch from the time the tests were successful, but they are failing now: https://github.com/aio-libs/aiohttp/pull/7060

Seems like something outside of our code has changed the CI behaviour, so it no longer imports...

[webknjaz]

Does it pass locally for you?

[Dreamsorcerer]

OK, seem to have got it working now, even if it seems a bit awkward. The Python invocation seems to end up with half as many entries in sys.path than the main process does, so I've had to copy it into PYTHONPATH to get it to run with the same path values.

[Dreamsorcerer]

OK, full CI run passed twice with a threshold of 410ms. Think we can merge this and take a look at the other 2 PRs relating to import time.

[patchback[bot]]

Backport to 3.9: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply a8dbb68e19d93fb0a49ff0eca9fa3ec3d18cd4c9 on top of patchback/backports/3.9/a8dbb68e19d93fb0a49ff0eca9fa3ec3d18cd4c9/pr-6591

Backporting merged PR #6591 into master

1. Ensure you have a local repo clone of your fork. Unless you cloned it from the upstream, this would be your origin remote.
2. Make sure you have an upstream repo added as a remote too. In these instructions you'll refer to it by the name upstream. If you don't have it, here's how you can add it:

   $ git remote add upstream https://github.com/aio-libs/aiohttp.git
   

3. Ensure you have the latest copy of upstream and prepare a branch that will hold the backported code:

   $ git fetch upstream
   $ git checkout -b patchback/backports/3.9/a8dbb68e19d93fb0a49ff0eca9fa3ec3d18cd4c9/pr-6591 upstream/3.9
   

4. Now, cherry-pick PR #6591 contents into that branch:

   $ git cherry-pick -x a8dbb68e19d93fb0a49ff0eca9fa3ec3d18cd4c9
   

   If it'll yell at you with something like fatal: Commit a8dbb68e19d93fb0a49ff0eca9fa3ec3d18cd4c9 is a merge but no -m option was given., add -m 1 as follows intead:

   $ git cherry-pick -m1 -x a8dbb68e19d93fb0a49ff0eca9fa3ec3d18cd4c9
   

5. At this point, you'll probably encounter some merge conflicts. You must resolve them in to preserve the patch from PR #6591 as close to the original as possible.
6. Push this branch to your fork on GitHub:

   $ git push origin patchback/backports/3.9/a8dbb68e19d93fb0a49ff0eca9fa3ec3d18cd4c9/pr-6591
   

7. Create a PR, ensure that the CI is green. If it's not — update it so that the tests and any other checks pass. This is it! Now relax and wait for the maintainers to process your pull request when they have some cycles to do reviews. Don't worry — they'll tell you if any improvements are necessary when the time comes!

🤖 @patchback I'm built with octomachinery and my source is open — https://github.com/sanitizers/patchback-github-app.