Fixed #33380 -- Prioritize likely user files when watching for file changes in auto-reloader

[hrushikeshrv]

This is an intial attempt at fixing #33380 by putting the user's project files at the start of all the files returned by iter_modules_and_files

This patch doesn't put only the user's project files first, there are some non-project files that shouldn't get as high a priority as project files, but for now I'm not sure how to filter those out. For example, for my installation of python, most of the project files are put at the very start, but a few files like ...\ProgramData\Anaconda3\DLLs\_socket.pyd also appear near the start. I'm not sure how to fix that.

[kezabelle]

For example, for my installation of python, most of the project files are put at the very start, but a few files like ...\ProgramData\Anaconda3\DLLs_socket.pyd also appear near the start.

Yeah I don't have a complete understanding of all the various places things can end up, especially given the different ways to set things up, so there's always going to be a few stragglers, and it won't matter (IMHO) to scan a few files erroneously as if they were 'user' files - they're just being prioritised ever so slightly earlier per tick, which is not any different to the current scenario.

Here for example, are the various paths my django repo has, given my specific setup:

In [2]: sys.base_exec_prefix
Out[2]: '/Users/kez/.asdf/installs/python/3.10.1'

In [3]: sys.base_prefix
Out[3]: '/Users/kez/.asdf/installs/python/3.10.1'

In [4]: sys.exec_prefix
Out[4]: '/Users/kez/Code/django/.direnv/python-3.10.1'

In [5]: sys.prefix
Out[5]: '/Users/kez/Code/django/.direnv/python-3.10.1'

In [6]: site.PREFIXES
Out[6]: ['/Users/kez/Code/django/.direnv/python-3.10.1']

In [7]: site.USER_SITE
Out[7]: '/Users/kez/.local/lib/python3.10/site-packages'

In [8]: site.USER_BASE
Out[8]: '/Users/kez/.local'

In [9]: site.getsitepackages()
Out[9]: ['/Users/kez/Code/django/.direnv/python-3.10.1/lib/python3.10/site-packages']

There's at least 3 different locations there - os and logging for example are actually within '/Users/kez/.asdf/installs/python/3.10.1/lib/python3.10/[^1] and IPython is in /Users/kez/Code/django/.direnv/python-3.10.1/lib/python3.10/site-packages/IPython/ and AFAIK nothing ends up in site.USER_SITE for my setup ... It's probably notably different on Windows and Linux compared to OSX too :/

[^1]: I wish I'd thought to document the setup I had where os and logging were in different places, but alas it was 3 years and 1 laptop ago. I suspect it was when I was using pyenv and not asdf though...

[hrushikeshrv]

I think sorting just the file paths returned by iter_modules_and_files won't be enough. I've just noticed that iter_modules_and_files is not returning all of my project files that I was testing it with. Crucially, it didn't return my/project/directory/app2/views.py and my/project/directory/app3/views.py for the test project I had set up.

They were returned later, at the very end, possibly by BaseReloader.watched_files() when it yields from self.extra_files (although I'm not 100% sure about that).

iter_modules_and_files() returned 577 files for the test project, and BaseReloader.snapshot_files() returned 718 files.

[felixxm]

@hrushikeshrv Do you have time to keep working on this?

[hrushikeshrv]

@felixxm I will be happy to work on this starting next week, if that's okay.

[hrushikeshrv]

@felixxm I've updated the checks to now correctly order by priority, but I can't figure out why tests in gis_tests are failing.