Mitigate dependency confusion attacks

[qequ]

What this PR does / why we need it: Changes to dockerfile and pip.conf for 1.10.15, 2.14, 2.3.3, 2.3.4 and main to mitigate Dependency confusion attacks.

reference; https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610

CC: @dgutson @arieltorti

[ashb]

This might cause problems at run/ONBUILD time. We'll run some checks

[dgutson]

This might cause problems at run/ONBUILD time. We'll run some checks

ping? :)

[dgutson]

Hi @ashb , this is a security issue and therefore we're in a zugzwang situation that leaves us no other alternative that create a fork and file the CVE. We do not want to reach that stage. Do you have an ETA for this?

[ashb]

@dgutson @qequ sorry for the silence, I was all hands on deck getting Airflow 2.4.0 out! I'm now looking at this and it's my top priority for now.

I'll get back to you shortly.

[ashb]

Fixed this issue another way (by creating name-squatting packages on pypi)

Also worth noting that we avoided this problem for our newer Astronomer Runtime images which we encourage all users to use for new deployments instead. There we fixed the problem by using a different pip repo which implements the "redirect through" approach, so the repo checks if the package exists locally, and if not will issue an HTTP redirect to the public PyPI repo.

[arieltorti]

Awesome @ashb ! That will work as well, we just need to make sure to name-squat all newly added dependencies.

Thanks for all the effort in understanding and fixing the issue