Update Python from 3.6 to at least 3.7

[StephenChan]

3.6's end of support is 2021-12-23: https://www.python.org/downloads/

3.7's is 2023-06-27.

If I recall, the main thing to watch out for when updating from Python 3.x to a newer 3.x is that older versions of third-party packages may not have binaries for newer Python versions. We might've previously had this concern with 3.7 and Pillow, but since our Pillow has been updated to 6 or higher now, I don't think that in particular will be an issue. We'll have to see if similar concerns come up for other packages.

Be sure to update our docs to say 3.7 instead of 3.6. And to actually roll out this change, new virtualenvs need to be created.

[kriegman]

Isn't the current version of Pythong 3.9 now. Same issues apply about 3rd party libraries, but 3.9 is a major release.

On Wed, Oct 27, 2021 at 1:10 PM StephenChan @.***> wrote:

3.6's end of support is 2021-12-23: https://www.python.org/downloads/ https://urldefense.proofpoint.com/v2/url?u=https-3A__www.python.org_downloads_&d=DwMCaQ&c=-35OiAkTchMrZOngvJPOeA&r=dbhP10nX7gjq4ROoAmC1QBmMeWGLBmcA6EaLZwq2afM&m=aup5kJLsYe00YOslWBUnloqMtxm1WUb0bRZkFxC8meM&s=susEg8_8xd-eeNmB_ryZtYJG8dcxewt-cYychxwIWJU&e=

3.7's is 2023-06-27.

If I recall, the main thing to watch out for when updating from Python 3.x to a newer 3.x is that older versions of third-party packages may not have binaries for newer Python versions. We might've previously had this concern with 3.7 and Pillow, but since our Pillow has been updated to 6 or higher now, I don't think that in particular will be an issue. We'll have to see if similar concerns come up for other packages.

Be sure to update our docs to say 3.7 instead of 3.6. And to actually roll out this change, new virtualenvs need to be created.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_beijbom_coralnet_issues_416&d=DwMCaQ&c=-35OiAkTchMrZOngvJPOeA&r=dbhP10nX7gjq4ROoAmC1QBmMeWGLBmcA6EaLZwq2afM&m=aup5kJLsYe00YOslWBUnloqMtxm1WUb0bRZkFxC8meM&s=co_YkoG5nTWGjtFEuGP3HHromVJA6QEJj8foDgDdVVQ&e=, or unsubscribe https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ABKA5APLYM6OAQWBSQHJHALUJBMDPANCNFSM5G3GKQQQ&d=DwMCaQ&c=-35OiAkTchMrZOngvJPOeA&r=dbhP10nX7gjq4ROoAmC1QBmMeWGLBmcA6EaLZwq2afM&m=aup5kJLsYe00YOslWBUnloqMtxm1WUb0bRZkFxC8meM&s=tOPc54azIBd_OpGJHnM8UYsAZdfKsYTeT_bHe09F2zk&e= . Triage notifications on the go with GitHub Mobile for iOS https://urldefense.proofpoint.com/v2/url?u=https-3A__apps.apple.com_app_apple-2Dstore_id1477376905-3Fct-3Dnotification-2Demail-26mt-3D8-26pt-3D524675&d=DwMCaQ&c=-35OiAkTchMrZOngvJPOeA&r=dbhP10nX7gjq4ROoAmC1QBmMeWGLBmcA6EaLZwq2afM&m=aup5kJLsYe00YOslWBUnloqMtxm1WUb0bRZkFxC8meM&s=j0pvBKRwgv5fo7f0vuw31zAEKDGNVitM0npxa4LJ09o&e= or Android https://urldefense.proofpoint.com/v2/url?u=https-3A__play.google.com_store_apps_details-3Fid-3Dcom.github.android-26referrer-3Dutm-5Fcampaign-253Dnotification-2Demail-2526utm-5Fmedium-253Demail-2526utm-5Fsource-253Dgithub&d=DwMCaQ&c=-35OiAkTchMrZOngvJPOeA&r=dbhP10nX7gjq4ROoAmC1QBmMeWGLBmcA6EaLZwq2afM&m=aup5kJLsYe00YOslWBUnloqMtxm1WUb0bRZkFxC8meM&s=n0Yw0rT0c1qkrLD1YQxh0yGg0dG8uJdavcJwDucdHn0&e=.

[StephenChan]

Yeah - I suppose this issue should be "update to at least 3.7". If going further is low effort, then we should go further. Looks like 3.10 was released this month actually, so that's the latest.

[StephenChan]

Issue #310 (updating celery) appears to be required to update to Python 3.8 (maybe also 3.7, but I decided to just try 3.8):

• Celery 3.x depends on anyjson
• Trying to install anyjson on Python 3.8, I get: error in anyjson setup command: use_2to3 is invalid.
• Celery 4.x does not depend on anyjson: link

[StephenChan]

I got pretty curious, so I tried to get CoralNet running on Python 3.8 with a reasonable minimum of package updates (given that I use Windows and need to rely on package binaries more often, especially for numpy/scipy). I tried 3.8 because I happened to have that installed on my machine already, and I didn't have 3.7.

I got the Python environment installed and the coralnet unit tests running (but not all passing) in Python 3.8, using:

• These package requirements for coralnet: https://github.com/beijbom/coralnet/commit/174864c2b9448a8fd0ceb527f2607926fff62c43
• scipy 1.7.1 and scikit-image 0.18.3 (pyspacer currently uses 1.4.1 and 0.15.0)

Almost all (or all?) of the coralnet test failures were related to unittest.TestCase.assertRaises() being called incorrectly. So it actually seems pretty easy to get all of coralnet's tests working. However, I didn't try pyspacer's unit tests (relevant because of numpy, scipy, and scikit-image updates) or coralnet's regtests (relevant because of celery and redis-py updates), so those results remain to be seen. Also, I didn't check for deprecation warnings.

I think for now, 3.6 to 3.8 feels like a reasonable balance between 'fewer update cycles' and 'easier update cycles'. Going straight to 3.9 would require updating more stuff, like the PyTorch version.

I think a good roadmap to update to Python 3.8 would be something like this - a couple of steps could be skippable if we end up being in a rush:

• Port remaining usages of the old boto to boto3 (issue #82) so it doesn't give us deprecation warnings.
• Start turning on deprecation-warning output during unit-test runs: python -Wa manage.py test. Ensure there aren't any other major sources of such warnings at this point.
• Read celery release notes. Update celery (issue #310) and redis as needed.
• Update pyspacer's package requirements. At least scikit-image in setup.py; at least numpy, scipy, and scikit-image in requirements.txt. Bump pyspacer's minimum Python version according to what's supported by those packages.
• Get pyspacer unit tests working, and check for warnings.
• Update coralnet to Python 3.8 and apply the rest of the coralnet requirements updates from here: https://github.com/beijbom/coralnet/commit/174864c2b9448a8fd0ceb527f2607926fff62c43
• Get coralnet unit tests working, and check for warnings.
• Get coralnet regtests working, and check for warnings.
• Get ready to roll out Python 3.8 in production. This may involve updating Dockerfiles or similar?

[StephenChan]

Completed by PR #466; now on Python 3.10.