Python 2 sunsetting in Jan 2020

[ridv]

Python 2 will officially sunset January 1st, 2020 according to the Python maintainers https://www.python.org/doc/sunset-python-2/

Since Aurora relies pretty heavily on Python to run tasks in critical spots, I think it would be quite dangerous to run with no further security patches.

At the same, I don't think we have the man power to move all our Py2 code to Py3. Any ideas on what our path forward should be?

[JustinVenus]

@rdelval I'm familiar with pants and have moved some internal projects from python2 to 3 ... I can help with this effort, however you'd like to proceed.

[ridv]

That would be a fantastic and much welcome contribution. Even moving a few things here and there would be a big help!

[StephanErb]

We have looked into Python 3, a bit as we consume Aurora 's API from within Python.

Code wise there will not be too many hurdles. However there is a bit of pain with Thrift: Aurora's API require Sets in many places and this requires Thrift objects to be hashable in Python. Unfortunately Thrifts generated code by default comes without suitable hash methods for Python 3.

We have resorted to this nasty hack for now:

# With python 3 objects need a hash method to put them inside of sets. This was not the
# case in python 2. We monkeypatch a hash method on to the the thrift objects to make them
# compatible with python 3 code without having to make any changes to the thrift api
# itself.
def __hash__(self):
    return hash(self.__class__) ^ hash(1)


setattr(Resource, "__hash__", __hash__)
setattr(Metadata, "__hash__", __hash__)
setattr(Constraint, "__hash__", __hash__)

Thrift have a few options to change how code is generated. This might be something worth exploring. Another potential option could be to migrate to https://pypi.org/project/thriftpy2/

[ridv]

Thanks Stephan! That's a lot of really great information.

So it sounds like we need to weigh our options between helping the Apache Thrift project fix this issue or switching to thriftpy2

Do we have any idea what it would take to switch to thriftpy2? I will try reaching out to the Thrift devs and see if they're open to working with us on fixing this bug.

[JustinVenus]

Hey @StephanErb Does this seem like a reasonable way to solve the unhashable thrift structure issue? https://github.com/JustinVenus/aurora/commit/3757b726660e186a83fba4cca2d3a4d65aeb50db

[ridv]

An update on the python 3 hash situation: https://github.com/apache/thrift/pull/1835

[cjbarth]

With the recent patch that landed for mutable exceptions, I've been running Thrift with Python 3 for quite some time very successfully. Is there something else that prevents the move to Python 3 for you? I want to make sure we aren't silently being bitten by it.

[ridv]

For us, we have a bunch of dependencies on libraries (graciously) donated to the OSS project by Twitter that haven't received an update to make them fully compatible with Python 3. Thrift support was a big concern though so thanks for landing that patch @cjbarth !