port to python 3

[warner]

Python 2's End-Of-Life date is Jan 1, 2020. Tahoe-LAFS currently depends upon Foolscap, as does git-foolscap and flancer and a handful of personal tools. Tahoe's nominal plan is to switch to a new HTTP-based transport layer and remove Foolscap entirely, but that will introduce backwards-compatibility problems (basically each grid will have a flag day where they must update everybody to the new py3-without-foolscap version of Tahoe at the same time).

I'm not making any scheduling promises, but I'm working on porting Foolscap to python3: watch the py3 branch to follow along. I'm aiming to make it work under both py2 and py3, like I did with magic-wormhole.

[warner]

API plan: the normal API surface will accept both bytes and text on both py2+py3. The return values will be version-native strings (so bytes on py2 and text on py3).

Since Tahoe is basically the primary customer of Foolscap, I'm gonna define the "normal" API surface as whatever Tahoe uses, which is a subset of foolscap.api plus some logging pieces and foolscap.connections, and also foolscap.base32. The parts of this API that involve strings are:

• RemoteReference.callRemote(methodname): methodname can be either bytes or text. Arguments are passed through unmodified, so if the wire crosses a py2/py3 boundary then you may have to change the receive-side application code to be tolerant of either bytes or strings (with e.g. six.ensure_text()), and/or change the sending-side code to coerce the arguments. The schema definitions of RemoteInterface are unchanged, so you may also have to modify the schema to allow the new types.
• RemoteReference.getSturdyRef(), .getRemoteTubID(): returns a FURL/tubid as a native string
• Copyable: the typeToCopy and copytype class-attributes can be either bytes or text
• logging.app_versions(): returns native strings
• Tub.setOption: name can be either, string-ish values can be either
• Tub.addConnectionHintHandler
• Tub.setLocation, .listenOn: argument can be either
• Tub.getTubID, getShortTubID: return value is native string
• Tub.registerReference: the optional name= argument can be either type, the FURL it returns will be a native string
• Tub.getReference, .connectTo: the FURL can be either

I'm removing some features that aren't yet supported in py3: socks, i2p, the (optional and never used) serialization of "unsafe" types like classes and modules.

The long term plan is to have everything return text, so some APIs which Foolscap provides but which tahoe does not use might change to return text/unicode in all places.

[warner]

Actually, I'm not going to bother supporting py2. I want the new py3 version to interoperate with the old py2 version (which means paying careful attention to the values you pass into callRemote, so that you aren't sending str aka unicode from py3 and expecting str aka bytes on a py2 receiver). But I don't think we have any downstream users who need py2+py3 compatibility: Tahoe will be py3-only, and my other tools can easily become py3-only.

Maintaining interoperability means e.g. method names must continue to be sent as bytestrings, since that's what py2 expects. We could conceivably add a negotiation phase where both sides realize they are running py3 and can switch to sending unicode methodnames, but I doubt it's going to be worth it.

[meejah]

To clarify, I believe the current Tahoe plan is not to have a "flag day" but to support python2 and python3, deprecate Foolscap and move to an HTTP-based API ... but there will certainly be an overlap there where Tahoe supports both Foolscap and HTTP and a time when python2 and python3 overlap.

That said, it could be that our Python2 dependencies list a specific Foolscap version (for example)...? (This might be fragile).

[exarkun]

To clarify, I believe the current Tahoe plan is not to have a "flag day" but to support python2 and python3

This is definitely the plan now. If this conflicts with the plan for Foolscap we really need to get together and sort this out.

[pythonhacker]

@exarkun - How does this change the plan for our Python3 port ? Seeing this thread now.

[exarkun]

I don't know. Since @warner has taken the task of porting Foolscap to Python 3 we really need to discuss with him.

[warner]

Hey, sorry to be out of the loop: swamped with work. Short answer: don't wait on me, do whatever is best for Tahoe. In April I thought I was going to have time to work on Foolscap, but that went away. If y'all decide to port foolscap as part of the tahoe+py3 effort, awesome, I'll help with what little time I can contribute (and feel free to agressively delete features to make the process easier). If you don't, cool, I might make an attempt myself later in the year to support the non-tahoe projects that could use it, but I can't predict when or if that might happen.

Feel free to mine my branch for patches or ideas, but it isn't in great shape and may or may not be useful as a starting point.

[warner]

Ok, that big PR lands the initial support: all tests now pass under py3 (py3.5, 3.6, 3.7, 3.8, and they still pass under py2.7 which should make tahoe testing easier). And tahoe's tests pass when using the new foolscap.

Next steps:

• [x] check interoperability between (py2 + tahoe + old-foolscap) and (py2 + tahoe + new-foolscap)
• [x] write some small foolscap-using apps and check interop between (py2 + old-foolscap), (py2 + new-foolscap), and (py3 + new-foolscap)
• [x] test logging code (flogtool tail, incident-gatherer, web-display, flogtool dump) between those three cases. I expect py2-py3 will break, and I won't be surprised if we need to do some work to make (py2+old) and (py2+new) work properly
  • [x] flogtool tail
  • [x] log gatherer
  • [x] incident gatherer
  • [x] web display
  • [x] flogtool dump
• [ ] test appserver code (flappserver)
• [ ] write a "py2-3 porting guide" for application developers (specifically the tahoe py3 porting team)
• [ ] update setup.py and other metadata to reflect the new py3-capable codebase
• [ ] make a release

[warner]

basic smoke test between py2-old, py2-new, py3-new appears to work

flogtool tail between those combinations works, default arguments (like time and message) are displayed normally (no spurious b'' or u'' prefixes, we render the message into a native string before converting into text for printing)

[warner]

Logging: when #67 lands, the compatibility situation will be:

• flogtool tail between py2-old, py2-new, and py3-new works without problems
• saving events (flogtool tail --save-to=, the log-gatherer and incident-gatherer) fails if the emitter is py2 and the receiver/saver is py3, because emitter creates dictionaries with bytes as keys, and the JSON module in py3 cannot serialize these (even when the keys are plain ASCII): #68
• saving events also fails if the emitter is py3-new, and the receiver is py2-old. I don't yet know why. It works if the receiver is py2-new, somehow.

I've not yet tested the web-display.

[warner]

web-display works, after landing be5178d

[warner]

flogtool dump works, pretty much like the web-viewer:

• if the saved flogfile came from py2, and the flogtool web-viewer / flogtool dump process is also py2, some messages get an extra u'' prefix. If the viewer/dumper process is py3, this prefix usually goes away
• if the viewer/dumper process is py3, some banana negotiation log events have a b'' prefix

but otherwise the tools work as expected