python 3 support?

[talumbau]

Do you have plans to support both python 2 and 3?

[wouterdebie]

At Spotify we're still on python2, so there is no real incentive to support python3. That said, I think we could support python3. I'm not sure what strategy to choose here; source code that is compatible with both 2 and 3 or have separate branches..

[talumbau]

Ah I see. I haven't looked at all of the source, but my instinct is that you could have one source that is compatible with 2 and 3.

[ravwojdyla]

To start with it, I used Fig to create fresh development environment with python 3.4 on Ubuntu Trusty. Unfortunately everything borked at very beginning:

pip install -r requirements-dev.txt

To be more specific for protobuf package. Protobuf is not supported for python 3.x, for more details please look here: https://groups.google.com/forum/#!topic/protobuf/sb2pOm0SY6w

It may be hard to move forward with this feature ... closing(?)

[wouterdebie]

Look at this :)

http://protobuf.googlecode.com/svn/trunk/CHANGES.txt mentions python 3 support for 2.6.0.

Maybe we don't close it?

[ravwojdyla]

wat!? protobuf 2.6.0 Supported protobuf for Hadoop is 2.5.0. I guess this ticket requires some investigation - how do you want to tackle this?

[wouterdebie]

I just checked our Travis builds and they already pull in 2.6.0 because we depend on >2.4.1 :) Seems to work fine. There is one weird thing though is that protobuf 2.6.0 depends on python-dateutil>=1.4,<2. On my local install I had python-dateutil 2.2 installed and broke my upgrade to protobuf 2.6.0. With new installations this shouldn't be too big of a deal (as is the case with the Travis builds)

[ravwojdyla]

True if I change dependancy to >=2.6.0 on ubuntu with python3.4 I can install all the dev requirements. setup.py test doesn't work tho. Simple snakebite -v either. Due to incompatibility py2 vs py3.

[wouterdebie]

I'm trying to see how much effort it is. There seems to be an issue with google-apputils, one of the dependencies of protobuf: https://github.com/google/protobuf/issues/9

[wouterdebie]

@ravwojdyla and myself did some more experimentation and it seems that google has to fix their protobuf for python3 before we can move on with this. We'll keep this issue open.

[vmarkovtsev]

OK, Google has fixed it. I ran 2to3 on all the sources and replaced str("", "utf-8") with bytes("", "utf-8") in protobuf and was able to successfully import snakebite. Not sure if it works correct though.

[vmarkovtsev]

Update: looks like protobuf defs should be updated, I am getting

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.4/dist-packages/snakebite/client.py", line 140, in ls
    recurse=recurse):
  File "/usr/local/lib/python3.4/dist-packages/snakebite/client.py", line 1082, in _find_items
    fileinfo = self._get_file_info(path)
  File "/usr/local/lib/python3.4/dist-packages/snakebite/client.py", line 1214, in _get_file_info
    return self.service.getFileInfo(request)
  File "/usr/local/lib/python3.4/dist-packages/snakebite/service.py", line 35, in <lambda>
    rpc = lambda request, service=self, method=method.name: service.call(service_stub_class.__dict__[method], request)
  File "/usr/local/lib/python3.4/dist-packages/snakebite/service.py", line 41, in call
    return method(self.service, controller, request)
  File "/usr/local/lib/python3.4/dist-packages/google/protobuf/service_reflection.py", line 267, in <lambda>
    self._StubMethod(inst, method, rpc_controller, request, callback))
  File "/usr/local/lib/python3.4/dist-packages/google/protobuf/service_reflection.py", line 284, in _StubMethod
    method_descriptor.output_type._concrete_class, callback)
  File "/usr/local/lib/python3.4/dist-packages/snakebite/channel.py", line 409, in CallMethod
    self.get_connection(self.host, self.port)
  File "/usr/local/lib/python3.4/dist-packages/snakebite/channel.py", line 219, in get_connection
    rpc_header = self.create_rpc_request_header()
  File "/usr/local/lib/python3.4/dist-packages/snakebite/channel.py", line 248, in create_rpc_request_header
    rpcheader.clientId = self.client_id[0:16]
  File "/usr/local/lib/python3.4/dist-packages/google/protobuf/internal/python_message.py", line 471, in field_setter
    self._fields[field] = type_checker.CheckValue(new_value)
  File "/usr/local/lib/python3.4/dist-packages/google/protobuf/internal/type_checkers.py", line 103, in CheckValue
    raise TypeError(message)
TypeError: '35bc231f-07ae-41' has type <class 'str'>, but expected one of: (<class 'bytes'>,)

[wouterdebie]

We will probably will need to compile the protobufs with version 3.0.0 before it to work. Unfortunately proto3 is still in alpha and he proto3 compiler isn't mainstream yet. Given that we need to recompile the protobufs, I'm not sure how to proceed, since this would make a python3/proto3 version incompatible with python2/proto2 version. @ravwojdyla any idea? Should we maintain two different versions? Or have a bunch of conditionals based on proto2/3, python2/3 and in the future probably hadoop versions in code?

[vmarkovtsev]

No need to recompile. Just apply the fixing patch to 2.6.1: https://github.com/google/protobuf/pull/50

[gpoulin]

@wouterdebie any idea on how you plan to proceed? I started to port over spotify/luigi#747 and snakebite is currently a blocker. I had a fast look at snakebite code and I don't think supporting both python2/3 with one source code would be difficult. The main work would require to use unicode and bytes literal and encode and decode at the appropriate place (I can help for this part if you want). The tricky part would be the dependency on protobuf.

[Tarrasch]

@gpoulin can't you solve this on luigi side? Core luigi shouldn't really depend on snakebite.

[wouterdebie]

@gpoulin would be great to see some progress here. I would like to have one version that supports both py2/3. And yes, the protobuf stuff we'll have to solve, but I don't mind keeping two different compiled sets for that.

[gpoulin]

@Tarrasch I was under the impression that it was a soft requirement for Luigi, but hard requirement for "Hadoop Luigi", since all the Hadoop tests were failing on import snakebite. However having a closer look it seems only to had more functionality if present. Anyway, having snakebite running on python 3 would still be great.

[Tarrasch]

@gpoulin Exactly, it's basically that luigi will run faster with snakebite because it don't need to shell out to hadoopcli. :)

[Tarrasch]

@talumbau @gpoulin @vmarkovtsev @wouterdebie. Do you guys have any updates on the status for the python 3 support?

[gpoulin]

@Tarrasch Even thou I'm currently working more in scala, I'm willing to help here. However, my understanding is that the version of protobuf that snakebite is currently using doesn't support python3. So if snakebite can support a protobuf version that support python3, I can help to make this project python3 compatible.

[Tarrasch]

@gpoulin, I did try to scan the status for protobuf and python3 in general, but it doesn't seem to be well supported if at all.

Since I can not wait for a way to do hdfs fast in python3, I'll probably switch over to webhdfs.

[Tarrasch]

For luigi users who are fine with using webhdfs, this could be relvant spotify/luigi#1438

[krzysztof-indyk]

@gpoulin protobuf compatible with python3 has been relased https://pypi.python.org/pypi/protobuf/3.0.0b2

[kirklg]

Hey all, first of all, great work, this project is exactly what I need for a hadoop project I'm working on, and speed is definitely important (py3 more specifically).

Anyways, I'm going to take a stab at this, python-protobuf3 looks stable and my initial py3 compat work looks promising. I will update when I have more, thank you.

[anneschuth]

@kirklg awesome! Did you make any progress?

[kirklg]

@anneschuth, yes, but its been slow moving, I have my fork, https://github.com/kirklg/snakebite/pull/2, that you can watch if you want to subscribe to progress :). I have a string of commits coming soon when I find the time, thanks.

[anneschuth]

Thanks @kirklg, looking forward to your commits!

[FlorianWilhelm]

@kirklg Any news on your PR? I am also looking forward to snakebite with Python 3 support. Thanks for making this possible.

[sumyeon]

Hey, are there any news on support for Python 3? I hope to heare any news, even if it is negative.

[wouterdebie]

Given that at Spotify we don't use python3 there is no incentive to work on this. I hope someone else from the community takes a stab at this, but it's hard to say it that will happen.

On Feb 15, 2017 9:10 AM, "sumyeon" [email protected] wrote:

Hey, are there any news on support for Python 3? I hope to heare any news, even if it is negative.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/spotify/snakebite/issues/62#issuecomment-279943784, or mute the thread https://github.com/notifications/unsubscribe-auth/AAKgBtIdHP1B6jCICDhOa286iaoYBrn-ks5rcrLdgaJpZM4CBlij .