TransientException: RpcBufferedReader only managed to read 0 out of 4 bytes

[synfinatic]

Looks similar to #169, but a few things I should mention:

1. I'm using the snakebite API via Inviso
2. The error is happening during iteration of client.ls()
3. The problem happens randomly, usually on large recursive listings.

I've looked on my namenode log files and haven't seen any red flags.

[synfinatic]

INFO:inviso-monitor:checking path: /tmp/hadoop-yarn/staging/history/done/2016/10/21
ERROR:inviso.jes:RpcBufferedReader only managed to read 0 out of 4 bytes
Traceback (most recent call last):
  File "jes.py", line 43, in main
    monitor.run()
  File "/home/aaron_turner/inviso/jes/inviso/monitor.py", line 317, in run
    for f in listing:
  File "/home/aaron_turner/venv/local/lib/python2.7/site-packages/snakebite/client.py", line 168, in ls
    recurse=recurse):
  File "/home/aaron_turner/venv/local/lib/python2.7/site-packages/snakebite/client.py", line 1245, in _find_items
    recurse=recurse):
  File "/home/aaron_turner/venv/local/lib/python2.7/site-packages/snakebite/client.py", line 1245, in _find_items
    recurse=recurse):
  File "/home/aaron_turner/venv/local/lib/python2.7/site-packages/snakebite/client.py", line 1245, in _find_items
    recurse=recurse):
  File "/home/aaron_turner/venv/local/lib/python2.7/site-packages/snakebite/client.py", line 1214, in _find_items
    fileinfo = self._get_file_info(path)
  File "/home/aaron_turner/venv/local/lib/python2.7/site-packages/snakebite/client.py", line 1342, in _get_file_info
    return self.service.getFileInfo(request)
  File "/home/aaron_turner/venv/local/lib/python2.7/site-packages/snakebite/service.py", line 40, in <lambda>
    rpc = lambda request, service=self, method=method.name: service.call(service_stub_class.__dict__[method], request)
  File "/home/aaron_turner/venv/local/lib/python2.7/site-packages/snakebite/service.py", line 46, in call
    return method(self.service, controller, request)
  File "/home/aaron_turner/venv/local/lib/python2.7/site-packages/google/protobuf/service_reflection.py", line 267, in <lambda>
    self._StubMethod(inst, method, rpc_controller, request, callback))
  File "/home/aaron_turner/venv/local/lib/python2.7/site-packages/google/protobuf/service_reflection.py", line 284, in _StubMethod
    method_descriptor.output_type._concrete_class, callback)
  File "/home/aaron_turner/venv/local/lib/python2.7/site-packages/snakebite/channel.py", line 447, in CallMethod
    return self.parse_response(byte_stream, response_class)
  File "/home/aaron_turner/venv/local/lib/python2.7/site-packages/snakebite/channel.py", line 395, in parse_response
    len_bytes = byte_stream.read(4)
  File "/home/aaron_turner/venv/local/lib/python2.7/site-packages/snakebite/channel.py", line 125, in read
    self._buffer_bytes(bytes_wanted)
  File "/home/aaron_turner/venv/local/lib/python2.7/site-packages/snakebite/channel.py", line 144, in _buffer_bytes
    raise TransientException("RpcBufferedReader only managed to read %s out of %s bytes" % (len(bytes_read), n))
TransientException: RpcBufferedReader only managed to read 0 out of 4 bytes

I should also mention, I'm using different library versions then what Inviso has out of the box. My requirements.txt:

argparse==1.2.1 arrow==0.4.2 boto==2.26.1 elasticsearch==1.0.0 futures==2.1.6 #protobuf==2.5.0 protobuf==2.6.1 python-dateutil==2.2 pytz==2013.9 requests==2.3.0 six==1.5.2 #snakebite==2.4.1 snakebite==2.11.0 #urllib3==1.7.1 urllib3==1.18

[pragmattica]

We recently also had some "RpcBufferedReader only managed to read" errors, and I wanted to make a note here with our suspicions. Hopefully these notes will prove useful to someone at some point.

We had never seen this error before, through several years of using snakebite. However, when it came up last week-end, it appeared to be related to using snakebite to download incredibly massive files from HDFS. We had always used it before to get maybe 10 GB files, but then someone tried grabbing some files of maybe 150 GB or so, and we saw this error, but only sometimes...

Very strange and intermittent. Looks like this kind of bug will not be easy to track down.