More NoneType errors

[mwidman]

Ok, this one does have to do with swiftclient. While running a long-running process, some of the objects I try to get from Rackspace produce the following error (in pyrax 1.4.9 with swiftclient 1.5.0.9 or commit https://github.com/openstack/python-swiftclient/commit/2f84a4e76b03863fe2e62515f150580b09dc20d1):

Traceback (most recent call last): File "", line 193, in _get_logged_data obj = self.container.get_object("%s%s.json" % (id, variable)) File "/usr/local/lib/python2.7/dist-packages/pyrax/cf_wrapper/container.py", line 111, in get_object full_listing=True) File "/usr/local/lib/python2.7/dist-packages/pyrax/cf_wrapper/client.py", line 60, in _wrapped ret = fnc(self, _args, *_kwargs) File "/usr/local/lib/python2.7/dist-packages/pyrax/cf_wrapper/client.py", line 995, in get_container_objects full_listing=full_listing) File "/home/.../src/python-swiftclient/swiftclient/client.py", line 1199, in get_container full_listing=full_listing) File "/home/.../src/python-swiftclient/swiftclient/client.py", line 1132, in _retry rv = func(self.url, self.token, _args, *_kwargs) File "/home/.../src/python-swiftclient/swiftclient/client.py", line 523, in get_container delimiter, end_marker, path, http_conn) File "/home/.../src/python-swiftclient/swiftclient/client.py", line 555, in get_container body = resp.read() File "/usr/lib/python2.7/httplib.py", line 548, in read s = self._safe_read(self.length) File "/usr/lib/python2.7/httplib.py", line 647, in _safe_read chunk = self.fp.read(min(amt, MAXAMOUNT)) File "/usr/lib/python2.7/socket.py", line 380, in read data = self._sock.recv(left) AttributeError: 'NoneType' object has no attribute 'recv'

[EdLeafe]

Can you give me a general idea as to what the long-running process does? I'd like to be able to reproduce this, but I ran the 1.4.9 release overnight and didn't see any problems like this, so I assume you're doing something in your code that my script didn't do.

[mwidman]

I have a process that uses a queue and 20 worker threads to take information from a no-sql database, process it, and then pass it into a queue where 10 worker threads write the processed data back to files in Rackspace Cloud Files.

[EdLeafe]

So would you say that the 10 write threads are working more or less continuously, or in a bursty sort of flow? And does this happen often, or just at a certain point in the evening? I'm asking because I'm wondering if there is a chance that when re-authentication is needed, we are getting a race condition as several threads are all asking for a new token at the same time.

[mwidman]

The 10 threads are working more or less continuously as far as I know.

I am not sure that this depends on a certain time of day or night but it just happened in 2 separate runs over the last couple of hours (the latest was at ~20:30 UTC).

I did notice that some of the objects that report the error are the same in both runs, but others are not so I am not sure if that is relevant or not. I also noticed that some of the objects also presented other errors (before or after the error I mentioned... the order seems somewhat random):

    obj = self.container.get_object("%s\\%s.json" % (id, variable))
  File "/usr/local/lib/python2.7/dist-packages/pyrax/cf_wrapper/container.py", line 111, in get_object
    full_listing=True)
  File "/usr/local/lib/python2.7/dist-packages/pyrax/cf_wrapper/client.py", line 60, in _wrapped
    ret = fnc(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pyrax/cf_wrapper/client.py", line 995, in get_container_objects
    full_listing=full_listing)
  File "/home/.../src/python-swiftclient/swiftclient/client.py", line 1199, in get_container
    full_listing=full_listing)
  File "/home/.../src/python-swiftclient/swiftclient/client.py", line 1132, in _retry
    rv = func(self.url, self.token, *args, **kwargs)
  File "/home/.../src/python-swiftclient/swiftclient/client.py", line 527, in get_container
    marker = listing[-1]['name']
TypeError: list indices must be integers, not str

---

    obj = self.container.get_object("%s\\%s.json" % (id, variable))
  File "/usr/local/lib/python2.7/dist-packages/pyrax/cf_wrapper/container.py", line 111, in get_object
    full_listing=True)
  File "/usr/local/lib/python2.7/dist-packages/pyrax/cf_wrapper/client.py", line 60, in _wrapped
    ret = fnc(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pyrax/cf_wrapper/client.py", line 995, in get_container_objects
    full_listing=full_listing)
  File "/home/.../src/python-swiftclient/swiftclient/client.py", line 1199, in get_container
    full_listing=full_listing)
  File "/home/.../src/python-swiftclient/swiftclient/client.py", line 1132, in _retry
    rv = func(self.url, self.token, *args, **kwargs)
  File "/home/.../src/python-swiftclient/swiftclient/client.py", line 523, in get_container
    delimiter, end_marker, path, http_conn)
  File "/home/.../src/python-swiftclient/swiftclient/client.py", line 570, in get_container
    return resp_headers, json_loads(body)
  File "/usr/lib/python2.7/dist-packages/simplejson/__init__.py", line 413, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/dist-packages/simplejson/decoder.py", line 402, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/dist-packages/simplejson/decoder.py", line 420, in raw_decode
    raise JSONDecodeError("No JSON object could be decoded", s, idx)
JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0)

[mwidman]

Moving back to pyrax v1.4.7 with swiftclient 1.5.0.8 seems to have removed the issue so I wonder if this may be due to code modified in 1.4.8 or 1.4.9 for get_object?

[mwidman]

Just tried with Pyrax 1.4.10 with Swiftclient 1.5.0.9 and still received the same error:

AttributeError: 'NoneType' object has not attribute in 'recv'

[EdLeafe]

There's definitely something odd going on; unfortunately, I'm on the road for the next few days, and don't know how much testing I can do. It does look as though swiftclient is losing its connection, based on the self._sock reference being None.