pyrax
pyrax copied to clipboard
pycontribs
Bad Request - Expecting username
Hi,
I have an issue with pyrax that continues to haunt me even at night. it has been bothering us for a long time now. I will try to give as much details as possible.
We use Pyrax in a big Django application with cloudfiles for storing images. We generate temporary urls that are then exposed to our customers. The django app is served by gunicorn.
Here is the raw traceback:
File "peopletools/utils.py", line 80, in get_insight_cloud_files_container
return client.create_container(settings.INSIGHT_CLOUDFILES_CONTAINER)
File "pyrax/cf_wrapper/client.py", line 77, in _wrapped
pyrax.authenticate(connect=False)
File "pyrax/__init__.py", line 369, in _wrapped
return fnc(*args, **kwargs)
File "pyrax/__init__.py", line 519, in authenticate
_auth_and_connect(connect=connect)
File "pyrax/__init__.py", line 496, in _auth_and_connect
identity.authenticate()
File "pyrax/identity/rax_identity.py", line 66, in authenticate
super(RaxIdentity, self).authenticate()
File "pyrax/base_identity.py", line 292, in authenticate
raise exc.AuthenticationFailed(err)
We are using sentry to capture the error, with the following stacktrace:
The weird thing is in the last frame. Password, username and tenantId are all equal to None. I'm pretty sure they are correctly set in our settings file and it was working for almost 12 days.
Here is the code that is trying to get or create the cloudfiles container:
def create_cloudfiles_client():
global CLOUDFILES_CLIENT
if CLOUDFILES_CLIENT is None:
pyrax.set_setting("identity_type", "rackspace")
pyrax.set_credentials(settings.CLOUDFILES_USERNAME,
settings.CLOUDFILES_API_KEY,
region=settings.CLOUDFILES_REGION)
CLOUDFILES_CLIENT = pyrax.connect_to_cloudfiles(
settings.CLOUDFILES_REGION, public=not settings.CLOUDFILES_SERVICENET)
return CLOUDFILES_CLIENT
def get_insight_cloud_files_container():
client = create_cloudfiles_client()
return client.create_container(settings.INSIGHT_CLOUDFILES_CONTAINER)
Here is another traceback, in another application, the one generating images and upload them to cloudfiles:
ERROR:swiftclient:Object PUT failed: https://snet-storage101.lon3.clouddrive.com:443/v1/MossoCloudFS_c0417f3b-7815-43a3-85a6-e291228f197e/previews_prod/v2/ca9bc610aac29879ebe3a6ae47c49d5e50ea495e/document_normal_p2.png 401 Unauthorized [first 60 chars of response] <html><h1>Unauthorized</h1><p>This server could not verify t
Traceback (most recent call last):
File "...virtualenvs/insight-reloaded/local/lib/python2.7/site-packages/swiftclient/client.py", line 1110, in _retry
rv = func(self.url, self.token, *args, **kwargs)
File "...virtualenvs/insight-reloaded/local/lib/python2.7/site-packages/swiftclient/client.py", line 922, in put_object
http_response_content=body)
ClientException: Object PUT failed: https://snet-storage101.lon3.clouddrive.com:443/v1/MossoCloudFS_c0417f3b-7815-43a3-85a6-e291228f197e/previews_prod/v2/ca9bc610aac29879ebe3a6ae47c49d5e50ea495e/document_normal_p2.png 401 Unauthorized [first 60 chars of response] <html><h1>Unauthorized</h1><p>This server could not verify t
Here is the code trying to get or create the container and call an HTTP PUT on cloudifle: https://github.com/novapost/insight-reloaded/blob/master/insight_reloaded/storage/cloudfiles.py#L14
Unfortunately I doesn't have sentry traceback for this application but I will try to get more details.
As I said before we had this issue for a long time, it appears just after a Pyrax update (doesn't remember the exact release but I maybe can find it in my mails). We tried a workaround at this time, the workaround was to call connect_to_cloudfiles for each request. It did the job but after another release of Pyrax, the workaround was no longer valid because Pyrax was hitting the US identity entrypoint (https://github.com/rackspace/pyrax/issues/271). So we tried to update to latest release of Pyrax. We are currently using release 1.6.2 on both applications. The version of python-swiftclient is also the same: 1.8.0.
Another weird thing is that before we put the workaround in production, we were having the problem every day. During the last 12 days we were having no problem at all. Maybe the token expiration time has been increased.
When we are facing this problem, the only solution we found is to restart our gunicorn.
I tried to dig into the code and I have some questions:
In file https://github.com/rackspace/pyrax/blob/master/pyrax/cf_wrapper/client.py, we are catching swift exception but pyrax doesn't check the status code of ClientException (https://github.com/openstack/python-swiftclient/blob/master/swiftclient/exceptions.py#L29). Is this comment still relevant?
I tried to see where username, password and tenantId could be set to None. I find no place where password could be set except https://github.com/rackspace/pyrax/blob/master/pyrax/base_identity.py#L107
By any chance are you running a multi-threaded application? The reason I ask is that another pyrax user experienced a similar issue around re-authentication, and the explanation I got from the swiftclient team was that multiple requests were not always handled properly.
We are launching our app application with this command:
newrelic-admin run-program app/bin/gunicorn_django -b 0.0.0.0:8000 -w 10
But for our application which generate previews, it's a vanilla python process, no multi-threading or multi-process and we have problems too.
FWIW, the token expiration time hasn't changed. I don't know why you haven't seen the problem for 12 days.
With regard to your first question, you might be on to something. There was an issue where swiftclient retried auth only once, even if the initial exception wasn't an auth failure. I've added #310 to fix this.
For the second question, there is also the unauthenticate() methods in base_identity.py and client.py, but you would have to explicitly call them; there isn't anything in the framework that calls those methods (they are probably going to be removed soon, in any case).
Yes we arre still seeing this issue but we didn't update pyrax? Do you want us to try the master?
I don't think that anything in the current master has changed regarding this. I'm currently working on a local branch that removes the swiftclient dependency, which will make debugging issues like this much, much easier. That won't be ready for another week or so.