mod_xsendfile serving files with url-encoded names

[AdrianLC]

Hello,

This is my first time using both xsendfile and downloadview so I might be doing something wrong. I apologize if that is the case.

So I'm trying to serve files with a non ascii characters and getting the file names all messed up.

My httpd.conf:

    Alias /priv/ /var/www/wsgi/site/priv/
    <Directory /var/www/wsgi/site/priv>
        Require all denied
        XSendFile On
        XSendFilePath /var/www/wsgi/site/priv/jobfiles
    </Directory>

settings.py:

MIDDLEWARE_CLASSES += ('django_downloadview.SmartDownloadMiddleware', )
DOWNLOADVIEW_BACKEND = 'django_downloadview.apache.XSendfileMiddleware'
DOWNLOADVIEW_RULES = [
    {
        'source_url': '/priv/',
        'destination_dir': "/var/www/wsgi/site/priv",
    },
]

The view:

class DownloadJobResultsView(ObjectDownloadView):
    model = Job
    file_field = 'zipped_results'

~~I've been reading the changelogs from both downloadview an mod_xsenfile and it appears that downloadview might have been upgraded too soon ???~~

Below is the last update of xsendfile which is still on beta and not available in the repositories of Ubuntu or Fedora.

Version 1.0
    Unescape/url-decode header value to support non-ascii file names
    XSendFileUnescape setting, to support legacy applications
    X-SENDFILE-TEMPORARY header and corresponding AllowFileDelete flag
    Fix: Actually look into the backend-provided headers for Last-Modified

So, if I understand correctly, previous versions don't url-decode the file names...

EDIT: Just tried with xsendfile 1.0 built from source and still have the problem so... I have no idea of what is happening...

Response headers:

(Status-Line)       HTTP/1.1 200 OK
Date                Thu, 22 May 2014 20:16:48 GMT
Server              Apache/2.4.9 (Fedora) PHP/5.5.12 mod_wsgi/3.4 Python/2.7.5
Content-Disposition attachment; filename=%5Bpbs%2Bssh%40example.com%3A22%5D-%5B2067%5D.zip; filename*=UTF-8''%255Bpbs%252Bssh%2540example.com%253A22%255D-%255B2067%255D.zip
Content-Language    en
Vary                Accept-Language,Cookie
X-Frame-Options     SAMEORIGIN
X-Sendfile          /var/www/wsgi/site/priv/jobfiles/results/%5Bpbs%2Bssh%40example.com%3A22%5D-%5B2067%5D.zip
Content-Length      0
Keep-Alive          timeout=5, max=98
Connection          Keep-Alive
Content-Type        application/zip; charset=utf-8

Do you know how I could fix this?

Many thanks, Adrian

[benoitbryon]

This is my first time using both xsendfile and downloadview so I might be doing something wrong. I apologize if that is the case.

And I am not an expert of xsendfile... But let's try to find what's going wrong ;)

If I understood correctly, in your client, you request a dowload of [[email protected]:22]-[2067].zip file. What is the filename shown in your client? %5Bpbs%2Bssh%40example.com%3A22%5D-%5B2067%5D.zip?

Just to be sure, does your setup work fine with full-ascii filenames?

It looks like the anomaly is in the Content-Disposition attachment; filename=%5Bpbs%2Bssh%40example.com%3A22%5D-%5B2067%5D.zip; filename*=UTF-8''%255Bpbs%252Bssh%2540example.com%253A22%255D-%255B2067%255D.zip response header. filename seems urlencoded. And filename*UTF-8'' seems double urlencoded. It think it should be filename contains only ascii, and filename*UTF-8'' is urlencoded, because of the behaviour of django_downloadview.response.content_disposition(filename)

That said, at the moment I do not know what is the cause of the anomaly above... I think we could check 2 things:

• what does django_downloadview.response.content_disposition('[[email protected]:22]-[2067].zip') return?
• what is sent to django_downloadview.response.content_disposition with your setup? I mean, is the filename already urlencoded when it reaches django_downloadview? Or perhaps django_downloadview does double urlencoding in some way?

(note: I cannot promise I will investigate today...)

[benoitbryon]

>>> from django_downloadview import response
>>> response.content_disposition('[[email protected]:22]-[2067].zip')
"attachment; filename=[[email protected]:22]-[2067].zip; filename*=UTF-8''%5Bpbs%2Bssh%40example.com%3A22%5D-%5B2067%5D.zip"

If django_downloadview is given not-urlencoded filename, it should return the content-disposition as shown above.

Another thing that could happen is: django_downloadview returns the content-disposition above, and xsendfile urlencodes it again before sending it to the client. You can check what django_downloadview returns by deactivating xsendfile in apache configuration, then perform the request again and watch the response. Since xsendfile does not handle the response, you should see django_downloadview's raw "internal redirect" response, using x-sendfile headers.

[AdrianLC]

Thank you for answering so soon.

If I understood correctly, in your client, you request a dowload of [[email protected]:22]-[2067].zip file. What is the filename shown in your client? %5Bpbs%2Bssh%40example.com%3A22%5D-%5B2067%5D.zip?

Yes.

Just to be sure, does your setup work fine with full-ascii filenames?

Yes (for the file name, please see below).

what does django_downloadview.response.content_disposition('[[email protected]:22]-[2067].zip') return?

>>> from django_downloadview import response
>>> response.content_disposition('[[email protected]:22]-[2067].zip')
"attachment; filename=[[email protected]:22]-[2067].zip; filename*=UTF-8''%5Bpbs%2Bssh%40example.com%3A22%5D-%5B2067%5D.zip"

Another thing that could happen is: django_downloadview returns the content-disposition above, and xsendfile urlencodes it again before sending it to the client. You can check what django_downloadview returns by deactivating xsendfile in apache configuration, then perform the request again and watch the response. Since xsendfile does not handle the response, you should see django_downloadview's raw "internal redirect" response, using x-sendfile headers.

I was getting the same response even after restarting httpd with XSendfile Off and the LoadModule commented out. Turns out that my xsendfile was never enabled. I realized I was serving 0 byte files. Now, I moved the XSendFile On outside of the <Directory> and this is in my logs:

(2)No such file or directory: [client 127.0.0.1:53519] xsendfile: cannot open file: /var/www/wsgi/site/priv/jobfiles/results/%5Bpbs%2Bssh%40example.com%3A22%5D-%5B2067%5D.zip, referer: http://localhost/execution/list/

so... perhaps my original assumption (that xsendfile pre-1.0 doesn't url-decode) was right after all ??? Because the file is there for sure.

EDIT: I tried again with xsendfile 1.0. It serves the file with the correct size but still the wrong name. Also, with XSendFileUnescape off (which supposedly restores <1.0 behaviour) it'll throw the same "No such file" error.

[AdrianLC]

Sorry, I closed it by mistake.

[AdrianLC]

Think I've found where the double encode is happening.

FileSystemStorage's url() method escapes the filename with django.utils.encoding.filepath_to_uri. The method is called in ProxiedDownloadMiddleware through the url property of FieldFile.

[pkaczynski]

#135 Will this fix this issue?