Garbaled text file instead of robots.txt in browser

[arayate]

This package is configured properly on my local machine. However when I pushed it on our staging server (apache + uwsgi), it is behaving weirdly.

When I go to https://test.mysite.com/robots.txt, it actually downloads the file of same name(robots.txt), but I cannot open it in gedit. cat command shows garbled texts in it.

Googled about this but couldn't find specific info.

What could be the problem here?

[yakky]

Looks like an encoding issue. Could you check through the inspector the file encoding when downloading from apache?

[arayate]

See this screenshot. This is what I'm getting.

[arayate]

Used this library to find out encoding.

https://github.com/BYVoid/uchardet

It shows ascii/unknown.

[arayate]

Through curl, it looks OK. I don't know what is the problem with browser.

I tried logging response objects at each middleware and all looked ok. Even in curl, it looks ok. But in browser the response looks garbled.

I tried this command to dump the headers through curl. curl -s -D - http://mysite.com/robots.txt /dev/null

Response Header:

HTTP/1.1 200 OK
Vary: Cookie
X-Frame-Options: SAMEORIGIN
Content-Type: text/plain

Above response also looks ok to me. But in browser it is garbled. I wondor whether crawler-bots will be able to parse this robots.txt file

[yakky]

@arayate how about the response dump from the browser? 99% there is some error in the browser-server content negotiation, outside the scope of robots application. If curl looks good, I bet bots will get the right file from the server

[some1ataplace]

There are a few possible reasons for this issue:

1. Misconfiguration of the Apache server or uWSGI: Check your Apache and uWSGI configuration files to ensure you're serving the correct robots.txt file and routing requests properly. You might want to look at the Alias and RewriteRule directives in your Apache configuration file as they are used to define paths for static files. For uWSGI, ensure that the correct application is being served.

2. Improper file encoding: This is less likely since you mentioned the file shows ASCII/unknown encoding, but it's still worth checking. Make sure your robots.txt file is saved with the proper encoding, such as UTF-8.

3. Server/Operating system-specific issues: Some operating systems or configurations might cause issues when reading or serving text files. Ensure your server is up-to-date and configured correctly.

It's also important to ensure that your django-robots configuration is correct in your Django settings. Double-check that the ROBOTS_USE_SITEMAP and other settings are set correctly, and that the package is included in your INSTALLED_APPS. If you have not done so already, also make sure to restart your Apache and uWSGI services after changes have been made.

As a side note, consider testing your setup with a different browser and/or incognito mode to rule out any browser cache or session-specific issues.

---

It's possible that the issue you are experiencing with the robots.txt file being garbled is related to an incorrect server or file encoding. Since the Content-Type of the file is set to text/plain, it's likely that the encoding used on the server is not compatible with the encoding used by your local machine.

To fix this issue, you should try changing the encoding of the file to a compatible encoding, such as UTF-8. You could also try adding an Accept-Encoding header to your curl command to specify an encoding that is compatible with your local machine.

Here's an example of how to change the encoding of the robots.txt file to UTF-8 in Django:

1. Open the robots/views.py file in the django-robots package.

2. Add the following import at the top of the file:

   from django.http import HttpResponse

3. Modify the RobotsView class to return an HttpResponse object with the appropriate encoding:

 class RobotsView(View):
       def get(self, request, *args, **kwargs):
           content = render_to_string('robots/robots.txt', {'url': get_current_site(request)})
           # Set the encoding of the file to UTF-8
           response = HttpResponse(content, content_type='text/plain; charset=utf-8')
           return response

This code will render the robots.txt template and return an HttpResponse object with the appropriate encoding set to UTF-8.

Another option is to check the encoding of the robots.txt file on the server and ensure that it is compatible with the encoding used by your local machine. You could also try using a different text editor or tool to download the file and see if that resolves the issue.