Use REDASH_CSV_WRITER_ENCODING for downloading CSV

[kaibadash]

What type of PR is this?

• [ ] Refactor
• [ ] Feature
• [x] Bug Fix
• [ ] New Query Runner (Data Source)
• [ ] New Alert Destination
• [ ] Other

Description

REDASH_CSV_WRITER_ENCODING environment variable is ignored and always downloads as UTF-8. We, engineers may not understand what the problem is that.

The administrative staffs in Japan like to edit the CSV directly in Excel and upload it to another system. Excel needs to be encoded in CP932(Shift-JIS) to edit it directly, so we need to be able to set the character encoding.

I think all be solved if Excel supported UTF-8, but Microsoft never do that and Japanese administrative staffs can only use Excel... 😫

How is this tested?

• [ ] Unit tests (pytest, jest)
• [ ] E2E Tests (Cypress)
• [x] Manually
• [ ] N/A

Related Tickets & Documents

Sorry for the Japanese documentation. There are people who are having the same problem as I am. https://qiita.com/koike_moyashi/items/d0d5dc37a93b398f0aaa

Mobile & Desktop Screenshots/Recordings (if there are UI changes)

[guidopetri]

@kaibadash thank you for your contribution! Would you mind resolving the merge conflicts, as well as adding the env var in the documentation here?

[kaibadash]

Thank you for reviewing. I resolved conflicts.

adding the env var in the documentation here?

I created a pull request. https://github.com/getredash/website/pull/679

[konnectr]

@kaibadash Could you add tests ?

[konnectr]

You need to run make format

[codecov[bot]]

Codecov Report

Merging #5826 (2286624) into master (6b98197) will increase coverage by 0.00%. The diff coverage is 33.33%.

@@           Coverage Diff           @@
##           master    #5826   +/-   ##
=======================================
  Coverage   61.26%   61.26%           
=======================================
  Files         158      158           
  Lines       12889    12890    +1     
  Branches     1755     1755           
=======================================
+ Hits         7896     7897    +1     
  Misses       4743     4743           
  Partials      250      250           

Files	Coverage Δ
redash/settings/__init__.py	98.66% <100.00%> (+<0.01%)	:arrow_up:
redash/handlers/query_results.py	81.57% <0.00%> (ø)

[kaibadash]

I tried to add tests in test_query_results.py . I'm not a pythonista, that was difficult for me... Could someone help me?

class TestMakeCsvResponse(BaseTestCase):
    # TODO: How can I mock CSV_WRITER_ENCODING ?
    def test_make_csv_response_with_cp932_encoding(self):
        # TODO: How can I get query_result with multi byte string?
        query_result = self.factory.create_query_result()
        response = self.make_csv_response(query_result)
        # TODO: How can I get result ?
        self.assertEqual(response["TODO"], "こんにちは means hello in Japanese".encode("cp932"))

[justinclift]

@guidopetri Do you have time to help out here? :smile: