CLI QuickTextRender output utf-8

[cam0200]

Volatility would running into an error when pipping Unicode output in Powershell (whether writing to file or terminal)

POC command: python .\vol.py -f C:\<path>\memdump.dmp windows.filescan.FileScan | Select-String -Pattern '\.\w{2}\b'

Volatility 3 Framework 2.26.2
Traceback (most recent call last):
  File "C:\tools\volatility3\vol.py", line 11, in <module>
    volatility3.cli.main()
    ~~~~~~~~~~~~~~~~~~~~^^
  File "C:\tools\volatility3\volatility3\cli\__init__.py", line 927, in main
    CommandLine().run()
    ~~~~~~~~~~~~~~~~~^^
  File "C:\tools\volatility3\volatility3\cli\__init__.py", line 515, in run
    renderer.render(grid)
    ~~~~~~~~~~~~~~~^^^^^^
  File "C:\tools\volatility3\volatility3\cli\text_renderer.py", line 330, in render
    grid.populate(visitor, outfd)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "C:\tools\volatility3\volatility3\framework\renderers\__init__.py", line 323, in populate
    accumulator = function(treenode, accumulator)
  File "C:\tools\volatility3\volatility3\cli\text_renderer.py", line 325, in visitor
    accumulator.write("{}".format("\t".join(line)))
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python313\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode characters in position 15-22: character maps to <undefined>

The change that is implemented will force output text to be utf-8 encoded.

[cam0200]

Looks like similar issue was reported here: https://github.com/volatilityfoundation/volatility3/issues/1754

[ikelos]

This appears to be python/windows outputting in codepage 1252 for some reason (I don't know if that's the default for python on windows, or information passed by the terminal, etc):

File "C:\Python313\Lib\encodings\cp1252.py", line 19, in encode

I believe volatility already outputs at utf-8, so I'd prefer to get more clarity as to exactly what's going wrong (is it volatility outputting a bad character, or is it python misrepresenting the output stream to the terminal, etc)... Otherwise my concern is that the errors='replace' may affect information that's relied on for forensic purposes.

A good test would be to pipe the output to a file and verify whether the file is all valid utf-8 or not. If it is, then there's some other issue somewhere is the connection between the terminal and python, if not we can take a look at which character is invalid and see if we can figure out how it's happening...

[cam0200]

This is an example of attempting to output to a file, which results in the same issue

(venv) PS C:\tools\volatility3> python .\vol.py -f C:\<path>\memdump.dmp windows.filescan.FileScan >> C:\<path>\test-out.tsv
Traceback (most recent call last):B scanning finished
  File "C:\tools\volatility3\vol.py", line 11, in <module>
    volatility3.cli.main()
    ~~~~~~~~~~~~~~~~~~~~^^
  File "C:\tools\volatility3\volatility3\cli\__init__.py", line 927, in main
    CommandLine().run()
    ~~~~~~~~~~~~~~~~~^^
  File "C:\tools\volatility3\volatility3\cli\__init__.py", line 515, in run
    renderer.render(grid)
    ~~~~~~~~~~~~~~~^^^^^^
  File "C:\tools\volatility3\volatility3\cli\text_renderer.py", line 330, in render
    grid.populate(visitor, outfd)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "C:\tools\volatility3\volatility3\framework\renderers\__init__.py", line 323, in populate
    accumulator = function(treenode, accumulator)
  File "C:\tools\volatility3\volatility3\cli\text_renderer.py", line 325, in visitor
    accumulator.write("{}".format("\t".join(line)))
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python313\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode characters in position 15-22: character maps to <undefined>

The point that it breaks at appears to be this output from windows.filescan.FileScan plugin 쮭蔌쮭蔌

[dotslashed]

Hi I faced the same Unicode error issue in Powershell. This issue occurred during piping or redirecting volatility output to other commands when output contains some unique kind of characters. I spent some time with chatGPT, and it gave me a solution which might be temporary it but worked. You can try and check if this works for you as well. Tested on Volatility3 Development version

Add this to the very top of (volatility3\__init__.py) just before starting of the code part:

import io
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding="utf-8", errors="replace")

Save the file and run your Volatility command.

[dotslashed]

Hi I faced the same Unicode error issue in Powershell. This issue occurred during piping or redirecting volatility output to other commands when output contains some unique kind of characters. I spent some time with chatGPT, and it gave me a solution which might be temporary it but worked. You can try and check if this works for you as well. Tested on Volatility3 Development version

Add this to the very top of (volatility3\__init__.py) just before starting of the code part:

import io
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding="utf-8", errors="replace")

Save the file and run your Volatility command.

This solution is already posted by @cam0200 in merge request