Cant pipe the out for File Scan or MFT scan

[TomGeek18]

Hello,

Vol 2.03, Win 10, Python 3.11, We are working with a Win 10 image using window 10 host. Vol3 works great, all the other plug ins worked great. When attempting to pipe the output of the MFT or File scan to a text file it crashes.
blah.blah windows.mftscan >txt.txt = crash blah blah windows.filescan >123.txt =crash

If you let the plugin output to the screen it will run and not crash.

[digitalisx]

Hello @TomGeek18

~~To solve the problem simply, I recommend lowering the Python version to a version between 3.6 and 3.10.~~

~~Python 3.11 that you told me is a new version that we haven't experienced enough in Alpha stage yet. It is probably a version that has not been validated for use with Volatility3.~~

~~Therefore, it will not be easy to solve issues quickly until Python 3.11 is stable enough and familiar to us.~~

If you think another problem, it would be better for community experts to help if you could provide dump file or more detailed logs regarding the process of generating files through the pipeline.

[digitalisx]

I've done it in a similar environment on my PC, but I've confirmed that it's going well. I'm sorry, but contrary to my expectation, it seems that it's not a Python version problem.

I think we need to get a detailed log of crash or dump file to solve this issue.

Python Version

(.venv)  donghyunkim > python3.11
Python 3.11.0a7 (v3.11.0a7:2e49bd06c5, Apr  5 2022, 23:34:36) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

MFTScan

(.venv)  donghyunkim > python3.11 vol.py -f case.vmem windows.mftscan > py3test.txt
Progress:  100.00   PDB scanning finished

(.venv)  donghyunkim > cat py3test.txt | head -10
Volatility 3 Framework 2.0.3

Offset	Record Type	Record Number	Link Count	MFT Type	Permissions	Attribute Type	Created	Modified	Updated	Accessed	Filename

0x8000dc4a3050	FILE	68284	2	File	N/A	STANDARD_INFORMATION	2019-12-07 09:06:37.000000 	2019-12-07 09:06:37.000000 	2021-07-14 16:49:43.000000 	2019-12-07 09:06:37.000000 	N/A
* 0x8000dc4a30b0	FILE	68284	2	File	Archive	FILE_NAME	2021-07-14 16:49:43.000000 	2021-07-14 16:49:43.000000 	2021-07-14 16:49:43.000000 	2021-07-14 16:49:43.000000 	AMB767~2.MAN
* 0x8000dc4a3128	FILE	68284	2	File	Archive	FILE_NAME	2021-07-14 16:49:43.000000 	2021-07-14 16:49:43.000000 	2021-07-14 16:49:43.000000 	2021-07-14 16:49:43.000000 	amd64_microsoft-wind

Filescan

(.venv)  donghyunkim > python3.11 vol.py -f case.vmem windows.filescan > py3test.txt
Progress:  100.00   PDB scanning finished

(.venv)  donghyunkim > cat py3test.txt | head -10
Volatility 3 Framework 2.0.3

Offset	Name	Size

0x97065ac760c0	\Windows\System32\taskhostw.exe	216
0x97065ac76250	\Windows\System32\ko-KR\KernelBase.dll.mui	216
0x97065ac763e0	\Windows\WinSxS\Manifests\amd64_microsoft.windows.systemcompatible_6595b64144ccf1df_6.0.19041.1_none_bcf22701031bcbf3.manifest	216
0x97065ac76570	\Windows\System32\sysmain.dll	216
0x97065ac76890	\Windows\System32\winevt\Logs\Microsoft-Windows-NCSI%4Operational.evtx	216
0x97065ac76ed0	\Windows\System32\InputService.dll	216

[ikelos]

It would be useful to understand what makes you think it's a crash (by getting the command line you're using and the contents of output file you're writing to, making sure to write to the same directory to ensure it's not a path issue).

[TomGeek18]

volatility3>python vol.py -f x:\memdump.mem windows.mftscan >txt.txt Traceback (most recent call last):B scanning finished \volatility3\vol.py", line 10, in volatility3.cli.main() volatility3\volatility3\cli_init_.py", line 635, in main CommandLine().run() \volatility3\volatility3\cli_init_.py", line 342, in run renderersargs.renderer.render(constructed.run()) \volatility3\volatility3\cli\text_renderer.py", line 177, in render grid.populate(visitor, outfd) \volatility3\volatility3\framework\renderers_init_.py", line 218, in populate accumulator = function(treenode, accumulator) volatility3\volatility3\cli\text_renderer.py", line 172, in visitor accumulator.write("{}".format("\t".join(line))) \Local\Programs\Python\Python310\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode characters in position 87-101: character maps to

[ikelos]

This sounds as though the merminal that you're trying to output to doesn't support the full unicode character set. It's unclear why that wouldn't pipe to a file without issue, but I'd recommend investigating changing you terminal's default character map. There are usually character maps that support the entire range of code pages. Some references that may help:

https://github.com/volatilityfoundation/volatility3/issues/674#issuecomment-1066686669 https://github.com/volatilityfoundation/volatility3/issues/514#issuecomment-913119962 https://github.com/volatilityfoundation/volatility3/issues/637#issuecomment-1034406379

Please let us know how you get on...

[TomGeek18]

Good Morning.

This is the results I get when I run the MFTscan and attempt to send to a file. The MFTscan will run fine without sending to a file, but will prove the below error if I >.

volatility3>python vol.py -f x:\memdump.mem windows.mftscan >txt.txt Traceback (most recent call last):B scanning finished \volatility3\vol.py", line 10, in volatility3.cli.main() volatility3\volatility3\cli_init_.py", line 635, in main CommandLine().run() \volatility3\volatility3\cli_init_.py", line 342, in run renderersargs.renderer.render(constructed.run()) \volatility3\volatility3\cli\text_renderer.py", line 177, in render grid.populate(visitor, outfd) \volatility3\volatility3\framework\renderers_init_.py", line 218, in populate accumulator = function(treenode, accumulator) volatility3\volatility3\cli\text_renderer.py", line 172, in visitor accumulator.write("{}".format("\t".join(line))) \Local\Programs\Python\Python310\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode characters in position 87-101: character maps to

On Sun, Apr 24, 2022 at 6:44 AM ikelos @.***> wrote:

It would be useful to understand what makes you think it's a crash (by getting the command line you're using and the contents of output file you're writing to, making sure to write to the same directory to ensure it's not a path issue).

— Reply to this email directly, view it on GitHub https://github.com/volatilityfoundation/volatility3/issues/707#issuecomment-1107814367, or unsubscribe https://github.com/notifications/unsubscribe-auth/AM5K27PDAUVYH3JPEDODBCTVGUQX7ANCNFSM5UDBNDEQ . You are receiving this because you were mentioned.Message ID: @.***>

[ikelos]

Hi there, this appears to be an encoding error in the output of volatility and may be related to the character map of the console that python is running within. You could try setting the following environment variables to see if that improves the situation:

set PYTHONIOENCODING=utf-8
set PYTHONLEGACYWINDOWSSTDIO=utf-8

This is usually encountered because the windows terminal is one of the last terminals that does not support unicode UTF-8 output (which is what volatility will output. The error message shows that volatility was trying to output a character that could not be encoded into "cp1252" which is the windows default character map. It appears even though this should be piping to a file, it's being processed by the terminal. See https://stackoverflow.com/a/72733078 for more of a description.

Code pages are tricky, but as far as I can tell volatility is outputting the correct data in the correct encoding and it's the windows terminal that is saying it can't properly handle full unicode and therefore causes the exception to be thrown. I hope this helps?

[github-actions[bot]]

This issue is stale because it has been open for 200 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 60 days since being marked as stale.