Software cannot read files with Chinese characters normally

[AlniyatYang]

I have a project code folder about the single-chip microcomputer, which are all C language, where if there exsits comment with Chinese charater , .c or .h will not be displayed in the generated markdown file inside, when I delete all the Chinese comments, and then generate, the code will be displayed

[ODAncona]

Hi @AlniyatYang, I tried to reproduce files with Chinese characters and it seems to work in the latest version (potentially since #71). Can you confirm it works on your side ? Thank you

[lilimu996]

I installed the latest version using cargo on April 25th and encountered the same issue in a single-chip microcomputer project. When the project's encoding format is GB2132, it is necessary to remove all Chinese comments for it to work properly.

[ODAncona]

Hello @lilimu996 Thanks for reproducing the issue.

This is interesting to see that code2prompt struggles with specific encoding.

It should be investigated as many legacy codebases would have special encoding.

It could be nice to write a general encoding implementation 🚀