[Feature Request] Allow Non-ASCII characters in identifier

[constfold]

While LuaJIT can handle Non-ASCII characters correctly(http://luajit.org/extensions.html), Prometheus was only accept ASCII characters. It is inconvenient to work with many domain specific scripts that may contain a lot Russian/Chinese identifiers.

For example, this script is totally valid in LuaJIT but unable to be obfuscated by Prometheus:

Привет = 1
print(Привет)

Error while parsing Non-ASCII characters:

PROMETHEUS: Applying Obfuscation Pipeline to test.lua ...
PROMETHEUS: Parsing ...
PROMETHEUS: Lexing Error at Position 15:26, Unexpected char "\178"!
luajit: ../Prometheus/src\logger.lua:54: Lexing Error at Position 15:26, Unexpected char "\178"! 
stack traceback:
        [C]: in function 'error'
        ../Prometheus/src\logger.lua:54: in function 'errorCallback'
        ../Prometheus/src\logger.lua:57: in function 'error'
        ../Prometheus/src\prometheus\tokenizer.lua:512: in function 'next'
        ../Prometheus/src\prometheus\tokenizer.lua:518: in function 'scanAll'
        ../Prometheus/src\prometheus\parser.lua:143: in function 'parse'
        ../Prometheus/src\prometheus\pipeline.lua:178: in function 'apply'
        ../Prometheus/src\cli.lua:132: in main chunk
        [C]: in function 'require'
        ../Prometheus/cli.lua:12: in main chunk
        [C]: at 0x005f31a0

Probably related to: https://github.com/levno-710/Prometheus/blob/e639587547077bab61047712720d88722d0c5fbd/src/prometheus/enums.lua#L33

[pnlmon]

What we could do instead is to convert non-ascii variable name to an ascii one, then convert all of them back in the output

[Brohammer5]

I think this could be a good addition. However this doesn't currently support LuaJIT but Lua51 and some of Roblox's LuaU.

[constfold]

this doesn't currently support LuaJIT but Lua51

AFAIK, there are not much differences between LuaJIT(without lua52 lang extensions) and Lua51 in syntax.

And, furthermore, in fact the lua51 language specification has never defined what exactly the identifiers could be as it describes identifiers as: Names (also called identifiers) in Lua can be any string of letters, digits, and underscores, not beginning with a digit and not being a reserved word. See https://stackoverflow.com/questions/55671742/why-is-the-following-piece-of-lua-code-completely-valid.

Actually, identifiers already could be unicode in *NIX in UTF-8 locale without any modification to lua, accroding to UnicodeIdentifers in lua-users.

[QAQQL]

Me, too. I have some functions and variables named in Chinese, which have no influence on luajit, but I get an error when I use Prometheus to confuse them.

cmd => lua ./cli.lua --nocolors --preset Medium ./tests.lua --out ./tests2.lua

PROMETHEUS: Applying Obfuscation Pipeline to ./tests.lua ...
PROMETHEUS: Parsing ...
PROMETHEUS: Lexing Error at Position 10:12, Unexpected char "\232"!
D:\Env\Lua5.1\lua.exe: ./src\logger.lua:54: Lexing Error at Position 10:12, Unexpected char "\232"!
stack traceback:
        [C]: in function 'error'
        ./src\logger.lua:54: in function 'errorCallback'
        ./src\logger.lua:57: in function 'error'
        ./src\prometheus\tokenizer.lua:534: in function 'next'
        ./src\prometheus\tokenizer.lua:540: in function 'scanAll'
        ./src\prometheus\parser.lua:143: in function 'parse'
        ./src\prometheus\pipeline.lua:178: in function 'apply'
        ./src\cli.lua:148: in main chunk
        [C]: in function 'require'
        ./cli.lua:12: in main chunk
        [C]: ?

-- This Script is Part of the Prometheus Obfuscator by Levno_710
--
-- test.lua
-- This file will Perform tests using all lua files within the tests directory


-- Enable Debugging
-- logger.logLevel = logger.LogLevel.Debug;
-- 测试
function QL.软件_初始化(参数1, 参数2)
    print("软件_初始化", 参数1, 参数2)
end

-- Config Variables - Later passed as Parameters
local noColors  = false;   -- Wether Colors in the Console output should be enabled
local isWindows = true;    -- Wether the Test are Performed on a Windows or Linux System
local ciMode    = false;   -- Wether the Test error are ignored or not

How can I fix this ?