Add support to Hermes version 83, 89 and improve contribution process

[bongtrop]

• Version support added 84 (via niosega), 85,89,83
• Test cases are now running some fixing was needed
• Added a readme on how to add new versions.

Very thank for https://github.com/cyfinoid @anantshri

[zrthstr]

Helo @bongtrop and @anantshri,

is there any way i can help with this?

Is it possible that https://github.com/cyfinoid/hbctool is still not fully working yet? When trying to use the cyfinoid version to disassemble 84'er bytecode, i get an IndexError: list index out of range form line 33 in hbc/hbc84/translator.py.

It appears bc[i] is 271 but len(opcode_mapper) only is 201.

Happy to know how i can help.

[gonzalo-hvega]

I am having issues too, trying to decompile an index.android.bundle which contains HBC Version 89, however I get the following error while it tries to unpack the instruction.hasm:

Traceback (most recent call last): File "runpy.py", line 193, in run_module_as_main File "runpy.py", line 85, in run_code File "D:\Programs\Python-3.6.8\Scripts\hbctool.exe_main.py", line 7, in File "d:\programs\python-3.6.8\lib\site-packages\hbctool_init.py", line 61, in main disasm(args['<HBC_FILE>'], args['<HASM_PATH>']) File "d:\programs\python-3.6.8\lib\site-packages\hbctool_init_.py", line 41, in disasm hasm.dump(hbco, hasmpath) File "d:\programs\python-3.6.8\lib\site-packages\hbctool\hasm.py", line 67, in dump write_func(f, hbc.getFunction(i), i, hbc) File "d:\programs\python-3.6.8\lib\site-packages\hbctool\hbc\hbc89_init_.py", line 59, in getFunction insts = disassemble(bc) File "d:\programs\python-3.6.8\lib\site-packages\hbctool\hbc\hbc89\translator.py", line 43, in disassemble val = conv_to(bc[i:i+size]) File "d:\programs\python-3.6.8\lib\site-packages\hbctool\util.py", line 300, in to_uint16 return unpack("<H", bytes(buf[:2]))[0] struct.error: unpack requires a buffer of 2 bytes

An updates? Happy to run against a previous version and output some debugging

[gonzalo-hvega]

OK so I figured this out, for version 89 at least. Turns out the structure.json in the data folder was very outdated, I wet through and manually updated it by looking at the ByteCodeFileFormat.h from the raw folder. Also the Parser needed to account for BigInts, FunctionSourceTable, etc..

I believe similar updates will be required for version 84, and so on, to bring them up to date with the correct file format.

Will look to create a pull request for hbc89 files, and someone else can use that as a basis to update other version.

Thanks for all those that contributed to this!

[BaseMax]

Thank you for your work. When it will be merged?

[BaseMax]

C:\Users\Max\hbctool\hbctool>python __init__.py disasm index.android.bundle out
Hello!
running disasm
[*] Disassemble 'index.android.bundle' to 'out' path
index.android.bundle
Load function
{85: <class 'hbctool.hbc.hbc85.HBC85'>, 84: <class 'hbctool.hbc.hbc84.HBC84'>, 76: <class 'hbctool.hbc.hbc76.HBC76'>, 74: <class 'hbctool.hbc.hbc74.HBC74'>, 62: <class 'hbctool.hbc.hbc62.HBC62'>, 59: <class 'hbctool.hbc.hbc59.HBC59'>}
Hey!

<hbctool.hbc.hbc84.HBC84 object at 0x0000025DDC01A020>
{'magic': 2240826417119764422, 'version': 84, 'sourceHash': [174, 232, 201, 88, 175, 39, 233, 36, 198, 59, 202, 82, 137, 60, 18, 124, 38, 85, 58, 154], 'fileLength': 10231884, 'globalCodeIndex': 0, 'functionCount': 37775, 'stringKindCount': 4, 'identifierCount': 32271, 'stringCount': 75979, 'overflowStringCount': 758, 'stringStorageSize': 3166346, 'regExpCount': 713, 'regExpStorageSize': 98501, 'arrayBufferSize': 225217, 'objKeyBufferSize': 38259, 'objValueBufferSize': 120303, 'cjsModuleOffset': 0, 'cjsModuleCount': 0, 'debugInfoOffset': 0, 'option': 36, 'padding': [32, 156, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}
84
[*] Hermes Bytecode [ Source Hash: aee8c958af27e924c63bca52893c127c26553a9a, HBC Version: 84 ]
Traceback (most recent call last):
  File "C:\Users\Max\hbctool\hbctool\__init__.py", line 81, in <module>
    main()
  File "C:\Users\Max\hbctool\hbctool\__init__.py", line 69, in main
    disasm(args['<HBC_FILE>'], args['<HASM_PATH>'])
  File "C:\Users\Max\hbctool\hbctool\__init__.py", line 48, in disasm
    hasm.dump(hbco, hasmpath)
  File "C:\Users\Max\AppData\Local\Programs\Python\Python310\lib\site-packages\hbctool\hasm.py", line 67, in dump
    write_func(f, hbc.getFunction(i), i, hbc)
  File "C:\Users\Max\AppData\Local\Programs\Python\Python310\lib\site-packages\hbctool\hbc\hbc84\__init__.py", line 59, in getFunction
    insts = disassemble(bc)
  File "C:\Users\Max\AppData\Local\Programs\Python\Python310\lib\site-packages\hbctool\hbc\hbc84\translator.py", line 33, in disassemble
    opcode = opcode_mapper[bc[i]]
IndexError: list index out of range

[BaseMax]

['Unreachable', 'NewObjectWithBuffer', 'NewObjectWithBufferLong', 'NewObject', 'NewObjectWithParent', 'NewArrayWithBuffer', 'NewArrayWithBufferLong', 'NewArray', 'Mov', 'MovLong', 'Negate', 'Not', 'BitNot', 'TypeOf', 'Eq', 'StrictEq', 'Neq', 'StrictNeq', 'Less', 'LessEq', 'Greater', 'GreaterEq', 'Add', 'AddN', 'Mul', 'MulN', 'Div', 'DivN', 'Mod', 'Sub', 'SubN', 'LShift', 'RShift', 'URshift', 'BitAnd', 'BitXor', 'BitOr', 'Inc', 'Dec', 'InstanceOf', 'IsIn', 'GetEnvironment', 'StoreToEnvironment', 'StoreToEnvironmentL', 'StoreNPToEnvironment', 'StoreNPToEnvironmentL', 'LoadFromEnvironment', 'LoadFromEnvironmentL', 'GetGlobalObject', 'GetNewTarget', 'CreateEnvironment', 'DeclareGlobalVar', 'GetByIdShort', 'GetById', 'GetByIdLong', 'TryGetById', 'TryGetByIdLong', 'PutById', 'PutByIdLong', 'TryPutById', 'TryPutByIdLong', 'PutNewOwnByIdShort', 'PutNewOwnById', 'PutNewOwnByIdLong', 'PutNewOwnNEById', 'PutNewOwnNEByIdLong', 'PutOwnByIndex', 'PutOwnByIndexL', 'PutOwnByVal', 'DelById', 'DelByIdLong', 'GetByVal', 'PutByVal', 'DelByVal', 'PutOwnGetterSetterByVal', 'GetPNameList', 'GetNextPName', 'Call', 'Construct', 'Call1', 'CallDirect', 'Call2', 'Call3', 'Call4', 'CallLong', 'ConstructLong', 'CallDirectLongIndex', 'CallBuiltin', 'CallBuiltinLong', 'GetBuiltinClosure', 'Ret', 'Catch', 'DirectEval', 'Throw', 'ThrowIfEmpty', 'Debugger', 'AsyncBreakCheck', 'ProfilePoint', 'CreateClosure', 'CreateClosureLongIndex', 'CreateGeneratorClosure', 'CreateGeneratorClosureLongIndex', 'CreateAsyncClosure', 'CreateAsyncClosureLongIndex', 'CreateThis', 'SelectObject', 'LoadParam', 'LoadParamLong', 'LoadConstUInt8', 'LoadConstInt', 'LoadConstDouble', 'LoadConstString', 'LoadConstStringLongIndex', 'LoadConstEmpty', 'LoadConstUndefined', 'LoadConstNull', 'LoadConstTrue', 'LoadConstFalse', 'LoadConstZero', 'CoerceThisNS', 'LoadThisNS', 'ToNumber', 'ToInt32', 'AddEmptyString', 'GetArgumentsPropByVal', 'GetArgumentsLength', 'ReifyArguments', 'CreateRegExp', 'SwitchImm', 'StartGenerator', 'ResumeGenerator', 'CompleteGenerator', 'CreateGenerator', 'CreateGeneratorLongIndex', 'IteratorBegin', 'IteratorNext', 'IteratorClose', 'Jmp', 'JmpLong', 'JmpTrue', 'JmpTrueLong', 'JmpFalse', 'JmpFalseLong', 'JmpUndefined', 'JmpUndefinedLong', 'SaveGenerator', 'SaveGeneratorLong', 'JLess', 'JLessLong', 'JNotLess', 'JNotLessLong', 'JLessN', 'JLessNLong', 'JNotLessN', 'JNotLessNLong', 'JLessEqual', 'JLessEqualLong', 'JNotLessEqual', 'JNotLessEqualLong', 'JLessEqualN', 'JLessEqualNLong', 'JNotLessEqualN', 'JNotLessEqualNLong', 'JGreater', 'JGreaterLong', 'JNotGreater', 'JNotGreaterLong', 'JGreaterN', 'JGreaterNLong', 'JNotGreaterN', 'JNotGreaterNLong', 'JGreaterEqual', 'JGreaterEqualLong', 'JNotGreaterEqual', 'JNotGreaterEqualLong', 'JGreaterEqualN', 'JGreaterEqualNLong', 'JNotGreaterEqualN', 'JNotGreaterEqualNLong', 'JEqual', 'JEqualLong', 'JNotEqual', 'JNotEqualLong', 'JStrictEqual', 'JStrictEqualLong', 'JStrictNotEqual', 'JStrictNotEqualLong', 'Add32', 'Sub32', 'Mul32', 'Divi32', 'Divu32', 'Loadi8', 'Loadu8', 'Loadi16', 'Loadu16', 'Loadi32', 'Loadu32', 'Store8', 'Store16', 'Store32']
len of opcode_mapper: 201

And the error:

    opcode = opcode_mapper[bc[i]]
IndexError: list index out of range

It looking for 204 in the opcode.json and cannot find it.

[hab12335]

['Unreachable', 'NewObjectWithBuffer', 'NewObjectWithBufferLong', 'NewObject', 'NewObjectWithParent', 'NewArrayWithBuffer', 'NewArrayWithBufferLong', 'NewArray', 'Mov', 'MovLong', 'Negate', 'Not', 'BitNot', 'TypeOf', 'Eq', 'StrictEq', 'Neq', 'StrictNeq', 'Less', 'LessEq', 'Greater', 'GreaterEq', 'Add', 'AddN', 'Mul', 'MulN', 'Div', 'DivN', 'Mod', 'Sub', 'SubN', 'LShift', 'RShift', 'URshift', 'BitAnd', 'BitXor', 'BitOr', 'Inc', 'Dec', 'InstanceOf', 'IsIn', 'GetEnvironment', 'StoreToEnvironment', 'StoreToEnvironmentL', 'StoreNPToEnvironment', 'StoreNPToEnvironmentL', 'LoadFromEnvironment', 'LoadFromEnvironmentL', 'GetGlobalObject', 'GetNewTarget', 'CreateEnvironment', 'DeclareGlobalVar', 'GetByIdShort', 'GetById', 'GetByIdLong', 'TryGetById', 'TryGetByIdLong', 'PutById', 'PutByIdLong', 'TryPutById', 'TryPutByIdLong', 'PutNewOwnByIdShort', 'PutNewOwnById', 'PutNewOwnByIdLong', 'PutNewOwnNEById', 'PutNewOwnNEByIdLong', 'PutOwnByIndex', 'PutOwnByIndexL', 'PutOwnByVal', 'DelById', 'DelByIdLong', 'GetByVal', 'PutByVal', 'DelByVal', 'PutOwnGetterSetterByVal', 'GetPNameList', 'GetNextPName', 'Call', 'Construct', 'Call1', 'CallDirect', 'Call2', 'Call3', 'Call4', 'CallLong', 'ConstructLong', 'CallDirectLongIndex', 'CallBuiltin', 'CallBuiltinLong', 'GetBuiltinClosure', 'Ret', 'Catch', 'DirectEval', 'Throw', 'ThrowIfEmpty', 'Debugger', 'AsyncBreakCheck', 'ProfilePoint', 'CreateClosure', 'CreateClosureLongIndex', 'CreateGeneratorClosure', 'CreateGeneratorClosureLongIndex', 'CreateAsyncClosure', 'CreateAsyncClosureLongIndex', 'CreateThis', 'SelectObject', 'LoadParam', 'LoadParamLong', 'LoadConstUInt8', 'LoadConstInt', 'LoadConstDouble', 'LoadConstString', 'LoadConstStringLongIndex', 'LoadConstEmpty', 'LoadConstUndefined', 'LoadConstNull', 'LoadConstTrue', 'LoadConstFalse', 'LoadConstZero', 'CoerceThisNS', 'LoadThisNS', 'ToNumber', 'ToInt32', 'AddEmptyString', 'GetArgumentsPropByVal', 'GetArgumentsLength', 'ReifyArguments', 'CreateRegExp', 'SwitchImm', 'StartGenerator', 'ResumeGenerator', 'CompleteGenerator', 'CreateGenerator', 'CreateGeneratorLongIndex', 'IteratorBegin', 'IteratorNext', 'IteratorClose', 'Jmp', 'JmpLong', 'JmpTrue', 'JmpTrueLong', 'JmpFalse', 'JmpFalseLong', 'JmpUndefined', 'JmpUndefinedLong', 'SaveGenerator', 'SaveGeneratorLong', 'JLess', 'JLessLong', 'JNotLess', 'JNotLessLong', 'JLessN', 'JLessNLong', 'JNotLessN', 'JNotLessNLong', 'JLessEqual', 'JLessEqualLong', 'JNotLessEqual', 'JNotLessEqualLong', 'JLessEqualN', 'JLessEqualNLong', 'JNotLessEqualN', 'JNotLessEqualNLong', 'JGreater', 'JGreaterLong', 'JNotGreater', 'JNotGreaterLong', 'JGreaterN', 'JGreaterNLong', 'JNotGreaterN', 'JNotGreaterNLong', 'JGreaterEqual', 'JGreaterEqualLong', 'JNotGreaterEqual', 'JNotGreaterEqualLong', 'JGreaterEqualN', 'JGreaterEqualNLong', 'JNotGreaterEqualN', 'JNotGreaterEqualNLong', 'JEqual', 'JEqualLong', 'JNotEqual', 'JNotEqualLong', 'JStrictEqual', 'JStrictEqualLong', 'JStrictNotEqual', 'JStrictNotEqualLong', 'Add32', 'Sub32', 'Mul32', 'Divi32', 'Divu32', 'Loadi8', 'Loadu8', 'Loadi16', 'Loadu16', 'Loadi32', 'Loadu32', 'Store8', 'Store16', 'Store32']
len of opcode_mapper: 201

And the error:

    opcode = opcode_mapper[bc[i]]
IndexError: list index out of range

It looking for 204 in the opcode.json and cannot find it.

the problem is deeper, bc is not correctly read

[hexpwn]

I have fixed the issues related to bytecode v84 with https://github.com/bongtrop/hbctool/pull/31