wt-tools
wt-tools copied to clipboard
klensy
Update to new (2.7.0.58+) vromfs/blk encoding
I tried to unpack 2.7.0.56 blk files, but i couldn't due to an error.
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 2: invalid start byte
I have no idea what this mean... I need help
Lines from errors looks wrong, you tried latest (https://github.com/klensy/wt-tools/releases/tag/v0.2.2.8) version?
Yes, it was the latest version. Re-download the tool, tried again and got another error
First error about attachable.blk says that file already unpacked not packed, strange.
There is no non-english path to files. pre-2.7.0.58 version files are successfully unpacked. post-2.7.0.58 version files are failed. Seems like something's changed with 0.58 update. Edit: Just realized that the game version I'm trying to unpack was 0.58, not 0.56.
Here https://github.com/gszabi99/War-Thunder-Datamine/commits/master version 2.7.0.56 unpacked at least. I'll try to update game and check.
From 2.7.0.58 onwards vromfs unpacking sometimes fails, but even if it's successful, blk unpacking always fails anyway.
The only major difference I have spotted in BLK files is the enumeration scheme. Type ids look the same.
Here's the last working aces file I've got, version 2.7.0.55 (version that was 3,5 GB and with leaked vehicles).
Something that you may really really really appreciate https://www.reddit.com/r/Warthunder/comments/o5r4v1/heads_up_on_vrom_and_blk_format_changes_for_data/
@gszabi99 Yeah, vromfs unpacking seems fixed, working on blk's.
Switched main branch to dev to show actual progress, instead of old master one.
Hello. Is it possible to restore the unpacking of the War Thunder game resource files? It's not working right now. I am using the latest version of wt-tools from the dev branch. Below I attach a piece of the log.
orig Здравствуйте. Возможно ли восстановить распаковку файлов ресурсов игры War Thunder? Сейчас она не работает. Использую последнюю версию wt-tools из ветки dev. Ниже прикладываю кусочек лога. log.txt
Hello. Is it possible to restore the unpacking of the War Thunder game resource files? It's not working right now. I am using the latest version of wt-tools from the dev branch. Below I attach a piece of the log.
orig Здравствуйте. Возможно ли восстановить распаковку файлов ресурсов игры War Thunder? Сейчас она не работает. Использую последнюю версию wt-tools из ветки dev. Ниже прикладываю кусочек лога. log.txt
in case you didn't know, gaijin updated the format of the files, which broke the tools. Klensy is already working on updating them
I know the file format has changed. It looks like the aces.vromfs.bin file is not unpacked either. Here is the log, can it help with what
orig text: Я знаю что формат файлов изменился. Похоже что и файл aces.vromfs.bin не распаковывается. Вот лог, может чем поможет... aces.txt
I know the file format has changed. It looks like the aces.vromfs.bin file is not unpacked either. Here is the log, can it help with what
orig text: Я знаю что формат файлов изменился. Похоже что и файл aces.vromfs.bin не распаковывается. Вот лог, может чем поможет... aces.txt
... if you know that the format change, why do you expect the aces to work?
if you know that the format change, why do you expect the aces to work?
No need to be toxic around here. I'm sure there is enough of it in game. No such thing as a stupid question.
if you know that the format change, why do you expect the aces to work?
No need to be toxic around here. I'm sure there is enough of it in game. No such thing as a stupid question.
sorry I didn't wanted to be toxic.
@klensy idk if you had a chance to look into it. Increasing max_output_size for zstd compression seems to work for vromfs ( I set it to 200_000_000). Considering the addition of the name map, maybe it would be better to include blk unpacker as part of vromfs unpacker
I have a prototype of a codec of the new blk format, restored using a public algorithm. It builds an intermediate representation of the tree as a multi-dictionary of names and values and it is very slow.
i5-4670 @ 1 core in: tmpfs, out: hdd
pytest -vvs --durations=0 test_unpack_dir.py
========================================================================================================================== test session starts ==========================================================================================================================
platform linux -- Python 3.9.6, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /home/kotiq/.virtualenvs/blk/bin/python
cachedir: .pytest_cache
rootdir: /media/shared/src/games/WarThunder/blk, configfile: pytest.ini
collected 2 items
test_unpack_dir.py::test_unpack_fat_dir[game.vromfs.bin_u] PASSED
test_unpack_dir.py::test_unpack_slim_dir[aces.vromfs.bin_u] PASSED
=========================================================================================================================== slowest durations ===========================================================================================================================
145.11s call tests/test_blk/test_binary/test_constructor/test_unpack_dir.py::test_unpack_slim_dir[aces.vromfs.bin_u]
0.05s call tests/test_blk/test_binary/test_constructor/test_unpack_dir.py::test_unpack_fat_dir[game.vromfs.bin_u]
0.00s setup tests/test_blk/test_binary/test_constructor/test_unpack_dir.py::test_unpack_fat_dir[game.vromfs.bin_u]
0.00s setup tests/test_blk/test_binary/test_constructor/test_unpack_dir.py::test_unpack_slim_dir[aces.vromfs.bin_u]
0.00s teardown tests/test_blk/test_binary/test_constructor/test_unpack_dir.py::test_unpack_slim_dir[aces.vromfs.bin_u]
0.00s teardown tests/test_blk/test_binary/test_constructor/test_unpack_dir.py::test_unpack_fat_dir[game.vromfs.bin_u]
===================================================================================================================== 2 passed in 145.18s (0:02:25) =====================================================================================================================
145s for the call but 0 second for the teardown ?
i would have understood the opposite, well as long as it works.
also 145 second for aces is like, good
Aces.vromfs generally takes a lot of time to unpack
try with Char.vromfs.bin, with the old files/unpacker it takes me around 11.3 seconds
Edit : old Aces with old unpacker takes 59 seconds, so well never mind
aces.vromfs.bin v 2.7.0.46 under the same conditions:
time python3 -m blk_unpack --format=strict_blk aces.vromfs.bin_u/
...
real 0m41.039s
user 0m38.300s
sys 0m0.823s
I will need to simulate the interfaces so that your scripts at a higher level work as before. If you write the wrappers yourself, I will publish the library. There are examples of unpacking in the tests.
@AidenTKZ
pytest -vvs --no-header --durations=0 test_unpack_dir.py::test_unpack_slim_dir[char.vromfs.bin_u]
============================================================================ test session starts ============================================================================
collected 1 item
test_unpack_dir.py::test_unpack_slim_dir[char.vromfs.bin_u] PASSED
============================================================================= slowest durations =============================================================================
13.76s call tests/test_blk/test_binary/test_constructor/test_unpack_dir.py::test_unpack_slim_dir[char.vromfs.bin_u]
0.00s setup tests/test_blk/test_binary/test_constructor/test_unpack_dir.py::test_unpack_slim_dir[char.vromfs.bin_u]
0.00s teardown tests/test_blk/test_binary/test_constructor/test_unpack_dir.py::test_unpack_slim_dir[char.vromfs.bin_u]
============================================================================ 1 passed in 13.78s =============================================================================
char.vromfs.bin v 2.5.1.125
time python3 -m blk_unpack --format=strict_blk char.vromfs.bin_u/
real 0m3.336s
user 0m3.238s
sys 0m0.074s
Для ознакомления с новым форматом repo.
Оцените указанные риски при возможной смене порядка в текстах, если будете их заливать в свои репозитории.
По-хорошему, гляньте код, если пост на reddit был запутанным, и закодируйте сериализацию правильно.
@kotiq Have you tried unpacking the new char vromfs? I always get a zstd decompression error. Is that something you can help with perhaps?
@gszabi99 1 2 I used 5_000_000 for unpacking.
@AidenTKZ wt_tools/blk_unpack.py:123,126,131. The specified lines contain ensure_ascii=False.
The scheme for the Color was changed. Now RGBA color is stored as BGRA for raw bytes in a binary file
b'AABBCCDD' -> [0xCC, 0xBB, 0xAA, 0xDD]
@kotiq For me blk_unpack_demo.py crashes on aces.vromfs.bin_u\config\tanksights.blk. Do you perhaps know what I'm doing wrong? All files before this specific file are unpacked correctly.
Traceback (most recent call last):
File "D:\Programming\datablock\blk\tests\demo\blk_unpack_demo.py", line 152, in <module>
main()
File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 1137, in __call__
return self.main(*args, **kwargs)
File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 1062, in main
rv = self.invoke(ctx)
File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 763, in invoke
return __callback(*args, **kwargs)
File "D:\Programming\datablock\blk\tests\demo\blk_unpack_demo.py", line 148, in main
process_dir(path, out_type, is_sorted)
File "D:\Programming\datablock\blk\tests\demo\blk_unpack_demo.py", line 115, in process_dir
process_slim_dir(dir_path, names, out_type, is_sorted)
File "D:\Programming\datablock\blk\tests\demo\blk_unpack_demo.py", line 132, in process_slim_dir
process_slim_dir(entry.path, names, out_type, is_sorted)
File "D:\Programming\datablock\blk\tests\demo\blk_unpack_demo.py", line 134, in process_slim_dir
process_file(entry.path, names, out_type, is_sorted)
File "D:\Programming\datablock\blk\tests\demo\blk_unpack_demo.py", line 76, in process_file
serialize_text(root, ostream, out_type, is_sorted)
File "D:\Programming\datablock\blk\tests\demo\blk_unpack_demo.py", line 19, in serialize_text
return jsn.serialize(root, ostream, out_type, is_sorted)
File "D:\Programming\datablock\blk\src\blk\json\serializer.py", line 124, in serialize
json.dump(mapper.map(root), ostream, cls=NoIndentEncoder, ensure_ascii=False, indent=2, separators=(',', ': '),
File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\json\__init__.py", line 180, in dump
fp.write(chunk)
File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 1-3: character maps to <undefined>
@Lord-Mustang The file contains lines in Russian. I didn't explicitly specify utf8 encoding for all text writer's instances. Fixed. Please, update the file.
@klensy any progress to share? @kotiq do you have any plan to share the exe version of your unpacker?
@MuhAbroomz Not before I set up the CI/CD pipeline parts on my VM. I'm new to this. Firstly, that demo script is not a full-fledged unpacker. Secondly, I don't see the benefits of an exe in the case of a python project. Usually Windows Defender blocks such executable files.