WolvenKit
WolvenKit copied to clipboard
WolvenKit
More cross-platform friendly
Is there any way of making this more cross-platform? I'm trying to build this in Linux. Could always use Wine but may be limiting
Since it has been moved to .NET 5, you should be able to compile it on Linux now.
If you want to build this as a single binary (all you need is CP2077tools.exe), you can do this:
dotnet publish -r win-x64 -c Release /p:PublishSingleFile=true /p:PublishTrimmed=true /p:PublishReadyToRun=true /p:PublishReadyToRunShowWarnings=true /p:IncludeNativeLibrariesInSingleFile=true /p:IncludeNativeLibrariesForSelfExtract=true
Change win-x64 to runtime id of choice.
Just to mention a few caveats:
- texconv.exe is used for texture conversion and there is no Linux version
-
oo2ext_7_win64.dll is used for decompressing. This library is proprietary and taken from Windows release of the game. Have to find a legal way to obtain a
.soversion because there is no a Linux version of the game yet.
Just to mention a few caveats:
* [texconv.exe](https://github.com/microsoft/DirectXTex/releases) is used for texture conversion and there is no Linux version
Imagemagick should be somewhat equivalent and is free software, runs on linux.
* [oo2ext_7_win64.dll](http://www.radgametools.com/oodlecompressors.htm) is used for decompressing. This library is proprietary and taken from Windows release of the game. Have to find a legal way to obtain a `.so` version because there is no a Linux version of the game yet.
There are reverse engineered projects made open source : https://github.com/rarten/ooz
I removed memory-mapped files, so that's one step done. But yes, the other two need to be addressed.
Also found this fork, which is the most up-to-date so far afaik https://github.com/baconwaifu/ooz
I removed memory-mapped files, so that's one step done.
Just curious what's wrong with memory mapped files? They are work just fine on Unix and improve performance on big files (less memory copying). IIRC only named mmf is Windows-specific feature.
yes named MMFs is the problem. RIght, I suppose we could just used normal MMFS.
TODOS:
General:
- [x] bring back (unnamed) Memorymapped files for reading
Remove external dependencies
- [x] replace the cp77 oodle.dll with ooz
- [x] replace texconv with something else for non-windows platforms
replace texconv with something else for non-windows platforms implemented here: #530
@ADawesomeguy I was using this fork before: https://github.com/baconwaifu/ooz
Nice! I got rarten's to work even though it had some issues, so hopefully I can take a step further with this.
Just got a test build working with ooz! Was able to make libraries for both Windows and Linux from rarten's ooz fork + one slight modification, and then I was able to import it and uh comment out the if (true) that used OodleLoadLib instead of OodleNative, as well as diasble all of the checks that made sure oo2ext was in the directory. After that I compiled it, moved ooz.dll to the directory and decompressed unusedhashes.krak. Gotta set up compression but I feel this is a pretty big step in terms of getting rid of oo2ext.
Ok now I got it working on Linux. Still only decompression but it's functional and I was able to decompress usedhashes.kark!
@rfuzzo Ooz is now working for both compression and decompression on Linux, think I just need to test for Windows and macOS before you can mark that as resolved.
Done:
- builds on Linux/MacOs with net6
- oodle works on all platforms
Todo:
- alternative for texconv on linux and macos. For linux we may be able to compile texconv and directXTex here: https://github.com/rfuzzo/DirectXTexSharp For mac we need an laternative (possible: https://github.com/nominom/bcnencoder.net and https://github.com/SixLabors/ImageSharp)
- use compression level 4 on ooz-based systems? compression level 6 which CDPR uses is not binary equal on ooz)
Does binary equality make a difference? I think it would be worth testing to check.
Good to hear ooz for Linux for Wkit is working!
Letting @rfuzzo know that oozlin, my Linux port of powzix/ooz that I began 3 months ago is released (repo is public) and not vaporware.
New job and new projects continue to take up the majority of my time - have virtually no time for Wkit atm.. But the team has new blood so :thumbsup:
That's awesome! It'd be awesome if I could add this to Wkit and I'm more than willing to give it a shot. Rarten's port is great but there are some things missing and I'm not experienced enough to work with it so it's amazing that you were able to get yours out and I'm excited to start working with it!
If you do use it for Wkit, let me know if you find bugs (open an issue). I know there are a couple, but there's bound to be more due to fat-fingering something. I'll be actively maintaining it as it'll be used for another project.
EDIT: It's also not multi-platform (win code was removed).
As stated in Discord, the unnamed memory maps were part of a hack solution to allow modders (actually one particular modder trying to script his CLI runs and needed multiple CLI's to shorten the processing time) the ability to run multiple instances of Wolvenkit CLI instances at the same time all accessing the game .archive files for uncooking/unbundling.
Before the changes, you would get the "The process cannot access the file because it is being used by another process" and this resolved it. If it's occurring again, it means all the recent changes to 8.5 broke the hack.
On another note, I wouldn't use Win memory maps at all for Linux (which wouldn't work anyway). Best to translate that code to Linux native mmap API (but that's just my opinion).
Cool! How would I go about translating the code to the Linux native mmap API?
The manpages for the system calls: mmap(), munmap(), mprotect() are a good reference, but for an in-depth read with plenty of examples, I recommend the following book: Advanced Programming in the UNIX Environment, 3rd Ed. by W. Richard Stevens and Stephen A. Rago.
There is a chapter on Advanced I/O (Ch. 14), specifically the section on Memory-Mapped I/O where the mmap() system call and its related functions are covered. The book is available as a free PDF here on GitHub: https://github.com/Lincheng1993/apue/blob/master/Advanced%20Programming%20in%20the%20UNIX%20Environment%203rd%20Edition.pdf.
Take a look at a Win memory map function call's parameters and what it returns and compare it with the corresponding Linux system call (and its parameters and return value, if any). See if you can write a small program using the Win memory map function and a small C program on Linux using the corresponding system call(s) to get each implementation to perform similar, if not the same result/output. Use the provided examples the text, see how they work as well.
There might be a better way, but reading, studying examples, and writing practice code to familiarize oneself has worked for me. (Of course formal study in CS (undergraduate and graduate) is a definite plus, but for this, hands on practical work is a very good teacher. I hope this helps (wrote too much lol).
I wrote a little POC in C++:
#include <iostream>
#include <fcntl.h>
#include <sys/mman.h>
#include <sys/stat.h>
const char filepath[] = "test.txt";
int main() {
struct stat64 st;
int fd = open(filepath, O_RDONLY);
stat64(filepath, &st);
char* data = static_cast<char*>(mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0));
std::cout << data << std::endl;
return 0;
}
How would I go about implementing this? Writing a library and including that or something else?
Thanks for your help and the book by the way :)
Done:
- builds on Linux/MacOs with net6
- oodle works on all platforms
Todo:
- alternative for texconv on linux and macos. For linux we may be able to compile texconv and directXTex here: https://github.com/rfuzzo/DirectXTexSharp For mac we need an laternative (possible: https://github.com/nominom/bcnencoder.net and https://github.com/SixLabors/ImageSharp)
- use compression level 4 on ooz-based systems? compression level 6 which CDPR uses is not binary equal on ooz)
@rfuzzo hasn't the compression stuff been fixed? Also is DirectXTex used in the CLI? If so, could you please point me to it so I can see how it's used and check out any alternatives?