OpenTomb
OpenTomb copied to clipboard
opentomb
Handling of external libraries
We need to converge with this discussion somewhere because right now, it is all over the place. So here are the possibilities for handling the dependencies our code has that I've been able to think of:
- Store the code directly in the repository. Pro: Very easy; no setup required; inherently works on all platforms. Can make own modifications. Cons: Can make own modifications that aren't visible to the outside world; update difficult.
- Store the binaries directly. Pro: Update becomes easy. Con: Need at least three versions of everything (Mac, Linux, Windows), possibly more depending on 32/64 bit. Any new platform (does FreeBSD have OpenGL?) increases the pain.
- Hope the user has installed the libraries. Pro: Nothing to do on our part except edit the Readme.md. Con: No idea who uses what version of what library.
- Use external Git references to the libraries (to known tags). Pro: Using official version; no modifications make it back. Same ease of use as storing in repository. Con: Git external references are really hard. Especially if some library we want doesn't have a git repository; there's some Git/SVN bridge but I've never tried it.
A few more notes:
- SDL is funny because the Mac version is a different format (Framework) than any other library. That's my duty to figure out, of course, for everyone else that's not an issue.
- How big a problem is incompatibility really? Probably not that big; incompatible OpenGL drivers hurt way more. But of course it takes only one issue to make the whole thing less fun.
- We definitely need some way to deal with our own version of OpenAL. It does things the normal version doesn't, and I don't want to throw that away without very good reason. We could put it in a separate repository here and then use it like any other third-party library, of course.
- My personal vote would be for Git external references, I just have very little experience with actually using them.
Cochrane, is it possible to install/compile/link OpenAL Soft (http://kcat.strangesoft.net/openal.html) for Mac along with stock OpenAL? OpenTomb-bundled version wasn't very different from default OpenAL Soft, except it used custom-written SDL backend (which I will really miss - but I wrote a letter to Chris Robinson asking if he can implement SDL backend officially, or at least XAudio2 output for Windows).
I have more concerns regarding recently introduced sndfile library. I won't babble about how easy and simple it was without it and with only using native SDL functions to read and decompress audio samples, but what is the whole point in using sndfile along with OGG/Vorbis, if sndfile supports OGG/Vorbis itself?! Basically, we now have FOUR sound library dependencies (SDL, OpenAL Soft, OGG/Vorbis and now sndfile)! I'm sorry, but OpenTomb isn't a sound editing or video editing software, we need it light and simple - so either get rid of OGG/Vorbis or sndfile.
By the way, if sndfile can provide EASY interface for streaming MS-ADPCM wave files (TR3-TR5), then it should absolutely stay in place. I just had no idea how to program MS-ADPCM streaming when I wrote audio engine.
P.S.: Sorry, but maybe I got something wrong... Maybe sndfile requires OGG/Vorbis lib to read .ogg files? If that's the case, then self-written OGG streaming code should go away, and sndfile streaming should be implemented instead.
MS-ADPCM is possible (see http://www.mega-nerd.com/libsndfile/); and from what I can tell from the sources (https://github.com/erikd/libsndfile/tree/master/src), it requires external libogg and libvorbis, so we still need to link against it.
Ok then, I thought libsndfile supports OGG/Vorbis itself! Maybe using libsndfile for streaming will solve current loading lag issue with soundtracks.
Generally - I'm all for removing third-party libs from source code, and using third approach offered by Cochrane - just put an info about version requirements into Readme, and hope that user will install libraries for himself. Because it's already happening anyway (I needed to install SDL and zlib anyway, and in case of zlib, I had funny time finding pre-compiled libs on the internet, as I'm too lazy and dumb to compile them myself).
Not sure if this approach will break nightly builds, we need to ask vobject about that... So far it worked with external SDL and zlib.
Maybe we should set up a wiki page with dev requirements, like limiting boost to only use headers, external library handling and versioning, coding style, whatever.
The wiki page is a good idea to document whatever we come up with here, once we have something.
@Lwmte: Installing a custom version of OpenAL is no problem on OS X, and I think actually what most games do. But again, throwing out our existing patched version needs strong justification that I just don't see.
I'm not fond of having everyone install their own libraries. Everything that makes it more difficult for new developers to join is a problem, not an advantage.
Actually, I see two problems here:
- a development environment
- a deployment environment
They can have different setups. Actually, there is a special MinGW case already for cross-compilation.
To summarize: my last merge request eliminated the need for SDL Audio, so the only depencies are now OpenAL and libsndfile (which in turn requires ogg/vorbis). Both libraries are available on all target platforms. The correct way of using XAudio in windows is to abstract the audio engine in OpenTomb so that on windows it uses XAudio directly, while using OpenAL on all other platforms. Patching XAudio into OpenAL is surely the wrong way, as they both are abstraction layers with the same underlaying layer. And well, if developers cannot even install a needed lib, then they are not worth the title "developer".
Sorry, but I still haven't seen any reasonable argument that let me even consider to leave the code in.
And well, if developers cannot even install a needed lib, then they are not worth the title "developer".
That's a very strong statement that I'd like to avoid. We don't need to alienate people without system administration experience. I know many professional Java developers working on Windows who don't even know what CMD.EXE is. Don't laugh - it's true.
I have my very strong opinion about Java "developers", and I'm not laughing, as I know that's true.
But we're talking about C/C++ developers here. If you want to attract people that do not know how to use/build/install external libs, this project is going to be doomed, as such people usually write code that's ... well, weird and not very error-prone.
Please don't think I'm only ranting. But the need for including the libs in-source comes from reasons that will lead to great problems in the long term.
I don't mean to be rude here, but the first and only time we've had absolutely massive problems with libraries was when someone decided to break all existing processes for dealing with them because it looked cleaner. I like insulting people as much as anybody, but in the end it has rarely, if ever, solved problems.
If you think you have a solution that works better than what we had, great, show it to me. So far I haven't seen it.
To be fair, nobody openly opposed that move. So, I think it's a scapegoating to blame the author of the pull request.
It's pretty simple, but requires some work: Implement two audio engines, one for OpenAL, one for XAudio. This needs an abstraction/indirection layer in OpenTomb, but it's the only clean way, because patching XAudio support into OpenAL is like patching DirectX support into OpenGL.
To be fair, nobody openly opposed that move. So, I think it's a scapegoating to blame the author of the pull request.
You're right. @stohrendorf : Sorry.
It's pretty simple, but requires some work: Implement two audio engines, one for OpenAL, one for XAudio. This needs an abstraction/indirection layer in OpenTomb, but it's the only clean way, because patching XAudio support into OpenAL is like patching DirectX support into OpenGL.
I don't really understand your objection here. Yeah, it is a bit weird, but it is also code that is already tested and works well. You propose to replace that well-tested code and replace it with a huge new thing that needs to be written and tested again and which will expose an interface that nobody is familiar with, for no actual benefit other than that a description of it sounds nicer.
I understand the desire for clean solutions, but maintenance cost is a real thing, too. Now, if you want to write that translation layer, be my guest. But I just don't see the benefit.
Then there's FMOD Ex; it isn't open source, but a fairly advanced sound engine for win/mac/linux. A colleague who worked for a game developer told me about that.
I'd second the maintenance concerns.
Then there's FMOD
Wouldn't that be incompatible with OpenTomb license?
To add to my previous post. There are such big projects as OpenStack and Xen which use their own development environment (DevStack and Raisin respectively). They have just the same concerns about consistency of their development environment for the developers and easy maintenance. Though, it has a big cost of designing and constructing such reference environment in the first place.
But having all reference libraries as an optional git submodules used by the developers can ease the pain considerably.
I think it requires some elaboration.
The idea is to move all external libraries into submodules to make it optional. Those who require special library versions should explicitly checkout the submodules. That means cross-compiling or some platform specific versions which should be hosted in the opentomb organization. Others shouldn't do anything special and use system libraries by default.
We can even have different branches for that, i.e. development or stable which could differentiate between debug and release specific libraries.
Why not use the DirectSound backend of openal-soft on Windows and leave XAudio alone for now? If you want to have XAudio, work on openal-soft.
In fact, I got reply from Chris Robinson (who works on OpenAL Soft), and he told me that there's no benefit in XAudio2, when compared to mmdevapi (actually, people say that XAudio2 is an abstraction layer over mmdevapi), and OpenAL Soft already uses mmdevapi as a backend for Windows systems since Vista. So it's no more actual.
And the mac-issues with missing extensions could also be solved with b3cb74737a542eef011b6ceea8ef62e059b7a267, as I had missing extensions here on windows, too, until I implemented probing all output devices for EFX support.
By the way, can we store OpenAL Soft as a reference too? OpenTomb makes no use of native SDL audio features, and now, when XAudio2/mmdevapi business is sorted out, we can use official version without modifications. P.S.: Chris also said to me that using SDL as a backend with OpenAL is unsafe, even in SDL2. There is a risk that OpenAL may still be shutting down while SDL is already destroyed on exit, which is dirty.
If you need a link to OpenAL git repo, here is it: http://repo.or.cz/w/openal-soft.git
Also, what about SDL2, SDL_Image, zlib and Lua? I don't remember whole story, but why these were left as links to pre-compiled libs, if we decided to convert everything to references? Or there is a reason for such setup?
As far as I remember, you and TeslaRus said that the only patched libraries were bullet and openal. Cochrane also needs patched openal for Mac OSX. Why freetype is there I don't remember. It was decided that it's better to use system libraries for everything else. Actually, I think that developers and users should have different setups.
Well, currently we're not using patched Bullet anymore (by the way, Lua was also patched, but it is now referenced as external library, not as submodule), and seems we don't need patched OpenAL either after all this discussion about XAudio. So effectively, only patched library we currently need is stohrendorf's LuaState fork (ironically, as Steffen proposed removing patched libs in the first place).
Cochrane had problems with stock OpenAL for Mac (I have no idea what's the version, probably it's a fork of last open-source Creative OpenAL, which should be ridiculously ancient), what he really needs is any version of OpenAL Soft, and there is no real difference between submodule reference to current git repo of OpenAL Soft and bundled patched version. Only difference is backend.
So... I believe it's time to finally decide how we manage libs. If we've decided to use submodules, we should get rid of system libs completely (meaning - SDL2, SDL_Image, zlib and Lua should be also added as submodules... ah, I also forgot about libsndfile and ogg/vorbis). Right?
And I'll say again - I believe there is an ongoing issue with different includes of Bullet modules, as I have seen <> and "" links to them roaming around.
I don't see why it should be either one or the other. Why not both? You can keep only patched libs as submodules and rely on the system libs for everything else. I should point out that if OpenTomb will ever get packaged for distribution, then it should rely on the system libraries.
As for includes I think it's better to keep them global as <>.