[Feature Request] Custom listener position/rotation

[ThreeDeeJay]

Like I mentioned in #637, some games have incorrect positional audio because of flipped axes, but there are also a couple similar issues:

• Rotation: some games have a rotated axis (usually 90 degrees), which makes 3D localization confusing.
  • Grand Theft Auto III: XP CMSS-3D - DSOAL
  • Grand Theft Auto: Vice City: reVC (native OpenAL) - XP CMSS-3D - ALchemy - DSOAL
  • Knight Rider: DSOAL https://github.com/kcat/dsoal/issues/41#issuecomment-1033688301
• Offset: in some games (particularly in third person), the listener is located on the character instead of the camera, which sometimes makes objects in frame sound like they're behind you.
  • Max Payne: DSOAL - XP CMSS-3D - ALchemy
  • The Thing: DSOAL
  • TMNT: DSOAL

So I'm proposing just basic listener (or sound field) transformations in alsoft.ini: rotation (in degrees) and translation (in 3D units) for all 3 axes. e.g., to fix games with a 90 degree pitch offset like the GTAs, I think it could be something like:

[game_compat]
rotate-x=90

and to move the listener position closer so it's aligned with the camera, I think it could be something like:

[game_compat]
translate-z=250

[ThreeDeeJay]

There's also the wildcards like Uebergame: https://youtu.be/orhURrhK_R8 I'm not even sure what's going on. Seems like the game thinks down is front and the listener doesn't rotate with the camera or something. AFAIK my gunshots should always sound like they're in the direction I'm facing as long as my weapon is within line of sight. but here when aiming at the horizon, gunshots sound like they're below me (notice my HRTF muffling the sound); when looking down, it does sound like it's coming from the direction I'm facing, so even if I could rotate the listener/sound field, it might offset the angles that are already working as they should.

[mirh]

The listener is point-wise, right? Is there any "property" that it has that could influence distinguishability (differentiability? separability?) of sound positions? https://github.com/kcat/dsoal/issues/84#issuecomment-1497985711

[ThreeDeeJay]

You mean like listener orientation? If I understand correctly, OpenAL (and thus OpenAL Soft) can control that independently of the source position, so I was hoping OpenAL Soft could take care of reorienting the listener according to the values for my proposed flags in alsoft.ini, so rotate-... would be the equivalent to moving the green circle while translate-... would move the red circle.

[ThreeDeeJay]

On a side note, I've been noticing both off-camera listener and rotation issues. Unreal II also has a wonky listener/soundfield rotation https://youtu.be/CQznj892uCE Though there are position jumps that might not be fixable with a manual sound field rotation/offset It goes to show how little games, even big titles like the OG GTA trilogy, were tested using 3D audio.

[kcat]

If I understand correctly, OpenAL (and thus OpenAL Soft) can control that independently of the source position, so I was hoping OpenAL Soft could take care of reorienting the listener according to the values for my proposed flags in alsoft.ini, so rotate-... would be the equivalent to moving the green circle while translate-... would move the red circle.

To be honest, I have a real hard time seeing how this would be able to fix issues in games. Even poorly tested ones, the effect of rotation or translation being off by some constant would be so significant that they can't be ignored. At least with reverse-z (and reverse-y), there's the excuse that not everyone has surround sound (or with-height) configurations to notice when a conversion goes awry and gives an incorrect sign. Rotation errors would be noticeable on anything not mono, and even mono would make translation errors noticeable.

Of the three rotation examples, maybe Knight Rider could be fixed by swapping X and Z (for an effective 90 degree counter-clockwise rotation), but the GTA examples sound like a more severe bug as the rotation error changes over time. It sounds like the listener is rotating faster than the camera, so there wouldn't be a single constant rotation offset you can set (whatever you set, it would still go off).

For third-person games having the listener at the character's head instead of the camera, that's a design decision on the developers' part. A translation offset to move the listener position from the character to the camera can only work if the camera position and orientation is at a constant offset from the character. The TMNT example wouldn't be fixable, for instance, because the character is constantly moving and rotating in relation to the camera. The Max Payne example shows the camera getting pushed closer to the character when up against the wall, and the The Thing example shows an occasional first-person view, so the apparent listener position would end up farther back from the camera in those instances.

As for Unreal 2 and Medal of Honor, that seems to be an engine bug that fails to produce a proper listener orientation when looking straight up or down. There's not likely anything OpenAL Soft can do since it's not a constant issue, there's likely not any way to detect when it's happening and there's no way to correct it (proper panning would need the very information the game failed to provide, which changes over time).

[mirh]

As a matter of fact, the source code for both gta and ue2 should be available online for study. Rayman 4 also got leaked recently, and that shouldn't be that far from TMNT (could there be similarities with BGAE too?) OpenMoHAA claims its sound code sucks, but perhaps it's still workable enough to make sense of something.

[ThreeDeeJay]

Sorry for the late reply, folks. been away for a couple weeks

@kcat Yeah, these are just suggestions that aren't guaranteed to fix games, but I just thought they'd be worth having to experiment with, though I'm not expecting it anytime soon since there's probably a ton of more important issues in the roadmap😅

I also thought of reversing 2 axes to see if it could simulate a 90 degree rotation but it doesn't quite work, unless I'm missing something. Like in Knight Rider if I start doing donuts, I hear the tires screeching on my side right. and if I reverse both X and Z axes, then I just hear them on my side left instead of the expected front center. My guess is that reverse-z isn't helping since the sound has little to no depth. and even if it did, it would still be somewhere on my left instead of front center, which is why I figured we'd need a rotation flag 🤔 As for the GTA ones, I'm not sure I hear the rotation variability. perhaps the audio's not perfectly synced, but whenever the sound's supposed to be heard behind me, it sounds exactly below me even when I stop moving the camera, and the same for front being heard above.

Now I completely agree the listener translation is more of a hail mary that'd be much less important and more experimental, to modify questionable intended behavior, than a consistent fix. E.g.: Starsky and Hutch has constant gunshots heard on our side/rear right, so pulling the listener back would push the gunshots forward relative to us, to be closer to the visuals and the soundfield would be more balanced. and I don't think it would significantly ruin positionality when the camera's so far that the gunshots are actually heard in front, like during the jumping cutscenes.

But yeah, the other few I mentioned would probably need a more robust modification to the game/sound engine itself to fix stuff outside of OpenAL Soft's scope.

@mirh I wonder if re3/reVC development would continue now that the Rockstar/Take Two takedown/lawsuit was settled 🤔 But the problem of open source reimplementations is that chances are it'd break support for many other mods and even introduce bugs on its own, especially if it's not actively maintained