Buffer overflow in `GetBoundsEdge`

[zear]

Whenever GetBoundsEdge gets called with a particular combination of plane1/plane2 values, the pos->v and edge->v vectors end up accessed at out-of-bounds indices. In one particular situation, when edge points to memory allocated at the start of a stack frame, this kicks up stack smashing protection and terminates the program.

Analysis

GetBoundsEdge was written with an assumption that d1 and d2 will always yield different digits in range [0..2]: https://github.com/dethrace-labs/dethrace/blob/48b70bb8ff08dae10b00d285a9cc6e6b24a1eeac/src/DETHRACE/common/car.c#L3884-L3885

As such, the value of d3 is then calculated as the remaining digit in said range: https://github.com/dethrace-labs/dethrace/blob/48b70bb8ff08dae10b00d285a9cc6e6b24a1eeac/src/DETHRACE/common/car.c#L3899

However, in practice this function is sometimes called with plane1/plane2 values that produce the same value as result of the & operation. After extensive testing, I observed two such cases:

1. plane1 = 5, plane2 = 1, producing:

   d0 = 0;
   d1 = 0;
   d2 = 3;
   

2. plane1 = 5, plane2 = 5, producing:

   d0 = 0;
   d1 = 0;
   d2 = 3;
   

In both cases d2 is used to incorrectly access pos->v[3] and edge->v[3], outside boundaries of those those two arrays. Moreover, pos->v[1]/edge->v[1] is never overwritten and will be left with old values.

This is practice, but in theory we could also have combinations of plane1/plane2 that produce:

d0 = 1;
d1 = 1;
d2 = 1;

d0 = 2;
d1 = 2;
d2 = -1;

d0 = 3;
d1 = 3;
d2 = -3;

I can't tell whether OG also receives the reproduced combinations of plane values, or if the bug is hiding somewhere in Dethrace's calculation of those variables (which would be LineBoxColl). Someone with more experience at debugging DOS/Windows programs would need to check those out-of-bound accesses in OG blobs at runtime.

To mitigate the buffer overflow, we could simply do:

if (d2 > 2)
    return 1;

If this is an OG bug, then that would put us on par with the OG. But I think we should further debug this issue and see if two planes with matching vector indices are legal, and if so, if we can somehow derive what is supposed to happen with the other vector values.

Steps to reproduce

There is no easy way to reproduce this bug (it might require multiple attempts), however I found this method to yield somewhat consistent results:

1. Select a strong car (e.g. Police APC)
2. Launch the "Fridge Racer" track
3. Pick up speed and mow down the row of safety posts located to the right of the start position (marked in blue below). It seems to create best results when in the process the car also collides with the lamp post.

This might require several minutes to reproduce, so make sure to have a debugger session running.