buffer overflow - optimizer 816-opt

[RetroAntho]

trafficstars

Hi,

I wanted to build with the last version of pvsneslib this project (linked in the wiki). The project does not respect programming best practices and have only a huge .c file but the compilation failed with a buffer overflow :

**/Repository/pvsneslib/devkitsnes/tools/816-opt sor2.ps >sor2.asp optimization pass 1: 1314 optimizations performed optimization pass 2: *** buffer overflow detected *: terminated Aborted (core dumped)

Here is the project :

sor2_snes.zip

[kobenairb]

Hi @RetroAntho,

It's weird, it works for me:

/home/kobenairb/workspace/tests/pvsneslib/devkitsnes/tools/816-opt sor2.ps >sor2.asp optimization pass 1: 1314 optimizations performed optimization pass 2: 646 optimizations performed optimization pass 3: 90 optimizations performed optimization pass 4: 24 optimizations performed optimization pass 5: 23 optimizations performed optimization pass 6: 0 optimizations performed 2097 optimizations performed in total

Can you give more details please ? which OS / Platform do you use ?

[RetroAntho]

Hi @kobenairb,

I use the last version of develop branch. I am on Ubuntu 22.04.3 LTS, Release: 22.04

gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)

So i will probably have to investigate further !

[alekmaul]

what is sor2 program? perhaps it is the same issue I have with windows but ubuntu is stopping execution with a buffer overflow. did you test with last commit of @kobenairb regarding the buffer size for optimization?

[RetroAntho]

It is the port of street of rage began by @odelot few years ago. I confirm that i tried the last commit and it does not solve the case

[alekmaul]

you tested "gdb /path/to/coredumpfile" to have more detail of your coredump file?

[RetroAntho]

For those who are on Ubuntu like me, the core dump is not generated by default. To get it, i installed "systemd-coredump" (sudo apt install systemd-coredump) then i found the core dump location by using the "coredumpctl" command. I also tried with the file generated here /var/lib/systemd/coredump

but any file does not works and i finally have the error : file format not recognized

I need to find a way to debug it (to avoid using printf :) )

[kobenairb]

I would like to help you @RetroAntho, but I can't reproduce your error (buffer overflow). A first step could to use valgrind, which can provides some useful informations to figure out at which line of the program the overflow happens.

This can be achieve by using valgrind --tool=memcheck --leak-check=full --undef-value-errors=no ./816-opt ...

Update I can finally reproduce it :) I mean sometimes it works, sometimes I got a buffer overflow on the second pass exactly like you.

First try: /home/kobenairb/workspace/tests/pvsneslib/devkitsnes/tools/816-opt sor2.ps >sor2.asp optimization pass 1: 1314 optimizations performed optimization pass 2: 646 optimizations performed optimization pass 3: 90 optimizations performed optimization pass 4: 24 optimizations performed optimization pass 5: 23 optimizations performed optimization pass 6: 0 optimizations performed 2097 optimizations performed in total cp sor2.asp sor2.opt.02.dbg Moving constants ... sor2.ps /home/kobenairb/workspace/tests/pvsneslib/devkitsnes/tools/constify sor2.c sor2.asp sor2.asm constify: Done 'Moved 0 variables (0 bytes)' cp sor2.asp sor2.ctf.03.dbg Doing obj files ... Building with -x flag: /home/kobenairb/workspace/tests/pvsneslib/devkitsnes/bin/wla-65816 -s -x -o sor2.obj sor2.asm /home/kobenairb/workspace/tests/pvsneslib/devkitsnes/bin/wla-65816 -d -s -x -o sor2.obj sor2.asm sor2.asm:1198: ERROR: Unknown symbol "��".

Second try (buffer overflow): /home/kobenairb/workspace/tests/pvsneslib/devkitsnes/bin/816-tcc -I/home/kobenairb/workspace/tests/pvsneslib/pvsneslib/include -I/home/kobenairb/workspace/tests/pvsneslib/devkitsnes/include -I/home/kobenairb/workspace/tests/pvsneslib/snes-examples/sor2_snes -Wall -c sor2.c -o sor2.ps sor2.c:141: warning: implicit declaration of function 'RGB15' sor2.c:543: warning: implicit declaration of function 'memset' sor2.c:783: warning: implicit declaration of function 'memcpy' cp sor2.ps sor2.ps.01.dbg Assembling ... sor2.ps /home/kobenairb/workspace/tests/pvsneslib/devkitsnes/tools/816-opt sor2.ps >sor2.asp optimization pass 1: 1314 optimizations performed optimization pass 2: make: *** [/home/kobenairb/workspace/tests/pvsneslib/devkitsnes/snes_rules:92: sor2.asm] Error 139

Third try: /home/kobenairb/workspace/tests/pvsneslib/devkitsnes/bin/816-tcc -I/home/kobenairb/workspace/tests/pvsneslib/pvsneslib/include -I/home/kobenairb/workspace/tests/pvsneslib/devkitsnes/include -I/home/kobenairb/workspace/tests/pvsneslib/snes-examples/sor2_snes -Wall -c sor2.c -o sor2.ps sor2.c:141: warning: implicit declaration of function 'RGB15' sor2.c:543: warning: implicit declaration of function 'memset' sor2.c:783: warning: implicit declaration of function 'memcpy' cp sor2.ps sor2.ps.01.dbg Assembling ... sor2.ps /home/kobenairb/workspace/tests/pvsneslib/devkitsnes/tools/816-opt sor2.ps >sor2.asp optimization pass 1: 1314 optimizations performed optimization pass 2: 646 optimizations performed optimization pass 3: 90 optimizations performed optimization pass 4: 24 optimizations performed optimization pass 5: 23 optimizations performed optimization pass 6: 0 optimizations performed 2097 optimizations performed in total cp sor2.asp sor2.opt.02.dbg Moving constants ... sor2.ps /home/kobenairb/workspace/tests/pvsneslib/devkitsnes/tools/constify sor2.c sor2.asp sor2.asm constify: Done 'Moved 0 variables (0 bytes)' cp sor2.asp sor2.ctf.03.dbg Doing obj files ... Building with -x flag: /home/kobenairb/workspace/tests/pvsneslib/devkitsnes/bin/wla-65816 -s -x -o sor2.obj sor2.asm /home/kobenairb/workspace/tests/pvsneslib/devkitsnes/bin/wla-65816 -d -s -x -o sor2.obj sor2.asm sor2.asm:1449: ERROR: Unknown symbol "I".

An other weird behavior it's when I use valgrind, no more buffer overflow: valgrind --tool=memcheck --leak-check=full --undef-value-errors=no ./816-opt sor2.ps.01.dbg > sor2.asp

[kobenairb]

@RetroAntho, the problem is that code is not C90, you can check it with cppcheck. All others C files in examples look okay, but this ones produces many errors. I think that if you fix them, it will works. You can try it by using cppcheck --std=c90 .

[kobenairb]

Hi @RetroAntho, I finally found the problem with the optimizer, I added a PR to fix. It was a good test from you, thank you to detect it. By the way, forget my last comments who were irrelevants with your problem.

[RetroAntho]

Hi @kobenairb, sorry i have less availabilities since some weeks. Thanks a lot for your investigation and for the fix. I just tried it, i confirm that the 816-opt step works correctly and finish without error.

I saw that the compilation step failed one step after :

sor2.asm:10944: DIRECTIVE_ERROR: Could not parse the .RAMSECTION offset.
sor2.asm:10944: ERROR: Couldn't parse "globram.data".

and i need to check why but we can consider this issue (related to the optimizer) as solved !

See you soon :)

[kobenairb]

Hi @RetroAntho,

You can fix it by renaming offset (in sor2.c) by whatever else (offset is a reserved keyword for wla). I also fixed data.asm by providing the full path for each resources. Now it works but fail on the linker with a FIX_REFERENCES: Reference to an unknown label "EmptyHandler". I'll try to figure out why later :)

Edit: I also added string.h to avoid tcc warnings.

[RetroAntho]

Hi @kobenairb, Thanks, i will try it soon. Do you think it could be interesting to check if the variable name is authorized or add a letter in tcc (v_) to avoid this kind of issue? For your error, are you sure the lib and all tools are correctly compiled? I have this kind of things when the .obj from the lib does not exists :)

[kobenairb]

You're completely right :) Now it works perfectly for me. My FIX_REFERENCES: Reference to an unknown label "EmptyHandler" error message was, indeed, because my mistake, I didn't rebuild pvsneslib :/ so the obj was missing.

Here the working version:

And the code... sor2.zip

[kobenairb]

About "Do you think it could be interesting to check if the variable name is authorized or add a letter in tcc (v_) to avoid this kind of issue?" Yes it can help, but to put it ? tcc is probably the best candidate, but I'm not sure if it's easy to perform. When I read the wla-dx doc, I understand that offset is reserved keyword when we use RAMSECTION, otherwise it's permitted... I will try to see how (and if) it's possible.