steamos-compositor does not support windows overlays

[Vladislavuz]

For most games you can skip the pre-setup by deleting the installscript.vdf file before the launch. But some games use pre-setup also after launch. For example GTA 4 checks the Rockstar launcher setting, where you need to perform some actions that are hidden by the first window. This makes it impossible to start the game.

[brain-anti-freeze]

Try switching to https://github.com/gamer-os/steamos-compositor-plus - I use it and haven't had any issues with Rockstar launcher (I'm no longer on SteamOS any more but I did use it when I was)