Clean up compilation build files

[avborup]

Either automatically delete the file at exe_path or add a post_test-like command to the config (command that runs after a test is complete).