Meaning of parameters in 'configuration.toml'

[wenlinyao]

Hi,

In the 'configuration.toml' file, I see a range of parameters, but not sure what those parameters control. Could you please provide one example config file that can produce the result in Table 1 and Table 2 of the paper? One example for each table will be great! Thank you!

[mrT23]

The table shows the results of running: https://github.com/Codium-ai/AlphaCodium?tab=readme-ov-file#solving-the-entire-dataset

With:

[dataset]
num_iterations=5

Beware - this is quite a long and expensive run...

To maximize the score, add some "randomness" between different iterations, meaning choose somewhat different params for the stages in different iterations, to get the best possible chances for solving.

I am not going to share at this point my full "recipe" for that, as it is just an ensemble detail, which is highly uncommon to share in academic implementations. even without adding randomness, you should get results close to that.

[wenlinyao]

Thanks for your reply!

When I run it with more iterations, I sometimes can see one error message as follows. Will it create problems in evaluation?

ERROR:asyncio:Task exception was never retrieved future: <Task finished name='Task-11476' coro=<AsyncClient.aclose() done, defined at /data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpx/_client.py:1996> exception=RuntimeError('Event loop is cl osed')> Traceback (most recent call last): File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpx/_client.py", line 2006, in aclose await proxy.aclose() File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpx/_transports/default.py", line 383, in aclose await self._pool.aclose() File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 324, in aclose await connection.aclose() File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_async/http_proxy.py", line 350, in aclose await self._connection.aclose() File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_async/http11.py", line 253, in aclose await self._network_stream.aclose() File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 54, in aclose await self._stream.aclose() File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/anyio/streams/tls.py", line 202, in aclose await self.transport_stream.aclose() File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1181, in aclose self._transport.close() File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/asyncio/selector_events.py", line 698, in close self._loop.call_soon(self._call_connection_lost, None) File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/asyncio/base_events.py", line 751, in call_soon self._check_closed() File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/asyncio/base_events.py", line 515, in _check_closed raise RuntimeError('Event loop is closed') RuntimeError: Event loop is closed

[mrT23]

can you reproduce this error ? does it happen on specific problems ? does it happen always ?

[wenlinyao]

Yeah, this error is quite random. The output is something like this:

2024-01-29 23:29:54.595 | INFO | alpha_codium.gen.stages.run_initial_solve:run_initial_solve:14 - --initial solve stage-- 2024-01-29 23:29:54.600 | INFO | alpha_codium.llm.ai_handler:chat_completion:86 - ----------------- 2024-01-29 23:29:54.601 | INFO | alpha_codium.llm.ai_handler:chat_completion:87 - Running inference ... 2024-01-29 23:30:00.540 | INFO | alpha_codium.llm.ai_handler:chat_completion:133 - done 2024-01-29 23:30:00.540 | INFO | alpha_codium.llm.ai_handler:chat_completion:134 - ----------------- 2024-01-29 23:30:00.606 | INFO | alpha_codium.code_contests.eval.code_test_runners:print_test_results:111 - test-0 :: status=ProgramStatus.kSuccess, passed=False 2024-01-29 23:30:00.607 | INFO | alpha_codium.code_contests.eval.code_test_runners:print_test_results:116 - ===================================================================== 2024-01-29 23:30:00.607 | INFO | alpha_codium.gen.stages.run_initial_code_generation:run_initial_code_generation:61 - Failed to pass tests after 2 attempts, d: 8.0, best_d so far: 5 2024-01-29 23:30:00.607 | INFO | alpha_codium.gen.stages.run_initial_solve:run_initial_solve:14 - --initial solve stage-- 2024-01-29 23:30:00.613 | INFO | alpha_codium.llm.ai_handler:chat_completion:86 - ----------------- 2024-01-29 23:30:00.613 | INFO | alpha_codium.llm.ai_handler:chat_completion:87 - Running inference ... ERROR:asyncio:Task exception was never retrieved future: <Task finished name='Task-24504' coro=<AsyncClient.aclose() done, defined at /data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpx/_client.py:1996> exception=RuntimeError('Event loop is cl osed')> Traceback (most recent call last): File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpx/_client.py", line 2006, in aclose await proxy.aclose() File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpx/_transports/default.py", line 383, in aclose await self._pool.aclose() File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 324, in aclose await connection.aclose() File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_async/http_proxy.py", line 350, in aclose await self._connection.aclose() File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_async/http11.py", line 253, in aclose await self._network_stream.aclose() File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 54, in aclose await self._stream.aclose() File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/anyio/streams/tls.py", line 202, in aclose await self.transport_stream.aclose() File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1181, in aclose self._transport.close() File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/asyncio/selector_events.py", line 698, in close self._loop.call_soon(self._call_connection_lost, None) File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/asyncio/base_events.py", line 751, in call_soon self._check_closed() File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/asyncio/base_events.py", line 515, in _check_closed raise RuntimeError('Event loop is closed') RuntimeError: Event loop is closed ERROR:asyncio:Task exception was never retrieved ERROR:asyncio:Task exception was never retrieved future: <Task finished name='Task-24506' coro=<AsyncClient.aclose() done, defined at /data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpx/_client.py:1996> exception=RuntimeError('Event loop is closed')> Traceback (most recent call last): File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpx/_client.py", line 2006, in aclose await proxy.aclose() File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpx/_transports/default.py", line 383, in aclose await self._pool.aclose() File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 324, in aclose await connection.aclose() File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_async/http_proxy.py", line 350, in aclose await self._connection.aclose() File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_async/http11.py", line 253, in aclose await self._network_stream.aclose() File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 54, in aclose await self._stream.aclose() File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/anyio/streams/tls.py", line 202, in aclose await self.transport_stream.aclose() File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1181, in aclose self._transport.close() File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/asyncio/selector_events.py", line 698, in close self._loop.call_soon(self._call_connection_lost, None) File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/asyncio/base_events.py", line 751, in call_soon self._check_closed() File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/asyncio/base_events.py", line 515, in _check_closed raise RuntimeError('Event loop is closed') RuntimeError: Event loop is closed 2024-01-29 23:30:06.222 | INFO | alpha_codium.llm.ai_handler:chat_completion:133 - done 2024-01-29 23:30:06.222 | INFO | alpha_codium.llm.ai_handler:chat_completion:134 - ----------------- 2024-01-29 23:30:06.290 | INFO | alpha_codium.code_contests.eval.code_test_runners:print_test_results:111 - test-0 :: status=ProgramStatus.kSuccess, passed=False 2024-01-29 23:30:06.290 | INFO | alpha_codium.code_contests.eval.code_test_runners:print_test_results:116 - ===================================================================== 2024-01-29 23:30:06.290 | INFO | alpha_codium.gen.stages.run_initial_code_generation:run_initial_code_generation:61 - Failed to pass tests after 3 attempts, d: 3.0, best_d so far: 5 2024-01-29 23:30:06.291 | INFO | alpha_codium.gen.stages.run_initial_solve:run_initial_solve:14 - --initial solve stage-- 2024-01-29 23:30:06.297 | INFO | alpha_codium.llm.ai_handler:chat_completion:86 - ----------------- 2024-01-29 23:30:06.297 | INFO | alpha_codium.llm.ai_handler:chat_completion:87 - Running inference ... 2024-01-29 23:30:12.458 | INFO | alpha_codium.llm.ai_handler:chat_completion:133 - done 2024-01-29 23:30:12.458 | INFO | alpha_codium.llm.ai_handler:chat_completion:134 - ----------------- 2024-01-29 23:30:12.524 | INFO | alpha_codium.code_contests.eval.code_test_runners:print_test_results:111 - test-0 :: status=ProgramStatus.kFailed, passed=False 2024-01-29 23:30:12.525 | INFO | alpha_codium.code_contests.eval.code_test_runners:print_test_results:116 - ===================================================================== 2024-01-29 23:30:12.525 | ERROR | alpha_codium.gen.stages.run_tests:run_tests:36 - failed to run solution 2024-01-29 23:30:12.525 | INFO | alpha_codium.gen.stages.run_initial_code_generation:run_initial_code_generation:61 - Failed to pass tests after 4 attempts, d: 5, best_d so far: 3.0 2024-01-29 23:30:12.526 | INFO | alpha_codium.gen.stages.run_initial_solve:run_initial_solve:14 - --initial solve stage-- 2024-01-29 23:30:12.531 | INFO | alpha_codium.llm.ai_handler:chat_completion:86 - ----------------- 2024-01-29 23:30:12.532 | INFO | alpha_codium.llm.ai_handler:chat_completion:87 - Running inference ... 2024-01-29 23:30:19.458 | INFO | alpha_codium.llm.ai_handler:chat_completion:133 - done 2024-01-29 23:30:19.458 | INFO | alpha_codium.llm.ai_handler:chat_completion:134 - -----------------

Another interesting thing I found is that when I run gpt-3.5-turbo-0613 on the valid split (iterations=1) twice, the total amount of data instances after evaluation is quite different. One is 9+87=96 instances and another one is 12+ 80=92. I checked the total amount should be 177? Is this due to the above error?

First time run: problem 8 passed all tests problem 17 passed all tests problem 27 passed all tests problem 31 passed all tests problem 60 passed all tests problem 66 passed all tests problem 73 passed all tests problem 84 passed all tests problem 96 passed all tests total_passed: 9, total_failed: 87 pass rate: 0.09375

Second time run: problem 4 passed all tests problem 8 passed all tests problem 16 passed all tests problem 17 passed all tests problem 34 passed all tests problem 50 passed all tests problem 60 passed all tests problem 66 passed all tests problem 73 passed all tests problem 83 passed all tests problem 96 passed all tests problem 114 passed all tests total_passed: 12, total_failed: 80 pass rate: 0.13043478260869565

[mrT23]

• The validation set contains 107 examples. i think 3 or 4 are invalid, because the provided generated tests contain illegal input. so you generated responses for 92 out of ~104 valid problems

• i think that if you ran the script again, with the same file, it will only try to solve problems it skipped (make sure to back up the file).

• i am not sure why you see the async error. its weird. the log doesn't report a problem in the run itself. maybe some memory issue ? make sure you run with 'sandbox=true'. or try with a machine with larger memory.

• for 1 or 2 iteration, 13% is a reasonable solve ratio. for more iterations, think of strategies how to create some randomness between each iteration, to increase the chance of pass@5 solve. (hint - try on 1-2 iterations the 'use_direct_solutions' flag. it is somewhat orthogonal, and helps with some problems)

[wenlinyao]

Thanks again! that is very helpful!

• I double-checked the validation set. It should be 117 original instances? Two of my "5 iterations" run returns "total_passed + total_failed = 105", so I guess the total valid problems is 105? Just want to make sure my experiment setting is correct. ;)
• It seems the "async error" can be solved using the original OpenAI API call instead of litellm.
• That is a good point. But I see 'use_direct_solutions' flag is defined in the 'configuration.toml' file. So how can I run 5 iterations with a partial of them applying 'use_direct_solutions=true'?

[mrT23]

"So how can I run 5 iterations with a partial of them applying 'use_direct_solutions=true'?"

edit the file https://github.com/Codium-ai/AlphaCodium/blob/main/alpha_codium/gen/dataset_solver.py#L75

if iteration <2 :
...
else:
...

[huanhuan6666]

@wenlinyao Hello, I've encountered a similar issue and would greatly appreciate your guidance:

ERROR:asyncio:Task exception was never retrieved
future: <Task finished name='Task-479' coro=<AsyncClient.aclose() done, defined at /data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/httpx/_client.py:1996> exception=RuntimeError('Event loop is closed')>
Traceback (most recent call last):
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/httpx/_client.py", line 2006, in aclose
    await proxy.aclose()
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/httpx/_transports/default.py", line 383, in aclose
    await self._pool.aclose()
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 324, in aclose
    await connection.aclose()
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/httpcore/_async/http_proxy.py", line 350, in aclose
    await self._connection.aclose()

Could you please provide detailed guidance on how to address this issue? I'm eager to learn and make the necessary modifications.🥰

[wenlinyao]

@huanhuan6666 I just saw your message. I am not sure how to solve this issue thoroughly. But you can try to replace litellm LLM call with the OpenAI official API call.