thunlp
Trouble Encountering 403 Forbidden Error When Using DebugBench with LeetCode
Hello!
I encountered the same issue 4, when i try to test the generated code through LeetCodeTester. I receive the following error:
Traceback (most recent call last): File "d:\MyProject\Work\code_eval_syh\DebugBench-main\evaluation\leetcode_oj\leetcode_tester.py", line 55, in
print(tester.test(code, task_id, "python")) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "d:\MyProject\Work\code_eval_syh\DebugBench-main\evaluation\leetcode_oj\leetcode_tester.py", line 42, in test status, reward, done, submission_result = self.env.step(sub) ^^^^^^^^^^^^^^^^^^ File "d:\MyProject\Work\code_eval_syh\DebugBench-main\evaluation\leetcode_oj\leetcode_env\environment.py", line 58, in step submission_result = self.__send_submission(action) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "d:\MyProject\Work\code_eval_syh\DebugBench-main\evaluation\leetcode_oj\leetcode_env\environment.py", line 86, in __send_submission submission_id = self.api_instance.problems_problem_submit_post( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\anaconda3\envs\private\Lib\site-packages\leetcode\api\default_api.py", line 385, in problems_problem_submit_post (data) = self.problems_problem_submit_post_with_http_info( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\anaconda3\envs\private\Lib\site-packages\leetcode\api\default_api.py", line 465, in problems_problem_submit_post_with_http_info return self.api_client.call_api( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\anaconda3\envs\private\Lib\site-packages\leetcode\api_client.py", line 345, in call_api return self.__call_api( ^^^^^^^^^^^^^^^^ File "D:\anaconda3\envs\private\Lib\site-packages\leetcode\api_client.py", line 159, in __call_api response_data = self.request( ^^^^^^^^^^^^^ File "D:\anaconda3\envs\private\Lib\site-packages\leetcode\api_client.py", line 422, in request return self.rest_client.POST( ^^^^^^^^^^^^^^^^^^^^^^ File "D:\anaconda3\envs\private\Lib\site-packages\leetcode\rest.py", line 338, in POST return self.request( ^^^^^^^^^^^^^ File "D:\anaconda3\envs\private\Lib\site-packages\leetcode\rest.py", line 250, in request raise ApiException(http_resp=r) leetcode.rest.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Date': 'Fri, 25 Apr 2025 07:56:10 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Content-Length': '8806', 'Connection': 'close', 'accept-ch': 'Sec-CH-UA-Bitness, Sec-CH-UA-Arch, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform, Sec-CH-UA, UA-Bitness, UA-Arch, UA-Full-Version, UA-Mobile, UA-Model, UA-Platform-Version, UA-Platform, UA', 'cf-mitigated': 'challenge', 'critical-ch': 'Sec-CH-UA-Bitness, Sec-CH-UA-Arch, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform, Sec-CH-UA, UA-Bitness, UA-Arch, UA-Full-Version, UA-Mobile, UA-Model, UA-Platform-Version, UA-Platform, UA', 'cross-origin-embedder-policy': 'require-corp', 'cross-origin-opener-policy': 'same-origin', 'cross-origin-resource-policy': 'same-origin', 'origin-agent-cluster': '?1', 'permissions-policy': 'accelerometer=(),autoplay=(),browsing-topics=(),camera=(),clipboard-read=(),clipboard-write=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=()', 'referrer-policy': 'same-origin', 'server-timing': 'chlray;desc="935c4843dba0a064"', 'x-content-type-options': 'nosniff', 'x-frame-options': 'SAMEORIGIN', 'Cache-Control': 'private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0', 'Expires': 'Thu, 01 Jan 1970 00:00:01 GMT', 'Vary': 'Accept-Encoding', 'Strict-Transport-Security': 'max-age=15552000; includeSubDomains; preload', 'Server': 'cloudflare', 'CF-RAY': '935c4843dba0a064-FRA'}) HTTP response body: Just a moment...
From the response, it appears that the request is being blocked by Cloudflare with a "Just a moment..." challenge page. It seems like the request is being flagged as suspicious or not originating from a real browser.
In my understanding, LeetCode may have implemented some protection mechanisms, but I was wondering if there’s anything I can do on my end to resolve this, or if there’s a recommended way to authenticate and avoid getting blocked.
I really enjoy using DebugBench and would love to continue using it effectively with LeetCode. Any guidance or suggestions you can provide would be greatly appreciated!
Thanks a lot for your time and support.
Hi syhzcx,
Thanks for using DebugBench!
Did you keep trying with the same instance? Some researchers facing the same error reported that changing test instance may help. Probably repeatedly submitting one case is prone to be viewed as malicious behavior by cloudflare.
Please tell me if it solves your problem. Thanks.
