[EPIC] AI Scientist 2 Issue cleanup

[erkinalp]

AI Scientist 2 is about to be released. Let's review and complete the issues for AI Scientist to avoid duplicate work.

[isCopyman]

When will it be released?

[erkinalp]

Soon, according to https://sakana.ai/ai-scientist-first-publication/

[feier-syh]

Could you send me the open source code of version 2? Thank you very much.

[conglu1997]

It is released here: https://github.com/SakanaAI/AI-Scientist-v2

[erkinalp]

The following issues from AI-Scientist v1 can be closed because they've been addressed in AI-Scientist v2 (recheck them)

Model Support Issues

• [ ] #196: Test with Grok 3
• [ ] #195: Test with DeepSeek-R1
• [ ] #184: Test with Kimi k1.5
• [ ] #121: Have anyone tried to replace the chatgpt model with DEEPSEEK?
• [ ] #101: Has anyone tried with open source LLM or gemini free usage?
• [ ] #100: Edit the code to be able using the hermes-3-llama-3.1-405b free version from openrouter
• [ ] #87: Support for huggingface llms
• [ ] #66: Add support for Grok-2
• [ ] #48: Can we add azure?
• [x] #211: Can I directly use the API of an AI model that is not on the list?

Template and Research Domain Issues

• [ ] #181: Updating NanoGPT with a "speed run" version
• [ ] #171: Template based on Img2Txt Diffusion finetuning?
• [ ] #159: About outputting other types of languages
• [ ] #149: Question: Adding a new template and getting accurate content for the paper
• [ ] #148: Add Prompt for topic Research with NanoGPT
• [ ] #142: Is it possible to generate ideas without running baseline (experiment.py)?
• [ ] #132: How to control the research subject?
• [ ] #110: How to run my own template?
• [ ] #70: Lift code restrictions and allow other subjects to be explored

Architecture and Design Issues

• [ ] #198: Calling AI scientists is overstatement
• [ ] #193: Inquiry Regarding Open-Ended Feedback Loop in AI Scientist
• [ ] #172: Is editing ideas.json and prompt.json enough?
• [ ] #147: Question: How are experiment results accessed when doing the paper writeup?
• [ ] #124: AI - Scientist for Non-Developers
• [ ] #117: Why Not Use The TeXLive.Net Server?
• [ ] #111: Use unsloth as a booster?
• [ ] #106: run_experiment() still seems to be just executing the original experiment.py file
• [ ] #85: For an existing idea, what exactly do i need to configure to run this project?
• [ ] #80: Sharing of AI-Scientist project and further automaton of sci. res. using the S.D.A.D. DAQ
• [ ] #19: A potential approach to enhance idea generation

API and External Service Issues

• [ ] #179: citationStyles key for OpenAlex API
• [ ] #155: rate_limit
• [ ] #133: Response Status Code: 429 - Too many requests
• [ ] #115: Tokens has exceeded your daily rate limit
• [ ] #104: S2_API_KEY for semanticscholar.org
• [ ] #84: https://www.semanticscholar.org/product/api#api-key-form", "code": "429"
• [ ] #61: Why set openai api key, show "Connection error", and Claude3.5 show "403"
• [x] #41: How to setup the api
• [ ] #18: Semantic Scholar requests limited

Error Handling and Bug Issues

• [ ] #154: The LLM did not conform to the edit format
• [ ] #151: litellm.BadRequestError
• [ ] #131: Stuck On Paper Generation Experiments
• [ ] #127: Review Error ! (file not found 'report.pdf')
• [ ] #126: How to comment out review paper innovative code
• [ ] #125: Failed to generate idea: 'str' object has no attribute 'choices'
• [ ] #118: The following error occurred
• [ ] #116: The process has been stuck at the retrieval phase for about an hour. Is this normal?
• [ ] #113: Checking novelty of idea 0: adaptive_block_size Response Status Code: 200 Response Content: {"totalHits":0,"limit":5,"offset":0,"results":[]}
• [ ] #108: When running in a Docker environment, the following log is output and there is no result
• [ ] #93: Execution error please!!
• [ ] #90: RuntimeError: probability tensor contains either inf, nan or element < 0
• [ ] #89: nanoGPT_lite sanity check...RuntimeWarning: Mean of empty slice
• [ ] #81: I get the following error python experiment.py --out_dir run_0
• [ ] #78: Always backing off
• [ ] #71: Process stops progressing after reaching "generating idea 2/2"
• [ ] #60: FileNotFoundError: [Errno 2] No such file or directory: 'DatasaurusDozen.tsv'
• [ ] #55: FileNotFoundError: [Errno 2] No such file or directory: '../../data/shakespeare_char/train.bin'
• [ ] #53: extract_json_between_markers doesn't handle responses missing json markers
• [ ] #51: No such file or directory: 'templates/nanoGPT_lite/run_0/final_info.json'
• [ ] #45: An error "The LLM did not conform to the edit format." reported and entering in an infinite loop when editting the template.tex
• [ ] #23: from aider.coders import Coder still can not be resolved

Environment and Setup Issues

• [ ] #200: Feature Request: Enhancing Accessibility and Efficiency by Hosting on Google Colab
• [ ] #186: Hardware/software requirements for repo
• [ ] #174: After trying several times installing it I get constantly reload window and WSL breaks

[BradKML]

@erkinalp pack each of these issues into clusters and re-start them in AI-Scientist v2. Assume v2 is a refactored rewrite instead of being a small update. And then each PR can rely on coding agents to enhace

[erkinalp]

@BradKML the listed are those that don't apply to v2

[BradKML]

Do you have a list that consistently applies between v1 and v2? Sorry for asking it again but I am not even sure where they are going with their new research project.

[erkinalp]

Here is a breakdown of the original issue list with a verification status based on the v2 codebase:

Model Support Issues

• [ ] #196: Test with Grok 3

Research Domain Issues

• [ ] #142: Is it possible to generate ideas without running baseline - The concept of a baseline still exists in v2, but the process is different.

Architecture and Design Issues

• [ ] #198: Calling AI scientists is overstatement
• [ ] #124: AI - Scientist for Non-Developers: - v2 is still a tool for users with a technical background.
• [ ] #117: Why Not Use The TeXLive.Net Server?
• [ ] #111: Use unsloth as a booster?

@BradKML The number of issues that also apply to v2 are suprisingly few.

[BradKML]

Thanks fo the good work, considering how they made ShinkaEvolve, I am not sure if they will v3 or update v2.

[erkinalp]

ShinkaEvolve doesn't have a good paper writer component, so the experimenter and coder of it could temporarily be combined with v2's literature reviewer and peer reviewer and v1's paper writer.