[Bug]: Configuration behavior is unpredictable

[khushvind]

Is there an existing issue for the same bug?

• [X] I have checked the troubleshooting document at https://docs.all-hands.dev/modules/usage/troubleshooting
• [X] I have checked the existing issues.

Describe the bug

When I change the model, and api_key in config.toml, OpenDevin still runs with the previous model, and api_key specified in the last session. To use the new model and key, I also need to specify them again in the UI.

Specifying it once should be enough.

Current OpenDevin version

0.8.2

Installation and Configuration

make build; make run

Model and Agent

Model: claude-3-5-sonnet Agent: CodeActAgent

Operating System

No response

Reproduction Steps

No response

Logs, Errors, Screenshots, and Additional Context

No response

[neubig]

We had a discussion on slack: https://opendevin.slack.com/archives/C06U8UTKSAD/p1722599155702189

Based on the discussion some of my personal suggestions for how we could improve the behavior now:

First, make consistent behavior of configuration:

1. Read from the defaults specified in config.py in the OpenDevin repo
2. Read from config.toml and override settings
3. (optional) Read from environment variables and/or command line and override settings

Here, 3. increases complexity, but it also could be handy because it allows overriding settings without writing to a file (e.g. for sensitive API keys).

Second, make UI-changed settings overwrite config.toml. This would allow the settings to be saved to and read from a consistent place.

[HenrikBach1]

Plz consider that your environment settings when using them in Dockerfile configuration use case, will be invalid as they are as of today in https://github.com/OpenDevin/OpenDevin/issues/3196.

[enyst]

First, make consistent behavior of configuration:

1. Read from the defaults specified in config.py in the OpenDevin repo
2. Read from config.toml and override settings
3. (optional) Read from environment variables and/or command line and override settings

Here, 3. increases complexity, but it also could be handy because it allows overriding settings without writing to a file (e.g. for sensitive API keys).

I think the current situation is, when running via:

• headless (main.py) - this works ✅
• cli.py - works ✅
• web UI - doesn't read any of it ❌
• resolver - not sure ❔
• external scripts - depends on scripts, evals explicitly do their own thing, except for custom llm configs, which they read from toml

Second, make UI-changed settings overwrite config.toml. This would allow the settings to be saved to and read from a consistent place.

Currently: (new from the time of this topic)

• UI saves settings in the filestore (locally, on local machine by default)
  • but it's a new format, a .json file
  • the toml file can be loaded from a particular location, but that feature is unused AFAIK
• has config handling in server module, not config module

Related:

• discussion and PR: https://github.com/All-Hands-AI/OpenHands/pull/6840#issuecomment-2669857237
• PR: https://github.com/All-Hands-AI/OpenHands/pull/6923

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[kripper]

I believe settings.json (UI config) should override config.toml (defaults), which are optional. It should not change the config.toml file content, just overwrite in memory.

But what is urgent is to implement the runtime agent to reload the settings when they are changed via UI or OH REST API.

MCP Servers should also be reloaded, injecting updated instructions to the context (https://github.com/All-Hands-AI/OpenHands/issues/9858).

Same for the search engine API key and other configurations.

[mamoodi]

@enyst is this going to be solved by the refactoring? Or will the refactoring enable us to do this easier?

[enyst]

I definitely hope it will be solved by the refactoring! It's a goal of mine to make sure of it. 😅

Related, for the refactoring:

• https://github.com/All-Hands-AI/OpenHands/issues/10947

[github-actions[bot]]

This issue is stale because it has been open for 40 days with no activity. Remove the stale label or leave a comment, otherwise it will be closed in 10 days.

[github-actions[bot]]

This issue was automatically closed due to 50 days of inactivity. We do this to help keep the issues somewhat manageable and focus on active issues.