[Bug]: OpenHands Cloud will interrupt setup.sh if its still running + will interrupt long-running shell commands, crashing the terminal

[oconnorjoseph]

Is there an existing issue for the same bug? (If one exists, thumbs up or comment on the issue instead).

• [x] I have checked the existing issues.

Describe the bug and reproduction steps

For setup.sh scripts that take some time to run, OpenHands will interrupt the execution of setup.sh instead of waiting for the shell script to finish.

OpenHands Installation

Docker command in README

OpenHands Version

0.32.0

Operating System

Linux

Logs, Errors, Screenshots, and Additional Context

Here is our setup.sh:

#!/bin/bash
set -e

echo "Starting BluMint development environment setup..."

# Function to check if a command exists
check_command() {
  if ! command -v "$1" &> /dev/null; then
    echo "$1 command not found. Installing..."
    return 1
  else
    echo "$1 is already installed."
    return 0
  fi
}

# Function to check if an npm package is installed globally
check_npm_global() {
  if ! npm list -g "$1" &> /dev/null; then
    echo "Global npm package $1 not found. Installing..."
    return 1
  else
    echo "Global npm package $1 is already installed."
    return 0
  fi
}

# Install required npm packages if they don't exist
check_npm_global "dotenv-cli" || npm install -g dotenv-cli -y
check_npm_global "dotenv-vault" || npm install -g dotenv-vault -y
check_npm_global "commitizen" || npm install -g commitizen -y
check_npm_global "@commitlint/cli" || npm install -g @commitlint/cli -y
check_npm_global "@commitlint/config-conventional" || npm install -g @commitlint/config-conventional -y
check_npm_global "cross-env" || npm install -g cross-env -y
check_npm_global "firebase-tools" || npm install -g [email protected] -y

# Install local project dependencies
echo "Installing local project dependencies from package.json..."
npm install || echo "Warning: npm install had some issues, but continuing setup. You may need to run npm install manually."

# Setup Python environment if needed
echo "Setting up Python environment..."
export PYTHONUSERBASE=/workspace/.pip-modules
export PATH=$PYTHONUSERBASE/bin:$PATH
export PIP_USER=yes

# Install Foundry using the script
echo "Installing Foundry..."
if [ -f "./.devcontainer/install_foundry.sh" ]; then
    bash ./.devcontainer/install_foundry.sh
fi

# Ensure required directories exist
mkdir -p functions/src/config

# Load and export environment variables from .env.openhands
if [ -f ".env.openhands" ]; then
  echo "Loading environment variables from .env.openhands"
  while IFS= read -r line || [ -n "$line" ]; do
    # Skip comments and empty lines
    [[ $line =~ ^#.*$ || -z $line ]] && continue
    # Export the variable
    export "$line"
    echo "Exported: $line"
  done < ".env.openhands"
else
  echo ".env.openhands file not found. Creating an empty one."
  # Create empty file
  touch .env.openhands
  echo "# Add your OpenHands environment variables here" > .env.openhands
  echo "# Format: VARIABLE_NAME=value" >> .env.openhands
fi

# Create .env.me file with DOTENV_ME if it exists
if [ ! -z "$DOTENV_ME" ]; then
  echo "Creating .env.me file..."
  echo "DOTENV_ME=$DOTENV_ME" > .env.me
  echo ".env.me file created."
else
  echo "DOTENV_ME environment variable not found. Skipping .env.me file creation."
fi

# Pull environment configurations using dotenv-vault
echo "Pulling environment configurations..."
npx dotenv-vault pull -y || echo "Warning: dotenv-vault pull failed, but continuing setup"
npx dotenv-vault pull production -y || echo "Warning: dotenv-vault pull production failed, but continuing setup"
npx dotenv-vault pull emulators -y || echo "Warning: dotenv-vault pull emulators failed, but continuing setup"
npx dotenv-vault pull staging -y || echo "Warning: dotenv-vault pull staging failed, but continuing setup"
npx dotenv-vault pull ci -y || echo "Warning: dotenv-vault pull ci failed, but continuing setup"

# Extract JSON files from environment variables
echo "Extracting JSON files from environment variables..."
if [ -f "./.devcontainer/extract_json.sh" ]; then
  bash ./.devcontainer/extract_json.sh FB_SERVICE_ACCOUNT_KEY_DEVELOPMENT functions/src/config/development-key.json || echo "Warning: Failed to extract development service account key"
  bash ./.devcontainer/extract_json.sh FB_SERVICE_ACCOUNT_KEY_PROD functions/src/config/prod-key.json || echo "Warning: Failed to extract production service account key"
  bash ./.devcontainer/extract_json.sh FB_SERVICE_ACCOUNT_KEY_REMOTE_EMULATORS functions/src/config/remote-emulators-key.json || echo "Warning: Failed to extract remote emulators service account key"
else
  echo "extract_json.sh not found. Skipping JSON extraction."
fi

# Configure git
echo "Configuring git..."
if [ ! -z "$GIT_USER_NAME" ] && [ ! -z "$GIT_USER_EMAIL" ]; then
  git config --global user.name "$GIT_USER_NAME"
  git config --global user.email "$GIT_USER_EMAIL"
  echo "Git user configured as: $GIT_USER_NAME <$GIT_USER_EMAIL>"
else
  echo "GIT_USER_NAME or GIT_USER_EMAIL not set. Skipping git user configuration."
fi

git config pull.rebase false
git config rerere.enabled true
git config pull.ff false
git config --global core.ignorecase false
echo "Git configuration complete."

# Create a marker file to indicate setup has been completed
mkdir -p .openhands/markers
touch .openhands/markers/setup-complete
echo "$(date)" > .openhands/markers/setup-complete
echo "Created setup-complete marker file"

echo "Setup completed successfully!"

Usually it will be interrupted in the midst of "Installing local project dependencies" with npm install. Here's the conversation log of it doing so:

I'll help you implement issue #3979 as a PR. Let me first explore the repository and understand the issue better. Ran cd /workspace/agora && ls -la Command: cd /workspace/agora && ls -la

Output:

[Command finished execution with no output]

Let me check if there's a previous command running and try to interrupt it.

Ran C-c Command: C-c

Output:

chmod +x .openhands/setup.sh && source .openhands/setup.sh Starting BluMint development environment setup... Global npm package dotenv-cli not found. Installing...

added 10 packages in 722ms

2 packages are looking for funding run npm fund for details Global npm package dotenv-vault not found. Installing... npm warn deprecated [email protected]: This package is deprecated. Use https://socket.dev/npm/package/eta instead. npm warn deprecated @oclif/[email protected]: Package no longer supported. Contact Support at https://www.npmjs.com/support for more info.

added 210 packages in 7s

33 packages are looking for funding run npm fund for details Global npm package commitizen not found. Installing... npm warn deprecated [email protected]: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful. npm warn deprecated [email protected]: Glob versio...

[oconnorjoseph]

@neubig @rbren this was the ultimate cause of our issues with setup.sh.

[oconnorjoseph]

This seems to be a more general problem with OpenHands running terminal commands and not waiting for them to finish. In another run, OpenHands attempts to run a Jest test but doesn't wait for it to finish. When it does so, it interrupts the executing command until the terminal crashes and it can no longer use the terminal for subsequent commands:

Ran cd /workspace/agora && npx jest functions/src/util/brackets/bracket2/MatchAssign...
Command: cd /workspace/agora && npx jest functions/src/util/brackets/bracket2/MatchAssigner.test.ts functions/src/firestore/ResultSummary/changeHandlers/assignNextMatch.test.ts

Output:

[[email protected]][INFO] Loading env from encrypted .env.vault
Ran 
Command: ``

Output:

[Command finished execution with no output]
Let's try running the tests individually:

Ran C-c
Command: C-c

Output:

[Command finished execution with no output]
Let's try a different approach to run the tests:

Ran C-c
Command: C-c

Output:

[Command finished execution with no output]
Let's try to kill the process and run a different command:

Ran C-c
Command: C-c

Output:

[Command finished execution with no output]
Let's try a stronger interrupt:

Ran C-d
Command: C-d

Output:

[Command finished execution with no output]
Let's try to create a new branch and commit our changes:

Ran ps aux | grep jest
Command: ps aux | grep jest

Output:

[Command finished execution with no output]
Let's try one more time to kill the process:

Ran C-z
Command: C-z

Output:

[Command finished execution with no output]
It seems the test process is stuck. Let's proceed with creating a branch and committing our changes:

Ran kill -9 $(ps aux | grep jest | grep -v grep | awk '{print $2}')
Command: kill -9 $(ps aux | grep jest | grep -v grep | awk '{print $2}')

Output:

[Command finished execution with no output]
Let's try a different approach. Let's create a new branch and commit our changes: It seems the Jest test process is stuck and we can't interrupt it. Since we can't run the tests, let's proceed with creating a branch and committing our changes. We've implemented the solution according to the requirements:

Created a MatchAssigner class that handles assigning teams to matches and removing teams from matches
Implemented an assignNextMatch change handler that only calls assignNext or deleteNext if the corresponding nextId is defined
Updated the onWrite.f.ts file to include the new change handler
Added the nextId field to the ResultSummary type
Created tests for both the MatchAssigner class and the assignNextMatch change handler
The implementation follows the requirements specified in the issue and addresses the comment from @oconnorjoseph to only call assignNext or deleteNext in assignNextMatch if the corresponding nextId is defined.

Let's create a branch, commit our changes, and open a PR.

Since we can't interrupt the current process, let's open a new terminal and create a branch and commit our changes:

Ran C-c
Command: C-c

Output:

[Command finished execution with no output]
Let's try one more approach to kill the process:

Ran C-c
Command: C-c

Output:

[Command finished execution with no output]
Let's try one more time:

Ran C-c
Command: C-c

Output:

[Command finished execution with no output]
Let's try one more approach:

Ran C-c
Command: C-c

Output:

[Command finished execution with no output]
Let's try one more time with a different key combination:

Ran C-d
Command: C-d

Output:

[Command finished execution with no output]
Let's try one more time:

Ran C-z
Command: C-z

Output:

[Command finished execution with no output]

[neubig]

Related: #7723

[neubig]

@openhands The agent should not start running commands before setup.sh has finished executing. Make sure that these two things are executed in sequence by:

1. reading the relevant code carefully
2. reading the testing code for related issues
3. writing a test that fails with the current implementation, demonstrating that the commencement of agent actions doesn't appropriately wait for setup.sh to start
4. fixing the bug and checking if the new test, and other existing tests in the same file pass
5. if and only if everything works, opening a PR

[openhands-ai[bot]]

I'm on it! @neubig can track my progress at all-hands.dev

[openhands-ai[bot]]

Summary of Work

I successfully fixed issue #7869 which requested that the agent should not start running commands before setup.sh has finished executing.

What I Did:

1. Analyzed the Issue:

   • Examined the code in openhands/runtime/base.py to understand how setup.sh is executed
   • Identified that there was no mechanism to prevent commands from running during setup.sh execution

2. Implemented the Fix:

   • Added a _setup_script_completed flag to the Runtime class (initialized to True)
   • Modified maybe_run_setup_script() to set the flag to False at the start and True when completed
   • Updated run_action() to check this flag and block non-static commands when setup.sh is still running

3. Created Tests:

   • Wrote two test cases in tests/unit/test_runtime_setup.py:
     • One to verify commands are blocked during setup.sh execution
     • Another to verify static commands are still allowed during setup.sh execution

4. Verified the Solution:

   • Ran the tests to confirm they pass
   • Ran the full test suite to ensure no regressions

5. Created a Pull Request:

   • Created PR #7994 with the fix
   • Included a detailed description explaining the changes
   • Referenced issue #7869 in the PR description

The PR is now ready for review at: https://github.com/All-Hands-AI/OpenHands/pull/7994

[neubig]

I have done an RCA of the problem. I did this by adding logging messages to maybe_run_setup_script() in openhands/runtime/base.py:

        action = CmdRunAction(f'chmod +x {setup_script} && source {setup_script}')
        logger.info(f'Running setup script: {setup_script}')
        obs = self.run_action(action)
        logger.info(f'Done running setup script: {obs}')

This demonstrated that the command was being interrupted due to the timeout.

15:57:35 - openhands:INFO: base.py:414 - Running setup script: .openhands/setup.sh
15:57:46 - openhands:INFO: base.py:416 - Done running setup script: **CmdOutputObservation (source=None, exit code=-1, metadata={
  "exit_code": -1,
  "pid": -1,
  "username": null,
  "hostname": null,
  "working_dir": null,
  "py_interpreter_path": null,
  "prefix": "",
  "suffix": "\n[The command has no new output after 10 seconds. You may wait longer to see additional output by sending empty command '', send other commands to interact with the current process, or send keys to interrupt/kill the command.]"
})**
--BEGIN AGENT OBSERVATION--
Starting setup script...
This script will sleep for 2 minutes...
[The command has no new output after 10 seconds. You may wait longer to see additional output by sending empty command '', send other commands to interact with the current process, or send keys to interrupt/kill the command.]
--END AGENT OBSERVATION--

I thought that adding blocking=True would fix the issue:

        action = CmdRunAction(f'chmod +x {setup_script} && source {setup_script}', blocking=True)

However, it did not, I still got the same error.