Environment Variables Lost on Container Restart Due to Missing Persistence of `env_vars.json`

[juanludataanalyst]

Description

When running Archon with the provided run_docker.py script, environment variables saved via the Streamlit UI are lost whenever the container is stopped and restarted. This issue arises from two key factors:

1. Container Removal: The run_docker.py script removes any existing container named archon-container before launching a new one, erasing all data stored within the container’s filesystem.
2. Incorrect Volume Mapping: The current docker run command maps the host’s ./workbench directory to /app/archon/workbench inside the container. However, the env_vars.json file—where environment variables are saved—is stored in /app/src/workbench/, which is not mapped to the host. As a result, env_vars.json resides in the container’s ephemeral filesystem and is lost when the container is removed.

How env_vars.json is Handled

The behavior of env_vars.json depends on whether it’s the first time environment variables are saved or subsequent interactions:

• Creation (First Time):

  • When a user saves an environment variable through the Streamlit UI, the save_env_var function in src/utils/utils.py creates env_vars.json if it doesn’t already exist.
  • The file’s location is determined by:

    current_dir = os.path.dirname(os.path.abspath(__file__))
    parent_dir = os.path.dirname(current_dir)
    workbench_dir = os.path.join(parent_dir, "workbench")
    env_file_path = os.path.join(workbench_dir, "env_vars.json")
    

    • Inside the container, __file__ resolves to /app/src/utils/utils.py, so workbench_dir becomes /app/src/workbench/, and the file is saved as /app/src/workbench/env_vars.json.
  • Since /app/src/workbench/ is not mapped to the host, this file is created in the container’s temporary filesystem.

• Search and Modification (Subsequent Times):

  • When the application retrieves or updates environment variables (e.g., via get_env_var or save_env_var), it looks for env_vars.json in /app/src/workbench/env_vars.json.
  • If the file exists (e.g., from a previous save within the same container session), it reads the JSON content, modifies it as needed (e.g., adding or updating variables), and writes it back to the same path.
  • This works during a single container session because the file persists in the container’s filesystem until the container is stopped or removed.

• Why It’s Lost:

  • The run_docker.py script stops and removes the archon-container each time it runs:

    subprocess.run(["docker", "rm", "-f", "archon-container"], check=False)
    

  • When the container is removed, its entire filesystem—including /app/src/workbench/env_vars.json—is discarded.
  • On the next run, a new container is created with a fresh filesystem. Since /app/src/workbench/ isn’t mapped to the host, the new container starts without env_vars.json, and the application behaves as if no variables were ever saved.

Current Setup in run_docker.py

The script includes a volume mapping, but it doesn’t align with where env_vars.json is stored:

volume_mount = f"{base_dir}/workbench:/app/archon/workbench"

• This maps the host’s ./workbench/ directory to /app/archon/workbench/ in the container.
• However, since env_vars.json is written to /app/src/workbench/, the volume mapping doesn’t preserve it. The file remains in the container’s ephemeral storage and is lost on restart.

---

Proposed Solution

To resolve this issue without altering the existing volume mapping in run_docker.py, we can modify src/utils/utils.py to store env_vars.json in the already mapped directory /app/archon/workbench/. This ensures the file persists on the host in ./workbench/ and survives container restarts.

Steps

1. Update src/utils/utils.py:

   • Replace the workbench_dir definition with:

     workbench_dir = "/app/archon/workbench"
     env_file_path = os.path.join(workbench_dir, "env_vars.json")
     

   • Ensure all functions interacting with env_vars.json (e.g., save_env_var, get_env_var, write_to_log) use this updated path. Example:

     import os
     import sys
     from supabase import Client, create_client
     from openai import AsyncOpenAI
     from dotenv import load_dotenv
     from datetime import datetime
     from functools import wraps
     from typing import Optional
     import streamlit as st
     import webbrowser
     import importlib
     import inspect
     import json
     
     sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
     load_dotenv()
     
     workbench_dir = "/app/archon/workbench"
     env_file_path = os.path.join(workbench_dir, "env_vars.json")
     
     def write_to_log(message: str):
         os.makedirs(workbench_dir, exist_ok=True)
         log_path = os.path.join(workbench_dir, "logs.txt")
         timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
         log_entry = f"[{timestamp}] {message}\n"
         with open(log_path, "a", encoding="utf-8") as f:
             f.write(log_entry)
     
     def get_env_var(var_name: str, profile: Optional[str] = None) -> Optional[str]:
         if os.path.exists(env_file_path):
             try:
                 with open(env_file_path, "r") as f:
                     env_vars = json.load(f)
                     current_profile = profile or env_vars.get("current_profile", "default")
                     if "profiles" in env_vars and current_profile in env_vars["profiles"]:
                         profile_vars = env_vars["profiles"][current_profile]
                         if var_name in profile_vars and profile_vars[var_name]:
                             return profile_vars[var_name]
                     if var_name in env_vars and env_vars[var_name]:
                         return env_vars[var_name]
             except (json.JSONDecodeError, IOError) as e:
                 write_to_log(f"Error reading env_vars.json: {str(e)}")
         return os.environ.get(var_name)
     
     def save_env_var(var_name: str, value: str, profile: Optional[str] = None) -> bool:
         os.makedirs(workbench_dir, exist_ok=True)
         env_vars = {}
         if os.path.exists(env_file_path):
             try:
                 with open(env_file_path, "r") as f:
                     env_vars = json.load(f)
             except (json.JSONDecodeError, IOError) as e:
                 write_to_log(f"Error reading env_vars.json: {str(e)}")
         if "profiles" not in env_vars:
             env_vars["profiles"] = {}
         if "current_profile" not in env_vars:
             env_vars["current_profile"] = "default"
         current_profile = profile or env_vars.get("current_profile", "default")
         if current_profile not in env_vars["profiles"]:
             env_vars["profiles"][current_profile] = {}
         env_vars["profiles"][current_profile][var_name] = value
         try:
             with open(env_file_path, "w") as f:
                 json.dump(env_vars, f, indent=2)
             return True
         except IOError as e:
             write_to_log(f"Error writing to env_vars.json: {str(e)}")
             return False
     
     # ... (rest of the file remains unchanged)
     

2. Run the Container:

   • Execute the existing run_docker.py script with no modifications:

     ./run_docker.py
     

   • The volume mapping (./workbench:/app/archon/workbench) in run_docker.py will persist env_vars.json to ./workbench/env_vars.json on the host.

3. Verification:

   • Save environment variables via the Streamlit UI (accessible at http://localhost:8501).
   • Check that ./workbench/env_vars.json appears in the project’s root directory on the host.
   • Stop and restart the container using run_docker.py. Confirm that the saved variables remain accessible in the UI.

Benefits

• Minimal Changes: Only requires modifying utils.py to align with the existing volume mount, avoiding changes to run_docker.py.
• Persistence: Ensures environment variables persist across container restarts by leveraging the current volume mapping.
• Simplicity: Maintains the existing setup without introducing additional complexity.

Alternative Consideration

If the preference is to store env_vars.json in /app/src/workbench/ (mapped to ./src/workbench/ on the host), the volume mount in run_docker.py could be updated to:

volume_mount = f"{base_dir}/src/workbench:/app/src/workbench"

However, since ./workbench:/app/archon/workbench is already in use and functional, adjusting utils.py to match this mapping is the simpler and more consistent approach.

[coleam00]

Thanks for this detailed bug report @juanludataanalyst! I'll respond to your PR.

[sid-newby]

same. spent the day trying to fix it. docker overwrites any fix i put in place.

[ayhlai]

Thanks for all the efforts you put in @coleam00 and @juanludataanalyst

Has this been fixed yet? I had the same issue.

[coleam00]

Thanks for submitting this issue! However I'm closing this now because the overhaul for Archon is coming soon so we need a blank slate for the repository. Some functionality from the old Archon will still apply to the new, so if you encounter this issue or something related with the new version of Archon, please feel free to reopen!