duckdb in read-only mode: database does not exist

[rodageve]

Bug Description

The following WARNING level errors are persistently reported in release 1.0.16

WARNING utils.py 166 migrate_transactions_from_monitor_service_to_database Error retrieving transactions from monitor service: IO Error: Cannot open database "/app/data/.cache/langflow/monitor.duckdb" in read-only mode: database does not exist

WARNING utils.py 30 migrate_messages_from_monitor_service_to_database Error retrieving messages from monitor service: IO Error: Cannot open database "/app/data/.cache/langflow/monitor.duckdb" in read-only mode: database does not exist

Reproduction

1. Refer to Astra production customers

Expected behavior

We shouldn't see these error logs when duckdb is no longer used

Who can help?

@nicoloboschi @zzzming @ogabrielluiz

Operating System

Astra docker image / 1.0.16.post4

Langflow Version

1.0.16.post4

Python Version

3.11

Screenshot

No response

Flow File

No response

[dosubot[bot]]

Hey @rodageve! I'm here to help you with any bugs, questions, or contributions. Let's tackle this issue together while we wait for a human maintainer.

To resolve the persistent WARNING level errors in release 1.0.16 related to DuckDB being in read-only mode and the database not existing, you can modify the new_duckdb_locked_connection function to handle the scenario where the database does not exist and ensure it is not opened in read-only mode when it needs to be created or modified.

Here is the updated code for the new_duckdb_locked_connection function:

@contextmanager
def new_duckdb_locked_connection(db_path: Union[str, Path], read_only=False):
    with worker_lock_manager.lock("duckdb"):
        # Ensure the database file exists before opening in read-only mode
        if read_only and not Path(db_path).exists():
            raise FileNotFoundError(f"Database file {db_path} does not exist.")
        
        with duckdb.connect(str(db_path), read_only=read_only) as conn:
            yield conn

This modification checks if the database file exists before attempting to open it in read-only mode. If the file does not exist, it raises a FileNotFoundError, which can be handled appropriately in your application logic. This should help resolve the warnings related to the database not existing and being in read-only mode.

Additionally, you can use the check_schema_health function from the DatabaseService class to ensure that the database schema is correctly set up, which can help prevent further issues:

def check_schema_health(self) -> bool:
    inspector = inspect(self.engine)

    model_mapping = {
        "flow": models.Flow,
        "user": models.User,
        "apikey": models.ApiKey,
        # Add other SQLModel classes here
    }

    # To account for tables that existed in older versions
    legacy_tables = ["flowstyle"]

    for table, model in model_mapping.items():
        expected_columns = list(model.model_fields.keys())

        try:
            available_columns = [col["name"] for col in inspector.get_columns(table)]
        except sa.exc.NoSuchTableError:
            logger.error(f"Missing table: {table}")
            return False

        for column in expected_columns:
            if column not in available_columns:
                logger.error(f"Missing column: {column} in table {table}")
                return False

    for table in legacy_tables:
        if table in inspector.get_table_names():
            logger.warning(f"Legacy table exists: {table}")

    return True

This function ensures that the table schema matches the model definition, and if not, it logs the missing tables or columns. This can help prevent schema-related warnings and errors ^[1][2][3].

To continue talking to Dosu, mention @dosu.