Failed connector becomes inaccessible

[emerzon]

Hi,

We had one of our connectors stuck in a failure loop. This created a side effect of the attempt index becoming too large and making the connector inaccesible from the UI.

The culprit seems to be the code in: backend/danswer/server/documents/cc_pair.py

(A debug line was added to the code below)

    index_attempts = get_index_attempts_for_connector(
        db_session,
        cc_pair.connector_id,
    )
    logger.debug(f"index_attempts size: {len(str(index_attempts))} bytes")

Which have the following result: index_attempts size: 233556145 bytes

Since from the UI, this is returned as-is, there was over a half GB of data being sent to the browser when trying to access the connector page.

    response = CCPairFullInfo.from_models(
        cc_pair_model=cc_pair,
        index_attempt_models=list(index_attempts),
        latest_deletion_attempt=latest_deletion_attempt,
        num_docs_indexed=documents_indexed,
    )

Probably some limitation on the index_attempts size is required, and also limiting the number of records the UI attempts to retrieve.