Connection error.

[paulacanva]

I'm having Connection error. if I try to evaluate with more than 1 thread (I'd expect RateLimit but not connection error). Using OpenAI directly, without dspy, I avoid errors with backoff, while leveraging asyncio to have concurrent calls to OpenAI and speed the inference process. The output of dspy doesn't give me much information to understand what's causing the error, if there's anything in my control to avoid this. Apparently, it's not RateLimit as this is the only sort of error handled with backoff here https://github.com/stanfordnlp/dspy/blob/main/dsp/modules/gpt3.py. What I'm doing is pretty simple, but evaluation is quite slow, taking hours, which makes the process quite slow.

elif config.DSPy_MODE == constants.DSPyMode.FEW_SHOT:
        teacher_model = dspy.OpenAI(
            model=config.COMPLETION_CONFIG["model_teacher"],
            max_tokens=config.COMPLETION_CONFIG["max_tokens"],
        )
        classificator_zero_shot = ThemeClassification(
            model_name=config.COMPLETION_CONFIG["model_student"], module=dspy.Predict
        )
        bootstrap_optimizer = BootstrapFewShot(
            max_bootstrapped_demos=8,
            max_labeled_demos=36,
            metric=response_accuracy,
            teacher_settings=dict(lm=teacher_model),
        )
        classificator_few_shot = bootstrap_optimizer.compile(
            classificator_zero_shot, trainset=train_data, valset=eval_data
        )
        logger.info("Evaluating model.")
        evaluator(classificator_few_shot, metric=response_accuracy)
        logger.info("Saving model.")
        classificator_few_shot.save(
            f"data/user_queries_processed_dspy_{config.DSPy_MODE.name}.json"
        )

The majority of the code is based on https://github.com/stanfordnlp/dspy/blob/main/examples/nli/scone/scone.ipynb and I get the same error using this notebook.

[arnavsinghvi11]

Hi @paulacanva , could you share the full error and stack trace if possible. Connection Error likely has to do with the configuration of the LM. Is config.COMPLETION_CONFIG["model_teacher"] a string?

[paulacanva]

I'm just doing:

 teacher_model = dspy.OpenAI(
            model=config.COMPLETION_CONFIG["model_teacher"],
            max_tokens=config.COMPLETION_CONFIG["max_tokens"],
        )

While my config is:

COMPLETION_CONFIG: dict[str, any] = {
    "max_tokens": 250,
    "temperature": 0.0,
    "model_teacher": "gpt-4",
    "model_student": "gpt-3.5-turbo",
    "presence_penalty": 0.0,
    "frequency_penalty": 0.0,
    "n": 1,
}

API key comes from env var.

I don't get a stack. Just a print in the middle of the tqdm progress bar saying that. And then it doesn't recover. It just happens with threads > 1 in the evaluator.

[arnavsinghvi11]

Could you attach the full ConnectionError here?

Additionally, are you configuring the LM using dspy.settings.configure(lm=...)?

[paulacanva]

As I said, I don't get anything that gives me more clues about the error, but it only happens when threads > 1. Once it happens, it doesn't recover.

class ThemeClassification(dspy.Module):
    def __init__(self, model_name: str, module: dspy.Module = dspy.Predict):
        super().__init__()

        model = dspy.OpenAI(
            model=model_name, max_tokens=config.COMPLETION_CONFIG["max_tokens"]
        )
        dspy.settings.configure(
            lm=model, trace=[], temperature=config.COMPLETION_CONFIG["temperature"]
        )

        logger.info(f"Using module {module}")
        self.generate_answer = module(ThemeSignature)

    def forward(self, prompt: str):
        return self.generate_answer(prompt=prompt)

[arnavsinghvi11]

Hi @paulacanva , it still seems to be like an error on the LM configuration end, potentially with connections or rate-limits (I found this issue to be potentially related to your OpenAI model configuration.

We will be working on a PR to enhance the error handling so the full stack trace can be outputted to help debug errors like these.

Couple things I would try is reducing the number of threads but still keeping n > 1 to see if that is compatible. or moving out the model initialization&configuration from the Module layer itself and having it defined externally. (If you need to configure different LMs for different stages of the pipeline, you can do with dspy.settings.context(lm=...) to configure this).