instructor icon indicating copy to clipboard operation
instructor copied to clipboard

Instructor multiclass classification is not providing proper result with gpt-3.5-turbo

Open gksoriginals opened this issue 1 year ago • 1 comments

  • [ ] This is an actually a bug report.
  • [x] I am not getting good LLM Results
  • [ ] I have tried asking for help in the community on discord or discussions and have not received a response.
  • [ ] I have tried searching the documentation and have not found an answer.

What Model are you using?

  • [x] gpt-3.5-turbo
  • [ ] gpt-4-turbo
  • [ ] gpt-4
  • [ ] Other (please specify)

Describe the bug Instructor text classification tutorial is not working. I was trying to run the multi-class classification code using this tutorial When I use gpt-3.5-turbo the output returns contain all labels instead of the right ones. Check the following code and output


client = instructor.patch(OpenAI())
def multi_classify(data: str) -> MultiClassPrediction:
    """Perform multi-label classification on the input text."""
    return client.chat.completions.create(
        model="gpt-3.5-turbo",
        response_model=MultiClassPrediction,
        messages=[
            {
                "role": "user",
                "content": f"Classify the following ticket: {data}",
            },
        ],
    )  # type: ignore

result = multi_classify("I didn't get bill.")
result.model_dump_json(indent=2)

# '{\n  "class_labels": [\n    "tech_issue",\n    "billing",\n    "general_query"\n  ]\n}'

To Reproduce Try the instructor tutorial mentioned above which use gpt-3.5-turbo and see the results of multiclass classification.

Expected behavior Ideally this should be the response

# '{\n  "class_labels": [\n    "billing"\n  ]\n}'

gksoriginals avatar Feb 08 '24 09:02 gksoriginals

I'll work on the example a bit it has to do with prompting

jxnl avatar Feb 09 '24 14:02 jxnl