Update on MLC Prompt

[iamollas]

Hey there,

I think that in this line: https://github.com/iryna-kondr/scikit-llm/blob/6b0d5f5466cb48a9622b6eb8a60dfe9dc0446b2d/skllm/openai/prompts.py#L29 there is a mistake. A new unknown instance can be classified to zero labels as well. However, the current implementation pushes ChatGPT to choose at least on label (category). It should be rephrased as suggested in the PR, or with something similar to this.

Best regards

[OKUA1]

Hi @iamollas,

To be honest this was more of an intentional behaviour. But maybe we need to re-evaluate the approach. Your fix, however, will not change the final output as even if the model returns no classes, a random class will be assigned during the output validation step.

I was thinking about an alternative idea: allow users to provide an additional hyperparameter, that would add an additional candidate label that could be something like "None of the proposed categories are applicable".

[iamollas]

@OKUA1 The last one seems to be the best idea indeed, but for evaluating the performance you have to make changes to the ground truth label vector (y). Many mll datasets contain instances with label vectors containing zeros. Therefore, I propose an alternative modification. Instead of using this code snippet:

https://github.com/iryna-kondr/scikit-llm/blob/6b0d5f5466cb48a9622b6eb8a60dfe9dc0446b2d/skllm/openai/prompts.py#LL30C1-L31C1

to have something like this: "a single key label and a value corresponding to a binary vector the same size as the number of categories with 0 or 1 values, where 0 indicates an unassigned category and 1 indicates an assigned category.

Consider this, and it's up to you!

Thanks for your answer!

[OKUA1]

@iamollas I believe this issue is somewhat related to the one you described. https://github.com/iryna-kondr/scikit-llm/issues/14

I guess it would be nice to find a uniform solution for both multi class and multi label classifiers.

I am going to close this PR as changing of the prompt might not be required anyway.