System Info

transformers 4.20.0

Who can help?

@Narsil

Information

• [ ] The official example scripts
• [ ] My own modified scripts

Tasks

• [ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• [ ] My own task or dataset (give details below)

Reproduction

4.19.4

from transformers import pipeline

nlp = pipeline("text-classification")
nlp("happy", return_all_scores=True)

Output:

[[{'label': 'NEGATIVE', 'score': 0.0001246821484528482}, {'label': 'POSITIVE', 'score': 0.9998753070831299}]]

4.20.0

from transformers import pipeline

nlp = pipeline("text-classification")
nlp("happy", return_all_scores=True)

Output:

[{'label': 'NEGATIVE', 'score': 0.0001246821484528482}, {'label': 'POSITIVE', 'score': 0.9998753070831299}]

Running with top_k=None also produces a single list (only difference is labels are ordered by score desc).

from transformers import pipeline

nlp = pipeline("text-classification")
nlp("happy", top_k=None)

Output:

[{'label': 'POSITIVE', 'score': 0.9998753070831299}, {'label': 'NEGATIVE', 'score': 0.0001246821484528482}]

4.19.4 returns a list of lists when return_all_scores=True. 4.20.0 only produces a single list.

It looks like this logic changed the outputs in the __call__ method.

        if isinstance(args[0], str) and isinstance(result, dict):
            # This pipeline is odd, and return a list when single item is run
            return [result]
        else:
            return result

Previously, it was this:

        if isinstance(args[0], str):
            # This pipeline is odd, and return a list when single item is run
            return [result]
        else:
            return result

Expected behavior

When passing a single text element, the pipeline up to 4.20 would return a list. If this change was expected, I can work with it but figured it was worth bringing to your attention in case it wasn't intentional.

Just wanted to check in on this and see if it's considered an issue or the new way the pipeline works.

@davidmezzetti Didn't see this issue, but I didn't see the regression.

Should have been fixed here https://github.com/huggingface/transformers/pull/17906. Sorry it had time to ship with 4.20. It will be reverted back in the next release.

We are really keen to not break anything format wise while we are in v4. But for v5 harmonizing return types of pipelines is definitely on the agenda I want to push (some return lists, lists of lists, but we're not super consistent across pipelines.).

Thank you for responding and the update!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.