`gpt-3.5-turbo` model cannot be used for summary with `map_reduce`

[conglei]

Currently, due to the constraint that the size of prompts has to be 1, gpt-3.5-turbo model cannot be used for summary.

    def _get_chat_params(
        self, prompts: List[str], stop: Optional[List[str]] = None
    ) -> Tuple:
        if len(prompts) > 1:
            raise ValueError(
                f"OpenAIChat currently only supports single prompt, got {prompts}"
            )
        messages = self.prefix_messages + [{"role": "user", "content": prompts[0]}]
        params: Dict[str, Any] = {**{"model": self.model_name}, **self._default_params}
        if stop is not None:
            if "stop" in params:
                raise ValueError("`stop` found in both the input and default params.")
            params["stop"] = stop
        return messages, params

Suggested change:

 async def _agenerate(
            self, prompts: List[str], stop: Optional[List[str]] = None
        ) -> LLMResult:

        if self.streaming:
            messages, params = self._get_chat_params(prompts, stop)
            response = ""
            params["stream"] = True
            async for stream_resp in await acompletion_with_retry(
                    self, messages=messages, **params
            ):
                token = stream_resp["choices"][0]["delta"].get("content", "")
                response += token
                if self.callback_manager.is_async:
                    await self.callback_manager.on_llm_new_token(
                        token,
                        verbose=self.verbose,
                    )
                else:
                    self.callback_manager.on_llm_new_token(
                        token,
                        verbose=self.verbose,
                    )
            return LLMResult(
                generations=[[Generation(text=response)]],
            )

        generations = []
        token_usage = {}
        for prompt in prompts:
            messages, params = self._get_chat_params([prompt], stop)
            full_response = await acompletion_with_retry(
                self, messages=messages, **params
            )
            generations.append([Generation(text=full_response["choices"][0]["message"]["content"])])
            #Update token usage
        return LLMResult(
            generations=generations,
            llm_output={"token_usage": token_usage},
        )

[joezhoujinjing]

Need to form a task group for the prompts, the proposed code will just loop through all requests.

[tikikun]

i try to change the code just like the fix, but not working

[joezhoujinjing]

i try to change the code just like the fix, but not working

It works for me, other than not parallelizing the prompts. What was your failure?

[joezhoujinjing]

Created PR: https://github.com/hwchase17/langchain/pull/1463

[dosubot[bot]]

Hi, @conglei. I'm Dosu, and I'm helping the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

Based on my understanding, the issue you opened addresses the limitation of the gpt-3.5-turbo model, where it can only use a single prompt for summary. It seems that joezhoujinjing suggested a code change that allows for multiple prompts to be used and even created a pull request to address the issue. However, tikikun mentioned that they tried the proposed fix but it didn't work for them.

Before we close this issue, we wanted to check with you if it is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you for your contribution to the LangChain repository!