Expose chat system message

[RobinDev]

Hello @MaximeThoonsen

It it possible to easily debug to expose systemMessage in Chat ?

How to

// To add in LLPhant\ChatChatInterface
public function getSystemMessage(): Message;

// To add in each chat impement the previous interface
    
public function getSystemMessage(): Message {
        return $this->systemMessage;
    }

I can do a PR

[RobinDev]

Or directly in question answering via lastMessageSystemWithContext :

namespace LLPhant\Query\SemanticSearch;

use LLPhant\Chat\ChatInterface;
use LLPhant\Chat\Message;
use LLPhant\Embeddings\Document;
use LLPhant\Embeddings\EmbeddingGenerator\EmbeddingGeneratorInterface;
use LLPhant\Embeddings\VectorStores\VectorStoreBase;
use Psr\Http\Message\StreamInterface;

class QuestionAnswering
{
    /** @var Document[] */
    protected array $retrievedDocs;

    public string $systemMessageTemplate = "Use the following pieces of context to answer the question of the user. If you don't know the answer, just say that you don't know, don't try to make up an answer.\n\n{context}.";

    /**
     * @param null|callable $contextPreparater
     */
    public function __construct(public readonly VectorStoreBase $vectorStoreBase,
        public readonly EmbeddingGeneratorInterface $embeddingGenerator,
        public readonly ChatInterface $chat,
        private readonly QueryTransformer $queryTransformer = new IdentityTransformer(),
        private readonly RetrievedDocumentsTransformer $retrievedDocumentsTransformer = new IdentityDocumentsTransformer(),
        private   $contextPreparater = null
        )
    {
    }

    /**
     * @param  array<string, string|int>|array<mixed[]>  $additionalArguments
     */
    public function answerQuestion(string $question, int $k = 4, array $additionalArguments = []): string
    {
        $systemMessage = $this->searchDocumentAndCreateSystemMessage($question, $k, $additionalArguments);
        $this->chat->setSystemMessage($systemMessage);

        return $this->chat->generateText($question);
    }

    /**
     * @param  array<string, string|int>|array<mixed[]>  $additionalArguments
     */
    public function answerQuestionStream(string $question, int $k = 4, array $additionalArguments = []): StreamInterface
    {
        $systemMessage = $this->searchDocumentAndCreateSystemMessage($question, $k, $additionalArguments);
        $this->chat->setSystemMessage($systemMessage);

        return $this->chat->generateStreamOfText($question);
    }

    /**
     * @param  Message[]  $messages
     * @param  array<string, string|int>|array<mixed[]>  $additionalArguments
     */
    public function answerQuestionFromChat(array $messages, int $k = 4, array $additionalArguments = [], bool $stream = true): string|StreamInterface
    {
        // First we need to give the context to openAI with the good instructions
        $userQuestion = $messages[count($messages) - 1]->content;
        $systemMessage = $this->searchDocumentAndCreateSystemMessage($userQuestion, $k, $additionalArguments);
        $this->chat->setSystemMessage($systemMessage);

        // Then we can just give the conversation

        if ($stream) {
            return $this->chat->generateChatStream($messages);
        }

        return $this->chat->generateChat($messages);
    }

    /**
     * @return Document[]
     */
    public function getRetrievedDocuments(): array
    {
        return $this->retrievedDocs;
    }

    public function getTotalTokens(): int
    {
        if (! method_exists($this->chat, 'getTotalTokens')) {
            $chatClass = $this->chat::class;
            throw new \BadMethodCallException("Method getTotalTokens does not exist on the chat object of class {$chatClass}");
        }

        return $this->chat->getTotalTokens();
    }

    public string $lastSystemMessageWithContext = '';

    /**
     * @param  array<string, string|int>|array<mixed[]>  $additionalArguments
     */
    private function searchDocumentAndCreateSystemMessage(string $question, int $k, array $additionalArguments): string
    {
        $questions = $this->queryTransformer->transformQuery($question);

        $this->retrievedDocs = [];

        foreach ($questions as $question) {
            $embedding = $this->embeddingGenerator->embedText($question);
            $docs = $this->vectorStoreBase->similaritySearch($embedding, $k, $additionalArguments);
            foreach ($docs as $doc) {
                //md5 is needed for removing duplicates
                $this->retrievedDocs[\md5($doc->content)] = $doc;
            }
        }

        // Ensure retro-compatibility and help in resorting the array
        $this->retrievedDocs = \array_values($this->retrievedDocs);

        $this->retrievedDocs = $this->retrievedDocumentsTransformer->transformDocuments($questions, $this->retrievedDocs);

        $contextPreparater = $this->contextPreparater ?? [$this, 'prepareContext'];
        $context = $contextPreparater($this->retrievedDocs, $k);

        return $this->lastSystemMessageWithContext = $this->getSystemMessage($context);
    }

    /**
     * @param Document[] $documentList
     */
    private function prepareContext(array $documentList, int $k): string
    {
        $context = '';
        $i = 0;
        foreach ($documentList as $document) {
            if ($i >= $k) {
                break;
            }
            $i++;
            $context .= $document->content.' ';
        }

        return $context;
    }

    private function getSystemMessage(string $context): string
    {
        return str_replace('{context}', $context, $this->systemMessageTemplate);
    }
}

[MaximeThoonsen]

Hey @RobinDev , we can add a getSystemMessage() on the interface yes. With pleasure for the PR!

[menturion]

IMHO, this could be easily handled within the app itself (i.e. outside of LLPhant) via a (sub) system message stack.