LangfuseConnector: Include System Prompt in Trace

[lc-bo]

Is your feature request related to a problem? Please describe. In our pipeline, we're providing a custom system_prompt when initializing the OpenAIGenerator component. Unfortunately, this system prompt is not captured in the Langfuse trace. This pipeline is configured successfully with the LangfuseConnector and a trace is being generated with inputs and outputs of the various pipeline components.

Describe the solution you'd like See the custom system prompt captured in the Langfuse trace at the OpenAIGenerator step. Ideally the trace would capture all OpenAIGenerator inputs (other than the sensitive api key).

Describe alternatives you've considered Writing a custom OpenAIGenerator component that would relocate the system_prompt as input variable to .run() so that it would be captured as input to the Trace. Not ideal as this would require self-maintaining an otherwise perfectly good component.

Additional context Here's a screenshot of what the ideal output would look like showing both system and user prompts in the Generation step. This is from a Langfuse demo trace: https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/85693874-9ddb-4fd4-a386-0031933cb784?observation=59065b29-25c7-42f4-8026-63dfbf0862d0

[ggdupont]

Same need here, adding more things in langfuse tracer would really be welcomed.

[sjrl]

Hey @lc-bo I wanted to check in with you and ask if this is still a problem you are facing?

If so an alternative I could recommend is moving over to the ChatGenerator version so using ChatPromptBuilder and OpenAIChatGenerator. Using these components properly captures the system prompt in Langfuse and comes with other benefits such as being async enabled and able to use Tools.

[lc-bo]

Hello, @sjrl thanks for reaching out. We just recently extended the OpenAIGenerator to enable structured response using the example here and then also added a custom span handler from the recent update to this package. The span handler adds the system prompt to this generation span and also converts user and system prompts to the ChatMessage format so that these will render in the Langfuse interface. Still testing, but it's working pretty well so far.

def handle(self, span: LangfuseSpan, component_type: Optional[str]) -> None:
        if component_type in _SUPPORTED_GENERATORS:
            meta = span._data.get(_COMPONENT_OUTPUT_KEY, {}).get('meta')
            if meta:
                m = meta[0]
                span._span.update(usage=m.get('usage') or None, model=m.get('model'))
            input = span._data.get(_COMPONENT_INPUT_KEY, {})
            if input:
                user_prompt = input.pop('prompt', None)
                input = [input]
                input.append(ChatMessage.from_user(user_prompt).to_openai_dict_format()) if user_prompt else None
                system_prompt = span._data.get(_COMPONENT_OUTPUT_KEY, {}).get('system_prompt')
                if system_prompt:
                    input.append(ChatMessage.from_system(system_prompt).to_openai_dict_format())
                span._span.update(input=input)

[sjrl]

I'm glad that's working out for you!

Our long term goal is to move away from the Generators and use the ChatGenerators. Also for the reason you're pointing out where you have to convert the prompts into ChatMessages which natively works with our ChatGenerators. I can see what's missing for you is probably enabling structured response in the OpenAIChatGenerator before you could switch over.

[julian-risch]

We recommend moving from Generators to ChatGenerators. For the latter, the issue doesn't apply.