Langfuse flow traces lack information about token usage and cost, multiple traces get logged per one flow run

[rors41]

Bug Description

I've stumbled upon a weird behaviour when using Langfuse tracing:

Scenario 1: I setup connection to Langfuse via dotenv component or by setting the env variables in docker compose. For each flow run, I get a flow trace and additional trace for every generation call I make.

For example this simple flow had two calls to OpenAI and I got 3 traces in total:

I am not sure if those OpenAI-generation traces are even supposed to be there: Langfuse-Langflow integration docs have link to your video that shows Langfuse integration and the OpenAI-generations are not there, nor are the token counts or cost visible in the traces. However if you scroll a bit there is a picture that shows this trace.

Scenario 2: I setup connection to Langfuse via dotenv component or by setting the env variables in docker compose. Then I change this connection to different project by setting new secret key and public key.

The flow trace goes to the correct project, but the OpenAI-generation traces still go to the project that was configured first. The only solution I've found is to restart the container so all the traces go to the correct Langfuse project.

I've managed to kinda reproduce the multiple traces issue it with langfuse sdk - call to LLM gets logged even though its outside of decorated function and invoke call doesn't even have langfuse langchain callback handler set:

import os
from langchain_openai import ChatOpenAI
from langfuse.decorators import langfuse_context, observe

# if we import this, the openai generation will show up in langfuse
# if not there will only be a trace of test function as expected
from langfuse.openai import openai 

langfuse_context.configure(
    public_key=os.getenv("LANGFUSE_PUBLIC_KEY"),
    secret_key=os.getenv("LANGFUSE_SECRET_KEY"),
    host=os.getenv("LANGFUSE_HOST"),
    enabled=True,
)

@observe()
def test():
    pass

test()
x = ChatOpenAI(model="gpt-4o")
x.invoke("Hello world")
langfuse_context.flush()

I've checked the codebase and couldn't find this import so its likely caused by something different or its completely unrelated to this.

Couple of questions: What should I expect to see in Langfuse after running a flow that makes a call to LLM? Is it discouraged to change Langfuse project once Langflow is already running?

Reproduction

1. Run Langflow v1.1.1 and connect to Langfuse
2. Run a flow that contains LLM call
3. Check Langfuse

Expected behavior

Not sure what the expected behaviour is supposed to be, but it would be nice if there was only one trace with cost/token usage information.

Who can help?

No response

Operating System

Langflow v1.1.1 image & latest Langfuse image

Langflow Version

v1.1.1

Python Version

None

Screenshot

No response

Flow File

No response

[dosubot[bot]]

Langflow does not have specific documentation or code comments that explain the purpose of multiple traces for a single flow run or how token usage and cost information should be displayed in Langfuse traces. However, Langflow does have built-in functionality for updating environment variables, such as Langfuse project keys, without requiring a container restart. This is achieved through the LangfuseInstance class, which includes an update method that reinitializes the instance with updated settings ^[1].

Regarding your questions:

1. What should I expect to see in Langfuse after running a flow that makes a call to LLM?

   • The expected behavior regarding the visibility of token usage and cost in traces is not clearly documented. It seems that the integration might not be fully displaying this information as expected.

2. Is it discouraged to change Langfuse project once Langflow is already running?

   • While there are no specific guidelines or best practices documented for changing Langfuse projects or keys during runtime, the update method in Langflow allows for dynamic updates without restarting the container. However, if traces are still being sent to the initial project, it might indicate a limitation or issue with the current setup.

For further troubleshooting, you might want to ensure that the update method is being called correctly after changing the environment variables to see if it resolves the issue with traces being sent to the wrong project.

To continue talking to Dosu, mention @dosu.

[severfire]

I think it would be great to be able to get additional information from components about token usage, actually the information that is given by API - that would be mine feature request :-)

[dosubot[bot]]

Hi, @rors41. I'm Dosu, and I'm helping the langflow team manage their backlog. I'm marking this issue as stale.

Issue Summary:

• The issue involves Langfuse tracing, where multiple traces are logged for a single flow run, including unexpected OpenAI-generation traces.
• Traces continue to log to the initially configured project even after switching projects unless the container is restarted.
• The LangfuseInstance class has an update method for updating environment variables without a restart, but documentation on token usage and cost visibility is lacking.
• @severfire showed interest in a feature for additional token usage information, supported by @snova-jorgep.

Next Steps:

• Please confirm if this issue is still relevant to the latest version of the langflow repository. If so, you can keep the discussion open by commenting on this issue.
• Otherwise, this issue will be automatically closed in 7 days.

Thank you for your understanding and contribution!