[FR] Trace "hidden" prompts

[alena-m]

Willingness to contribute

No. I cannot contribute this feature at this time.

Proposal Summary

When working with Mlflow Evaluation or AI agents, there are "hidden" system prompts that are not visible in the UI. For example, this prompt is not shown in mlflow UI. It's good to be able to track "hidden" prompts too - see them in the UI and see their token consumption. Ideally with toggle: show / don't show system prompts.

Motivation

What is the use case for this feature?

Work with frameworks that use prompts behind the scenes. Such as Mlflow Evaluation, Autogen, LangChain.

Why is this use case valuable to support for MLflow users in general?

It's useful to be able to track "hidden" prompts. Seeing them in the UI gives better visibility of what is happening behind the scenes. Seeing their token consumption allows better cost control.

Why is this use case valuable to support for your project(s) or organization?

We use mfllow Evaluation and agents that have multiple system prompts and we want to see the whole picture in one UI.

Why is it currently difficult to achieve this use case?

Because it's currently unavailable.

Details

No response

What component(s) does this bug affect?

• [ ] area/artifacts: Artifact stores and artifact logging
• [ ] area/build: Build and test infrastructure for MLflow
• [ ] area/deployments: MLflow Deployments client APIs, server, and third-party Deployments integrations
• [ ] area/docs: MLflow documentation pages
• [ ] area/examples: Example code
• [ ] area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
• [ ] area/models: MLmodel format, model serialization/deserialization, flavors
• [ ] area/recipes: Recipes, Recipe APIs, Recipe configs, Recipe Templates
• [ ] area/projects: MLproject format, project running backends
• [ ] area/scoring: MLflow Model server, model deployment tools, Spark UDFs
• [ ] area/server-infra: MLflow Tracking server backend
• [ ] area/tracking: Tracking Service, tracking client APIs, autologging

What interface(s) does this bug affect?

• [ ] area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
• [ ] area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
• [ ] area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
• [ ] area/windows: Windows support

What language(s) does this bug affect?

• [ ] language/r: R APIs and clients
• [ ] language/java: Java APIs and clients
• [ ] language/new: Proposals for new client languages

What integration(s) does this bug affect?

• [ ] integrations/azure: Azure and Azure ML integrations
• [ ] integrations/sagemaker: SageMaker integrations
• [ ] integrations/databricks: Databricks integrations

[BenWilson2]

Any thoughts on us potentially supporting this? @prithvikannan @dbczumar

[dbczumar]

@alena-m I agree that this should be represented as part of the LLM-as-a-judge information in the trace table produced by mlflow.evaluate(). I'll add the help wanted label. Please let us know if you'd like to reconsider and help contribute this!

[github-actions[bot]]

@mlflow/mlflow-team Please assign a maintainer and start triaging this issue.