[FR] Enhance mlflow.log_text() to support append functionality for continuous log updates.

[ankitagr01]

Willingness to contribute

No. I cannot contribute this feature at this time.

Proposal Summary

Introduce an append functionality in mlflow.log_text() to allow continuous updates to a single log file during model training or other MLflow experiments. This feature will enable users to efficiently track evolving log information without the need to create new log files each time. Even when overriding is creates error: 'file already exists'

We can do something like mlflow.log_text('text to be logged', filename, 'a'), where 'a' represents append mode.

Motivation

What is the use case for this feature?

The ability to append logs continuously to a single file is crucial for long-running experiments and iterative model training. It provides a streamlined approach for monitoring and debugging without generating excessive numbers of log files or artifacts, which can clutter the workspace. If I want to continuously log details (not only metrics) for each epochs/Step, I need to create a separate file.

Currently, MLflow lacks the flexibility to append to an existing log file, and users are forced to work around this limitation by generating multiple files or handling logging outside of MLflow, which is inefficient and cumbersome.

Why is this use case valuable to support for MLflow users in general?

This feature would significantly improve the user experience, particularly for large-scale projects where continuous logging is necessary.

Why is this use case valuable to support for your project(s) or organization?

My trainings are running in servers, and for anyone to view the logs, they can't usually log into the server. they want to see continuous logs in mlflow (connected to Azure ML workspace for my case). currently I am creating new file with timestamps and it creates 1000s of files. otherwise, I have to consolidate all logs in a local file and then dump it at the end. which is also not good approach.

Why is it currently difficult to achieve this use case?

Details

All other loggers (example wandb) have it in built and its very useful feature to have.

What component(s) does this bug affect?

• [X] area/artifacts: Artifact stores and artifact logging
• [X] area/build: Build and test infrastructure for MLflow
• [X] area/deployments: MLflow Deployments client APIs, server, and third-party Deployments integrations
• [ ] area/docs: MLflow documentation pages
• [X] area/examples: Example code
• [ ] area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
• [ ] area/models: MLmodel format, model serialization/deserialization, flavors
• [ ] area/recipes: Recipes, Recipe APIs, Recipe configs, Recipe Templates
• [ ] area/projects: MLproject format, project running backends
• [ ] area/scoring: MLflow Model server, model deployment tools, Spark UDFs
• [ ] area/server-infra: MLflow Tracking server backend
• [X] area/tracking: Tracking Service, tracking client APIs, autologging

What interface(s) does this bug affect?

• [ ] area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
• [ ] area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
• [ ] area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
• [ ] area/windows: Windows support

What language(s) does this bug affect?

• [ ] language/r: R APIs and clients
• [ ] language/java: Java APIs and clients
• [ ] language/new: Proposals for new client languages

What integration(s) does this bug affect?

• [ ] integrations/azure: Azure and Azure ML integrations
• [ ] integrations/sagemaker: SageMaker integrations
• [ ] integrations/databricks: Databricks integrations