Feature/audio artifact waveforms

[nnethery]

Related Issues/PRs

Resolve #6446

What changes are proposed in this pull request?

Add a waveform visualization and play/pause control to audio artifacts in the artifacts viewer in a specific experiment's run page.

How is this patch tested?

• [x] I have written tests (not required for typo or doc fix) and confirmed the proposed feature/bug-fix/change works.

Does this PR change the documentation?

• [x] No. You can skip the rest of this section.
• [ ] Yes. Make sure the changed pages / sections render correctly by following the steps below.

1. Click the Details link on the Preview docs check.
2. Find the changed pages / sections and make sure they render correctly.

Release Notes

Is this a user-facing change?

• [ ] No. You can skip the rest of this section.
• [x] Yes. Give a description of this change to be included in the release notes for MLflow users.

Audio file artifacts that are logged in experiments are now visualized with their waveforms in the artifact viewer. The interface also provides simple Play/Pause buttons and a timecode.

What component(s), interfaces, languages, and integrations does this PR affect?

Components

• [ ] area/artifacts: Artifact stores and artifact logging
• [ ] area/build: Build and test infrastructure for MLflow
• [ ] area/docs: MLflow documentation pages
• [ ] area/examples: Example code
• [ ] area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
• [ ] area/models: MLmodel format, model serialization/deserialization, flavors
• [ ] area/pipelines: Pipelines, Pipeline APIs, Pipeline configs, Pipeline Templates
• [ ] area/projects: MLproject format, project running backends
• [ ] area/scoring: MLflow Model server, model deployment tools, Spark UDFs
• [ ] area/server-infra: MLflow Tracking server backend
• [ ] area/tracking: Tracking Service, tracking client APIs, autologging

Interface

• [x] area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
• [ ] area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
• [ ] area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
• [ ] area/windows: Windows support

Language

• [ ] language/r: R APIs and clients
• [ ] language/java: Java APIs and clients
• [ ] language/new: Proposals for new client languages

Integrations

• [ ] integrations/azure: Azure and Azure ML integrations
• [ ] integrations/sagemaker: SageMaker integrations
• [ ] integrations/databricks: Databricks integrations

How should the PR be classified in the release notes? Choose one:

• [ ] rn/breaking-change - The PR will be mentioned in the "Breaking Changes" section
• [ ] rn/none - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
• [x] rn/feature - A new user-facing feature worth mentioning in the release notes
• [ ] rn/bug-fix - A user-facing bug fix worth mentioning in the release notes
• [ ] rn/documentation - A user-facing documentation change worth mentioning in the release notes

[github-actions[bot]]

@nnethery Thanks for the contribution! The DCO check failed. Please sign off your commits by following the instructions here: https://github.com/mlflow/mlflow/runs/7885677502. See https://github.com/mlflow/mlflow/blob/master/CONTRIBUTING.rst#sign-your-work for more details.

[nnethery]

[nnethery]

Also I sense that this feature may be too jarring, and would be a better experience if waveform-drawing is opt-in. I can work on a toggle that only shows on audio file artifacts and when this toggle is set, this preference will be remembered and subsequent loads of audio artifact previews will have their waveforms drawn

[nnethery]

UPDATED UX @harupy

Toggle not active

Toggle active

Toggle doesn't show when not selecting an audio file

[dbczumar]

@harupy Can you take another look?

[bushaev]

Hey, is there any plan on moving forward with this ? It's a very useful feature to have!

[bushaev]

@harupy could you take another look ?

[amie-roten]

I'd love this feature to be added as well, any status updates?

It would be nice to display the spectrogram too, but that may be more difficult/time-consuming.

[nnethery]

@amie-roten Probably even more helpful to display the spectrogram in most cases. I've been thinking about adding this so maybe I will. No updates from the maintainers, though.

[Reichenbachian]

Also bumping on this. Would love this feature to be added.

[kureta]

Hi. Are there any plans on merging this feature? Don't mean to clutter the comments but it seems abandoned.

[nnethery]

Hi. Are there any plans on merging this feature? Don't mean to clutter the comments but it seems abandoned.

@kureta I think at this point I'm not confident enough in the design for this component to look nice inside of MLflow. If anyone is able to offer a better design for this and a spectrogram, I will implement it and try again. I know there should be better support for audio artifacts but I think that's what's preventing this from getting to the finish line.

[kureta]

Hi. Are there any plans on merging this feature? Don't mean to clutter the comments but it seems abandoned.

@kureta I think at this point I'm not confident enough in the design for this component to look nice inside of MLflow. If anyone is able to offer a better design for this and a spectrogram, I will implement it and try again. I know there should be better support for audio artifacts but I think that's what's preventing this from getting to the finish line.

what would you think about using a component such as wavesurfer.js?

[github-actions[bot]]

Documentation preview for dd95953d928fc888d07934a9a72dde181ae314d4 will be available here when this CircleCI job completes successfully.

More info

• Ignore this comment if this PR does not change the documentation.
• It takes a few minutes for the preview to be available.
• The preview is updated when a new commit is pushed to this PR.
• This comment was created by https://github.com/mlflow/mlflow/actions/runs/7744981415.

[kgoba]

Anything that adds even the simplest audio preview/playback functionality would be golden. One simple feature is infinitely better than zero features, no need to push for spectrograms immediately.