yocto-gl
yocto-gl copied to clipboard
xelatihy
We'd like your feedback on MLflow Experiment Tracking
MLflow Experiments lets you easily track your experiments and compare runs. We're looking to further improve the tracking and comparison UX and would love to get your feedback on some prototypes we are exploring.
If you're interested in chatting with us and giving us feedback, please sign up here 🙌
@BenWilson2 @dbczumar @harupy @WeichenXu123 Please assign a maintainer and start triaging this issue.
In Table View, the metrics that are displayed are the ones from the last epoch. Would be nice to have the minimum / maximum achieved for that model instead of the last metric obtained.
In Table View, the metrics that are displayed are the ones from the last epoch. Would be nice to have the minimum / maximum achieved for that model instead of the last metric obtained.
Or even the number from a custom step?
One difficulty I find in the UI is it only loading 100 runs at a time. The charts in "Chart View" are only initially representative of a small portion of the dataset. One has to repeatedly click "load more" to see the charts fill out. It would be nice to be able to load all trials, or at least a larger number at once.
Another improvement would be if the Charts built in "Chart View" were persistent. As well as transferable to other experiments, since often I'm always building the same charts repeatedly.
Anyway, hope these recommendations are helpful. I enjoy using MLFlow, keep up the good work.
Bug - in "Chart View", a tooltip shows on mouseover, however clicking on the relevant model (sometimes) causes it to disappear. Therefore, the buttons on the tooltip are unreachable.
(it seems this is an intermittent bug. Tough to pin down a consistent way to reproduce but it seems to happen about 20-30% of the time)
Feedback - it would be great to be able to set a different sort metric for bar charts. "First N" seems to be the only option - would be great to get "Top N". We can also only select N from a list of [5, 10, 20, 100, 250, 500] - a "nice to have" would be the ability to set a custom value.
Feedback - add a "hide most recent run" button for the Parallel Coordinates chart - the most recent (currently training) run tends to populate this chart with null values, greatly reducing the resolution of the true parameter values.
Also, a related bug - filtering, e.g. by val_accuracy > 0.6, does not remove this run where val_accuracy is null.
Another feature request - some way to show calculated summary statistics, e.g. average accuracy across all runs. E.g. a new chart type, "Statistics" or "Table", which updates based on the SQL-style filter in the search box and can show the average of any metrics.
Thanks for sharing your this feedback @jmcapra. Would you be up for a brief chat about this issues and any other feedback on MLflow? For context, I'm a designer working on MLflow. You can book some time with me here.
Hi @ridhimag11, I'd be happy to. I've booked some time with you next week.
Another tiny bug: I have a run named "ConvNet - Augmented - FFT/Wavelet" but it is interpreted as a file path and just shows as "Wavelet" in some sections.
Another feature request: The ability to sort the z-level of different runs in the parallel coordinates plot (and line charts). It's not clear how they are currently sorted, however I think a better default would be to sort by the performance metric, so the best performing models are on top/in front. That way, it would be much easier to judge which parameter values are performing better. Currently, the only way I've been able to bring them to the front is by selecting by mouseover/click in the final section where they move from the last parameter value to the metric (top-down). Naturally, all of the lines between the other parameter values overlap and make it difficult to analyse performance from the bottom up.
how to remove the banner 'We’ve made several improvements to the new runs comparison experience.... If you prefer to use this experience, we'd love to know more.' ?
I can see the following experiment being run on localhost:5000 but there are no files saved at the given location. Experiment ID: 1 Artifact Location: /home/ubuntu/notebooks/07-project/training_tracking_registry/mlruns/1
What could be the possible issue ? I'm using mlflow 2.3.1 on python3.10.0
Thanks for this issue. Is there any reason why the plots are not displayed by default like the chat_view when comparing? It is pretty hard to open and analyse each metrics and impossible to compare metrics of the same run. Am I missing something?
Just upgraded to the version with chart view, and it's very cool and nice to be able to see comparisons side by side now :)
A bit of feedback:
More color diversity would be great, a lot of my charts end up looking like this by accident (two grays, two blues)
On a cold start of the chart page, a lot of charts are generated, would be convenient if there was an option to "clear the dashboard".
Would be very useful if selecting an experiment from the sidebar would highlight the corresponding chart elements, seeing as occlusion sometimes prevents clicking the element directly in the chart.
Another useful feature would be a paired zoom, e.g. focusing on the same step range across all charts.
It would be nice to be able to "Show Differences Only" in the Table View among the Parameters/Metrics as it was possible in the older versions (i.e. 1.23.1) to distinguish the runs at a glance.
I think it would be helpful to have a datasets section of the run comparison, and be able to see the dataset hashes in the comparison view so we can easily see what, if any, datasets have changed between runs!
+1 @claudiofernandez ; please add back "Show diffs". Very important feature in practice. @harupy
pls hide the artifactory location s3://aml....
I personally find it very useful to see where data is stored!
I find the new useless. It lacks the ability to select runs (there's only option to select 5, 10, 100 etc). Since I can't select runs to analyze it's of no use for me.
I find the new useless. It lacks the ability to select runs (there's only option to select 5, 10, 100 etc). Since I can't select runs to analyze it's of no use for me.
@emsi drive-by comment: you can use "Eye" icon in order to add/remove runs from the comparison.
@emsi drive-by comment: you can use "Eye" icon in order to add/remove runs from the comparison.
There are selctbox which are NOT honored when entering comparison and I end up in comparison with dozens to uncheck with eye now? So instead of checking 3 I have to un-eye 50? That's not convenient.
Hi, is it possible to add token numbers to all parts of prompts? Currently we can only see the total number of tokens, but in addition to the total number it would be useful to see the tokens for each part separately.
Is it possible to add the ability to change the row width in the Evaluation table? Sometimes the output is too long for cells, but I still want to compare them side by side.
Is it possible to automatically fill in variable values from the previous run when we do "Duplicate run" for Prompt Engineering? If we have many variables it is tedious to fill them again and again.
I think showing the min values of train_loss and val_loss in the main table would be more helpful.
I miss being able to see the datasets used in a specific run. I find it quite difficult to filter a dataset in order to find which runs used it. The previous run view (with a dataset dropdown) was friendlier.
Hi @mendelson - can you share a screenshot of what you're seeing regarding the missing datasets. AFAIK, you should still be able to see the dataset dropdown in the list view.