[Feature Request] Parallel Categories Plot

[austinmw]

(Apologies for creating multiple recent GitHub issues, this is the last one, I promise!)

I took the DataFrame from my experiment results and used Plotly's plotly.express.parallel_categories plot to visualize hyperparameter interactions, dropping any features that only have one unique value. This is an interactive plot, and you can wrap it in a function that refreshes periodically when new data is available:

This has been super useful for myself, so I thought that it may be useful to others as well if it were added as a plotting capability to the library? Although I'd understand if it's not desirable to add another dependency. Just thought I'd share!

[geoalgo]

(Apologies for creating multiple recent GitHub issues, this is the last one, I promise!)

Not at all! Your feedback and suggestions are highly appreciated :-)

It would certainly be helpful to add more visualization in general (for now we just have the best plot over time that is built-in).

As you mention, adding too many dependencies is something we generally want to be watchful about. That being said, adding extra-dependencies is fine, as long as they are optional.

We have this folder to store benchmarking functionalities, perhaps this tool could be added here?

[austinmw]

Thanks, that would be great! Here's the quick and dirty implementation I used:

import os
import pandas as pd
from pathlib import Path
import shutil
import json
import time
import boto3
import pickle
import logging
from typing import Optional
import matplotlib.pyplot as plt
import plotly.express as px
from IPython.display import display, clear_output
from botocore.exceptions import ClientError
import sagemaker
from sagemaker.debugger import TensorBoardOutputConfig, CollectionConfig, DebuggerHookConfig, ProfilerConfig
from syne_tune.backend.sagemaker_backend.sagemaker_utils import get_execution_role
from syne_tune.backend import LocalBackend, SageMakerBackend
from syne_tune.search_space import loguniform, uniform, finrange, randint, choice
from syne_tune.optimizer.schedulers.hyperband import HyperbandScheduler
from syne_tune.stopping_criterion import StoppingCriterion
from syne_tune.tuner import Tuner
from syne_tune.remote.remote_launcher import RemoteLauncher
from syne_tune.experiments import experiment_path, download_single_experiment, ExperimentResult
from syne_tune.constants import SYNE_TUNE_FOLDER

pd.options.display.max_columns = 100
pd.options.display.max_rows = 100


# Slightly modified load_experiment
def load_experiment(
        tuner_name: str,
        experiment_name: str = None,
        download_if_not_found: bool = True,
) -> ExperimentResult:
    """ Loads an experiment from the local cache or from S3.
    
    :param tuner_name: name of a tuning experiment previously run
    :param download_if_not_found: whether to fetch the experiment from s3 if not found locally
    :return:
    """

    path = experiment_path(tuner_name)

    metadata_path = path / "metadata.json"
    try:
        with open(metadata_path, "r") as f:
            metadata = json.load(f)
    except FileNotFoundError:
        logging.info(f"experiment {tuner_name} not found locally, trying to get it from s3.")
        if download_if_not_found:
            download_single_experiment(tuner_name=tuner_name, experiment_name=experiment_name)
        metadata = None
    try:
        if (path / "results.csv.zip").exists():
            results = pd.read_csv(path / "results.csv.zip")
        else:
            results = pd.read_csv(path / "results.csv")
    except Exception:
        results = None
    try:
        tuner = Tuner.load(path)
    except FileNotFoundError:
        tuner = None
    except Exception:
        tuner = None

    return ExperimentResult(
        name=tuner.name if tuner is not None else path.stem,
        results=results,
        tuner=tuner,
        metadata=metadata,
    )


def get_tuner_results(tuner_name, experiment_name=None, width=None, height=900, 
                      refresh_rate=60*3, return_df=False):
    """ Get results from a tuner.
    If plotly graph does not display, see: https://plotly.com/python/getting-started/#jupyterlab-support
    """

    while True:
        cont = False
        
        # Clear cached data
        clear_output(wait=True)
        syne_local_path = os.path.join(str(Path.home()), SYNE_TUNE_FOLDER)
        if os.path.exists(syne_local_path):
            logging.info('clearing local syne tune cache...')
            shutil.rmtree(syne_local_path)

        # Download data
        tuning_experiment = load_experiment(tuner_name, experiment_name)
        if tuning_experiment.metadata is None:
            clear_output(wait=True)
            tuning_experiment = load_experiment(tuner_name, experiment_name)

        # Get metadata
        metadata = tuning_experiment.metadata

        try:
            # Get first metric
            metric = metadata['metric_names'][0]

            # Get best config
            best_config = tuning_experiment.best_config()

            cont = True
            clear_output(wait=True)

        except Exception as e:
            clear_output(wait=True)
            print('Waiting for tuning information to be logged...')

        if cont:
            
            print('tuner_name:', tuner_name)
            print('experiment_name:', experiment_name)
            
            # Print metadata
            print(f'Metadata:\n{json.dumps(metadata, indent=4)}')

            # Filter dataframe
            results = (tuning_experiment.results.sort_values(by=metric, 
                        ascending=False).drop_duplicates('trial_id'))
            trials = tuning_experiment.results.trial_id.unique()
            #print(f'Trial IDs: {sorted(trials)}')
            keep_cols = []
            for c in results.columns:
                if ((metric in c) or
                    ('config_' in c) or
                    ('epoch' in c) or
                    ('st_status' in c) or
                    ('trial_id' in c) or
                    ('st_decision' in c)):
                    keep_cols.append(c)
            results = results[keep_cols]
            for col in results.columns:
                if len(results[col].unique()) == 1:
                    results.drop(col, inplace=True, axis=1)
            results.reset_index(drop=True, inplace=True)
            display(results)

            # Plot basic performance
            tuning_experiment.plot()

            # Parallel Categories Diagram
            try:
                fig = px.parallel_categories(results,
                    color=metric,
                    title='Syne Tune Parameters vs. Metrics',
                    color_continuous_scale=px.colors.diverging.Portland,
                    height=height, width=width,
                )
                fig.show()
            except:
                print(f'Waiting for metric {metric} information to be logged...')

            if return_df:
                return results
                #return tuning_experiment, results

        time.sleep(refresh_rate)

And then,

# Run helper function to get HPO results
get_tuner_results(tuner.name, hpo_experiment_name, refresh_rate=60*3)

[geoalgo]

Would you be interested in contributing this? (we have this folder that can be used for visualization scripts for instance)

To be clear, we would probably need some small changes (for instance having general columns names as "epoch" may not be always present).

[mseeger]

@austinmw Any news on this one? We'd love to have this in Syne Tune.

For now, the best place would be in benchmarking/nursery. Once it is there, users can already use it.

[austinmw]

Hey, I'll try to take a stab at this within the next week.

[mseeger]

Thanks!