Yusuke Oda comments

Results 232 comments of


                                            Yusuke Oda

AnalysisLevel does not own distinctive information

I also think that the current AnalysisLevel can become an attribute of the Processor at initialization, or we can implement separate Processors for different AnalysisLevels, to avoid aditional control flow...

Saving `is_training_set_available` in `sys_info` during `get_overall_statistics()`

@OscarWang114 Thanks for reporting the issue! First, could `skip_failed_analyses=True` in `Processor.process` be a quick fix, or does not it satisfy the use case? I also agree with having more specific...

Centralized type registry

Still RFC: @neubig @pfliu-nlp

Centralized type registry

I'm planning to apply this change after the all manual serialization was replaced by the `serialization` subpackage. #419

Centralized type registry

Remaining tasks: The ultimate goal of this is to serialize `SysOutputInfo` by the `serialization` subpackage. It requires to implement the `Serializable` interface for following classes: - All classes in `analyses`...

list[(name, Thing)] to dict[name, Thing], and remove MetricConfig and directly store configs in Metric

`MetricConfig` is stored in `MetricResult` too, but it seems this information is not necessarily required. There is also `Performance` that has almost the same information with `MetricResult`, and the only...

list[(name, Thing)] to dict[name, Thing], and remove MetricConfig and directly store configs in Metric

Potential change would be: - Remove MetricConfig, store every information necessary to perform metrics to Metric. - Integrate `MetricResult` and `Performance` into a single class: ```python class MetricResult: metric_name: str...

list[(name, Thing)] to dict[name, Thing], and remove MetricConfig and directly store configs in Metric

> When we merge MetricConfig into Metric, I think we will have two names for each Metric class: metric_name and metric_class_name. It looks natural to me, though it also looks...

list[(name, Thing)] to dict[name, Thing], and remove MetricConfig and directly store configs in Metric

@pfliu-nlp I think we can first implement `dict[str, MetricConfig]` and then come back to this issue. (Generally I recommend `dict[key, something]` over `list[(key, something)]`)

list[(name, Thing)] to dict[name, Thing], and remove MetricConfig and directly store configs in Metric

I found that `EaaSMetric` has some abuse of the `name` field: it is used to specify external metric. https://github.com/neulab/ExplainaBoard/blob/215785c7f8ad938ae74c47b398ca82419ecc12a5/explainaboard/metrics/eaas.py#L107 `EaaSMetricConfig` should have a field to specify this value (it may...