Custom Analysis Todo List

[qjiang002]

Custom features

Now users can specify custom features in either the output json file or a global config file. Potential improvements are:

• Only json output file can support defining custom features and custom analysis, and tsv file cannot. For consistency, it may be better to separate the custom analysis settings from the output file and put them into a json config file.
• Support uploading custom analysis config file.

Custom functions

[unsupported] we also aim to allow users to specify feature function so that the analysis could be performed more flexibly.

[odashi]

@qjiang002 Good point. Moreover, the FileLoaderMetadata mechanism reads the entire output file to load the metadata, which is obviously redundant. Decoupling the metadata file from the output may also improve this efficiency. https://github.com/neulab/ExplainaBoard/blob/37145588a75b34cc9b9f5feb82a76f23bf692c95/explainaboard/loaders/file_loader.py#L177