ExplainaBoard maintains the hyper-parameter features?

[pfliu-nlp]

After discussion with Kiril and Haris, the idea of enabling ExplainaBoard to record hyper-parameter features for each task comes out.

To achieve this, the potential workflow is:

• design a reasonable schema for hyper-parameter, e.g., shared hyper-parameters, or task-specific ones
• add this schema into the SysOutput class
• design interface for users to specify their hyper-parameters
  • SDK side
  • Web platform side

[neubig]

I think this is a good idea.

My suggestion would be that we have something simple like:

"system_details": {...}

where the system details are left completely underspecified for now (other than having to be expressed in json), but we gradually refine the schema as we see how people would like to use this field. This would allow us to implement this quickly without over-thinking things now.

In the SDK, people can specify system_details as a json file, and in the web interface this can be an optional upload field.