SMAC limits for number of objectives and hyperparameters

[ViktorooReps]

I am exploring massively multi-task training and wonder if SMAC could handle each of the task objectives separately.

Here is my setup:

~20 tasks ~1-3 hyperparameters per task ~5 shared hyperparameters

Each task has its own objective, task-specific hyperparameters theoretically should influence only their corresponfing task objective (but most of the model is shared between tasks, so anything is possible).

In the end we have up to 65 hyperparameters with 20 objectives. Would SMAC be an appropriate tool for the job?

[alexandertornede]

Hi!

I cannot think of a way that SMAC would support this out of the box at the moment. It is also not really trivial to add this, as the optimization process would also need to consider when to optimize further on which of the tasks.

Nevertheless, it is a very interesting use case!

[ViktorooReps]

Wanted to add some context for potential discussion.

My use-case is training NER on multiple datasets with non-matching entity type sets (for example, OntoNotes + CoNLL 2003). I use architecture similar to the one proposed in https://aclanthology.org/P19-1441.pdf

I think it could be benefitial to tune task layers hyperparameters (for now all the parameters are shared between tasks).

[alexandertornede]

Thanks for giving more details on this! Like said, it looks like a very interesting use case, but I do not think that we will support something like this rather soon as we only have a limited amount of time working on SMAC, sorry.