AutoModel: How does greedy tuner and max_max_trials together perform / run / work? Existing bug?

[cordeirojoao]

Bug Description

I have performed a Customized Search Space using AutoModel for a classification problem. For the Automodel I have used all 4 tuners: greedy, bayesian, hyperband, random, and max_trials=400. For all tuners, 400 trials were performed but not for greedy. On greedy I have experimented the end of the search with a random number of trials (eg: 93, 117, 92) and just once it achieved the desired 400 trials.

Can you please clarify if this is a bug or if not how greedy tuner works / behaviour?

Bug Reproduction

input_node = ak.Input() #shape=(5,1) output_node = ak.ConvBlock(separable=False)(input_node) output_node = ak.ClassificationHead(num_classes=2, multi_label=True)(output_node)

auto_model = ak.AutoModel(inputs=input_node, outputs=output_node, overwrite=True, max_trials=400, tuner='greedy')

Data used by the code: 1D data. On this case we were using a 1D CNN. Data it's private and can be shared :(

Expected Behavior

I understand that with max_trials "The search may finish before reaching the max_trials" but I was not expecting that can happen with greedy as it virtuals explores all possible combinations and those would run for a very long number of trials. I have a search on AutoKeras documentation for clarification on how greedy tuner works, but I was not able to get that info. Can you please clarify if this is a bug or if not, how greedy tuner works / behaviour?

Setup Details

• OS type and version: Ubuntu 18.04
• Python: 3.7.7
• autokeras: 1.0.10
• keras-tuner: greedy
• scikit-learn: 0.22.1 (its should be irrelevant as I'm using CNNs)
• numpy: 1.18.5
• pandas: 1.1.4
• tensorflow: 2.3.0

[haifeng-jin]

It is probably because it is a defect of the algorithm. It prefers exploitation over exploration. For a small search space, it cannot find a new set of hps.

[koh-joshua]

I had similar issues with AutoModel using the default greedy tuner. This is for 2D image inputs for regression task. It will complete up to ~110 max trials but no more than that. Was wondering what is happening until I saw this post. I will try 'bayesian' for now.

[koh-joshua]

Unfortunately, when I set to 'bayesian', it crashed at only trial #8 with "ValueError: NAN, value too big for float64". Update: Provided a quick fix for the NAN error. However, I have noticed that Bayesian takes 3 x longer to train compared to greedy. In addition, the performance may not be better or even worse compared to greedy (up to 100 trials). The only problem with greedy is the premature termination, which is quite annoying. Different random seed seems to lead to different max trials being completed in greedy tuner, which is erratic and unstable in my opinion. There's definitely merit in keeping greedy as it is fast and able to provide good performance but work needs to be done to improve it.