"Oracle triggered exit" for ak.StructuredDataClassifier

[LiangqunLu]

when I run

clf = ak.StructuredDataClassifier(max_trials=5, metrics = ["AUC", "accuracy"], seed = 123, objective = "val_AUC")
clf.fit(X, Y, epochs = 1000, validation_data = (valx, valy) , verbose=0 )

it always triggers "INFO:tensorflow:Oracle triggered exit", then the worst model is selected for the evaluation.

Apparently, I expected the autokeras to select the optimized parameters and have the best model for evaluation.

[john012343210]

INFO:tensorflow:Oracle triggered exit
C:\ProgramData\Anaconda3\lib\site-packages\numpy\core\_methods.py:151: RuntimeWarning: invalid value encountered in reduce
  ret = umr_sum(arr, axis, dtype, out, keepdims)
C:\ProgramData\Anaconda3\lib\site-packages\numpy\core\_methods.py:183: RuntimeWarning: invalid value encountered in reduce
  arrmean = umr_sum(arr, axis, dtype, keepdims=True)

don't know whether I am facing the same issue

[utkarshgupta137]

Facing the same issue. Get 'INFO:tensorflow:Oracle triggered exit' just after all trials complete. My code:

X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.8, random_state=0) # 116883 rows in X, y ak = StructuredDataRegressor(max_trials=1, seed=0) ak.fit(X_train, y_train, epochs=100) ak.evaluate(X_test, y_test) ak.export_model().save('ak')

In addition, I get the following when trying to save the exported model:

ak.export_model().save('ak') WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.iter WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_1 WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_2 WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.decay WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.learning_rate WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details. WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.iter WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_1 WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_2 WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.decay WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.learning_rate WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details. 2020-06-16 23:58:37.896026: W tensorflow/python/util/util.cc:319] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1786: calling BaseResourceVariable.init (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version. Instructions for updating: If using Keras pass *_constraint arguments to layers. Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/network.py", line 1008, in save signatures, options) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/save.py", line 115, in save_model signatures, options) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/saved_model/save.py", line 78, in save save_lib.save(model, filepath, signatures, options) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/saved_model/save.py", line 909, in save meta_graph_def, saveable_view, signatures, options.namespace_whitelist) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/saved_model/save.py", line 553, in _fill_meta_graph_def object_map, resource_map, asset_info = saveable_view.map_resources() File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/saved_model/save.py", line 251, in map_resources new_resource = obj._create_resource() File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/lookup_ops.py", line 1932, in _create_resource name=self._name) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_lookup_ops.py", line 1113, in mutable_dense_hash_table_v2 max_load_factor=max_load_factor, name=name) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/op_def_library.py", line 468, in _apply_op_helper preferred_dtype=default_dtype) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1280, in convert_to_tensor raise RuntimeError("Attempting to capture an EagerTensor without " RuntimeError: Attempting to capture an EagerTensor without building a function.

I can't view the summary of the exported model either:

model = ak.export_model() model.summary() Model: "model"

---

Layer (type) Output Shape Param #

input_1 (InputLayer) [(None, 6)] 0

---

Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/network.py", line 1310, in summary print_fn=print_fn) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/utils/layer_utils.py", line 226, in print_summary print_layer_summary(layers[i]) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/utils/layer_utils.py", line 184, in print_layer_summary fields = [name + ' (' + cls_name + ')', output_shape, layer.count_params()] File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 1607, in count_params return int(sum(np.prod(w.shape.as_list()) for w in self.weights)) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 1607, in return int(sum(np.prod(w.shape.as_list()) for w in self.weights)) AttributeError: 'TrackableWeightHandler' object has no attribute 'shape'

[utkarshgupta137]

Issue solved in the latest version.

[mcmar]

@utkarshgupta137 @haifeng-jin This issue is absolutely not solved in the latest version. I'm still getting this on 1.0.8. Can you please re-open this issue?

[jisho-iemoto]

Dear Martin-san, Thanks for your input.  However, I can’t understand what you mean about “Can you please re-open this issue?”Could you kindly tell me what I shall do with the article/issue? Thanks Jisho-Iemoto Windows 10 版のメールから送信 差出人: Mike Martin送信日時: 2020年9月12日 11:50宛先: keras-team/autokerasCC: Subscribed件名: Re: [keras-team/autokeras] "Oracle triggered exit" for ak.StructuredDataClassifier (#1046) @utkarshgupta137 @haifeng-jin This issue is absolutely not solved in the latest version. I'm still getting this on 1.0.8. Can you please re-open this issue?—You are receiving this because you are subscribed to this thread.Reply to this email directly, view it on GitHub, or unsubscribe. 

[jisho-iemoto]

Dear Martin-san, Sorry for my misunderstanding.  The issue in question is not in my scope.Please forget and discard the e-mail I sent you. Thanks, Jisho-Iemoto Windows 10 版のメールから送信 差出人: 上月 秀一送信日時: 2020年9月12日 11:55宛先: keras-team/autokeras件名: RE: [keras-team/autokeras] "Oracle triggered exit" for ak.StructuredDataClassifier (#1046) Dear Martin-san, Thanks for your input.  However, I can’t understand what you mean about “Can you please re-open this issue?”Could you kindly tell me what I shall do with the article/issue? Thanks Jisho-Iemoto Windows 10 版のメールから送信 差出人: Mike Martin送信日時: 2020年9月12日 11:50宛先: keras-team/autokerasCC: Subscribed件名: Re: [keras-team/autokeras] "Oracle triggered exit" for ak.StructuredDataClassifier (#1046) @utkarshgupta137 @haifeng-jin This issue is absolutely not solved in the latest version. I'm still getting this on 1.0.8. Can you please re-open this issue?—You are receiving this because you are subscribed to this thread.Reply to this email directly, view it on GitHub, or unsubscribe.  

[mcmar]

@jisho-iemoto okay, no problem

[mcmar]

Here's a minimal reproduction for the issue. code:

import autokeras as ak
import numpy as np

x = input_node = ak.Input()
x = ak.RNNBlock()(x)
x = ak.TemporalReduction()(x)
output_node = ak.RegressionHead(loss='cosine_similarity')(x)
reg = ak.AutoModel(
    inputs=input_node,
    outputs=output_node,
    overwrite=True,
    max_trials=1)
reg.fit(x=np.zeros((100, 50, 20)), y=np.ones((100, 20)), epochs=2)

output:

Trial 1 Complete [00h 00m 10s]
val_loss: -0.86031574010849

Best val_loss So Far: -0.86031574010849
Total elapsed time: 00h 00m 10s
INFO:tensorflow:Oracle triggered exit
Epoch 1/2
4/4 [==============================] - 0s 19ms/step - loss: -0.5827 - mean_squared_error: 0.9994
Epoch 2/2
4/4 [==============================] - 0s 18ms/step - loss: -0.8673 - mean_squared_error: 0.9979

[haifeng-jin]

Is it an error that exits the program completely? or it is keep running to the end and you can still export the found best model?