Bug: predict doesn't work when final_fit is not performed

[psaks]

Bug Description

When final_fit is not performed, there are downstream issues with predict.

The issue is in autokeras/engine/tuner.py in AutoTuner.search when the else branch is evaluated in line 220.

            model = self.get_best_models()[0]
            history = None
            pipeline = pipeline_module.load_pipeline(
                self._pipeline_path(self.oracle.get_best_trials(1)[0].trial_id)
            )

Then when predict is subsequently performed there are first a bunch of warnings;

INFO:tensorflow:Oracle triggered exit
INFO:tensorflow:Assets written to: ./structured_data_classifier/best_model/assets
WARNING:tensorflow:Detecting that an object or model or tf.train.Checkpoint is being deleted with unrestored values. See the following logs for the specific values in question. To silence these warnings, use `status.expect_partial()`. See https://www.tensorflow.org/api_docs/python/tf/train/Checkpoint#restorefor details about the status object returned by the restore function.
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.iter
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.beta_1
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.beta_2
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.decay
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.learning_rate
2022-06-29 08:05:16.340694: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at lookup_table_op.cc:929 : FAILED_PRECONDITION: Table not initialized.
2022-06-29 08:05:16.340722: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at lookup_table_op.cc:929 : FAILED_PRECONDITION: Table not initialized.
2022-06-29 08:05:16.340759: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at lookup_table_op.cc:929 : FAILED_PRECONDITION: Table not initialized.
2022-06-29 08:05:16.340777: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at lookup_table_op.cc:929 : FAILED_PRECONDITION: Table not initialized.
2022-06-29 08:05:16.340797: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at lookup_table_op.cc:929 : FAILED_PRECONDITION: Table not initialized.
2022-06-29 08:05:16.340811: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at lookup_table_op.cc:929 : FAILED_PRECONDITION: Table not initialized.
2022-06-29 08:05:16.340824: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at lookup_table_op.cc:929 : FAILED_PRECONDITION: Table not initialized.

and then the following error;

Traceback (most recent call last):

  File ~/Programming/temp/autokeras_structured.py:44 in <module>
    predicted_y = clf.predict(x_test)

  File ~/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/autokeras/tasks/structured_data.py:165 in predict
    return super().predict(x=x, **kwargs)

  File ~/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/autokeras/auto_model.py:455 in predict
    y = model.predict(dataset, **kwargs)

  File ~/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/keras/utils/traceback_utils.py:67 in error_handler
    raise e.with_traceback(filtered_tb) from None

  File ~/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/tensorflow/python/eager/execute.py:54 in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,

FailedPreconditionError: Graph execution error:

Detected at node 'model/multi_category_encoding/string_lookup_10/None_Lookup/LookupTableFindV2' defined at (most recent call last):
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/runpy.py", line 197, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/runpy.py", line 87, in _run_code
      exec(code, run_globals)
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/spyder_kernels/console/__main__.py", line 23, in <module>
      start.main()
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/spyder_kernels/console/start.py", line 328, in main
      kernel.start()
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/ipykernel/kernelapp.py", line 677, in start
      self.io_loop.start()
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/tornado/platform/asyncio.py", line 199, in start
      self.asyncio_loop.run_forever()
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/asyncio/base_events.py", line 601, in run_forever
      self._run_once()
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/asyncio/base_events.py", line 1905, in _run_once
      handle._run()
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/asyncio/events.py", line 80, in _run
      self._context.run(self._callback, *self._args)
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 471, in dispatch_queue
      await self.process_one()
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 460, in process_one
      await dispatch(*args)
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 367, in dispatch_shell
      await result
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 662, in execute_request
      reply_content = await reply_content
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/ipykernel/ipkernel.py", line 360, in do_execute
      res = shell.run_cell(code, store_history=store_history, silent=silent)
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/ipykernel/zmqshell.py", line 532, in run_cell
      return super().run_cell(*args, **kwargs)
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2881, in run_cell
      result = self._run_cell(
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2936, in _run_cell
      return runner(coro)
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
      coro.send(None)
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3135, in run_cell_async
      has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3338, in run_ast_nodes
      if await self.run_code(code, result, async_=asy):
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3398, in run_code
      exec(code_obj, self.user_global_ns, self.user_ns)
    File "/var/folders/_h/8xl2b_ss5tq_0k8c51zx4vb00000gn/T/ipykernel_61037/3742600826.py", line 1, in <cell line: 1>
      runfile('/Users/PS/Programming/temp/autokeras_structured.py', wdir='/Users/PS/Programming/temp')
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/spyder_kernels/customize/spydercustomize.py", line 577, in runfile
      exec_code(file_code, filename, ns_globals, ns_locals,
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/spyder_kernels/customize/spydercustomize.py", line 465, in exec_code
      exec(compiled, ns_globals, ns_locals)
    File "/Users/PS/Programming/temp/autokeras_structured.py", line 44, in <module>
      predicted_y = clf.predict(x_test)
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/autokeras/tasks/structured_data.py", line 165, in predict
      return super().predict(x=x, **kwargs)
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/autokeras/auto_model.py", line 455, in predict
      y = model.predict(dataset, **kwargs)
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/keras/engine/training.py", line 2033, in predict
      tmp_batch_outputs = self.predict_function(iterator)
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/keras/engine/training.py", line 1845, in predict_function
      return step_function(self, iterator)
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/keras/engine/training.py", line 1834, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/keras/engine/training.py", line 1823, in run_step
      outputs = model.predict_step(data)
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/keras/engine/training.py", line 1791, in predict_step
      return self(x, training=False)
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/keras/engine/training.py", line 490, in __call__
      return super().__call__(*args, **kwargs)
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/keras/engine/functional.py", line 458, in call
      return self._run_internal_graph(
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/keras/engine/functional.py", line 596, in _run_internal_graph
      outputs = node.layer(*args, **kwargs)
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/autokeras/keras_layers.py", line 99, in call
      for input_node, encoding_layer in zip(split_inputs, self.encoding_layers):
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/autokeras/keras_layers.py", line 100, in call
      if encoding_layer is None:
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/autokeras/keras_layers.py", line 108, in call
      output_nodes.append(
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/keras/layers/preprocessing/index_lookup.py", line 628, in call
      lookups = self._lookup_dense(inputs)
    File "/Users/PS/opt/miniconda3/envs/envspyder/lib/python3.9/site-packages/keras/layers/preprocessing/index_lookup.py", line 657, in _lookup_dense
      lookups = self.lookup_table.lookup(inputs)
Node: 'model/multi_category_encoding/string_lookup_10/None_Lookup/LookupTableFindV2'
Table not initialized.
	 [[{{node model/multi_category_encoding/string_lookup_10/None_Lookup/LookupTableFindV2}}]] [Op:__inference_predict_function_10353]

When final_fit is performed there are no issues, so my guess is that there is problem with retrieving the model or pipeline, but unfortunately I can't seem to narrow it down further. Any ideas are greatly appreciated.

Bug Reproduction

Code for reproducing the bug:

import numpy as np
import pandas as pd
import tensorflow as tf

import autokeras as ak
import pandas as pd

from tensorflow.keras import callbacks as tf_callbacks

TRAIN_DATA_URL = "https://storage.googleapis.com/tf-datasets/titanic/train.csv"
TEST_DATA_URL = "https://storage.googleapis.com/tf-datasets/titanic/eval.csv"

train_file_path = tf.keras.utils.get_file("train.csv", TRAIN_DATA_URL)
test_file_path = tf.keras.utils.get_file("eval.csv", TEST_DATA_URL)

train_data = pd.read_csv(train_file_path)
test_data = pd.read_csv(test_file_path)

y_train = train_data.survived
y_test = test_data.survived

x_train = train_data.drop('survived', axis=1)
x_test = test_data.drop('survived', axis=1)

# Initialize the structured data classifier.
clf = ak.StructuredDataClassifier(
    overwrite=True, max_trials=2
)  # It tries 3 different models.
# Feed the structured data classifier with training data.

callbacks = [tf_callbacks.EarlyStopping(patience=10, min_delta=1e-4)]
   
clf.fit(x_train, y_train, epochs=1, validation_data=(x_test, y_test), callbacks=callbacks)
# Predict with the best model.
predicted_y = clf.predict(x_test)
# Evaluate the best model with testing data.
print(clf.evaluate(x_test, y_test))

This problem affects all classifiers and regressors.

Expected Behavior

That predict works without errors or warnings.

Setup Details

Include the details about the versions of:

• OS type and version: macOS Big Sur: version 11.6.7
• Python: Python 3.9.12
• autokeras: 1.0.19
• keras-tuner: 1.1.2
• scikit-learn: 1.1.1
• numpy: 1.23.0
• pandas: 1.4.2
• tensorflow: 2.9.1

Additional context

[psaks]

Possibly related to #1733

[psaks]

A workaround that prevents the downstream issues is to replace the else branch as follows;

            copied_fit_kwargs = copy.copy(fit_kwargs)

            # Remove early-stopping since no validation data.
            # Remove early-stopping since it is inserted.
            copied_fit_kwargs["callbacks"] = self._remove_early_stopping(callbacks)
            # Decide the number of epochs.
            copied_fit_kwargs["epochs"] = 0

            self.hypermodel.set_fit_args(0, epochs=copied_fit_kwargs["epochs"])
            pipeline, model, history = self.final_fit(**copied_fit_kwargs)

[KirkSuD]

Hi, I encountered same issue.

I made a workaround without modifying code of autokeras because I'm using Google Colab.

ak_model = StructuredDataClassifier(
    column_names=features,
    project_name=project_name,
    directory="model",
    seed=0,
)
history = ak_model.fit(train_X, train_y, validation_data=(test_X, test_y))
# print("eval train:", ak_model.evaluate(train_X, train_y))  # same error here
# print("eval test:", ak_model.evaluate(test_X, test_y))  # same error here

# fix it buy calling adapt()
model = keras.models.load_model(f"model/{project_name}/best_model",
    custom_objects=autokeras.CUSTOM_OBJECTS)
dataset, validation_data = ak_model._convert_to_dataset(
    x=train_X, y=train_y, validation_data=(test_X, test_y), batch_size=32
)
ak_model.tuner.adapt(model, dataset)

# now save adapted model, we can do predict, evaluate now  :)
model.save(f"model/{project_name}/best_model_adapted", save_format="tf")
print("eval train:", model.evaluate(train_X, train_y))
print("eval test:", model.evaluate(test_X, test_y))

I think this issue is due to adapt() not being called for some reason under some situations,
so MultiCategoryEncoding's table is not initialized. It hasn't seen any data, so it doesn't know how to encode data.

[arturdaraujo]

I had the same issue

[sideshot]

@KirkSuD Why did you set the batch size? batch_size=32

Hi, I encountered same issue.

I made a workaround without modifying code of autokeras because I'm using Google Colab.

ak_model = StructuredDataClassifier(
    column_names=features,
    project_name=project_name,
    directory="model",
    seed=0,
)
history = ak_model.fit(train_X, train_y, validation_data=(test_X, test_y))
# print("eval train:", ak_model.evaluate(train_X, train_y))  # same error here
# print("eval test:", ak_model.evaluate(test_X, test_y))  # same error here

# fix it buy calling adapt()
model = keras.models.load_model(f"model/{project_name}/best_model",
    custom_objects=autokeras.CUSTOM_OBJECTS)
dataset, validation_data = ak_model._convert_to_dataset(
    x=train_X, y=train_y, validation_data=(test_X, test_y), batch_size=32
)
ak_model.tuner.adapt(model, dataset)

# now save adapted model, we can do predict, evaluate now  :)
model.save(f"model/{project_name}/best_model_adapted", save_format="tf")
print("eval train:", model.evaluate(train_X, train_y))
print("eval test:", model.evaluate(test_X, test_y))

I think this issue is due to adapt() not being called for some reason under some situations, so MultiCategoryEncoding's table is not initialized. It hasn't seen any data, so it doesn't know how to encode data.

[KirkSuD]

Because I wanted to call adapt(), so I made a dataset, and batch_size seems to be a required argument. I haven't used AutoKeras for quite a while, things can change, though.

[sideshot]

Thanks. My project was running for 48 hours and then failed at this part. I noticed it was required too. I haven't gotten the results yet. Still working my way through it.

Are you using something else?

[deathsaber]

@sideshot - Thanks. I used your workaround and it works perfectly for me.