Evaluator in airflow example raises: AttributeError: 'Tensor' object has no attribute 'numpy'

[jason-brian-anderson]

When going through the airflow tutorial in preparation for standing up TFX as an MLOPS solution for Garmin, I ran into the following failure on the model Evaluator:

Traceback (most recent call last): File "apache_beam/runners/common.py", line 780, in apache_beam.runners.common.DoFnRunner.process File "apache_beam/runners/common.py", line 441, in apache_beam.runners.common.SimpleInvoker.invoke_process File "/home/tfx/tfx-env/lib/python3.6/site-packages/tensorflow_model_analysis/model_util.py", line 382, in process result = self._batch_reducible_process(elements) File "/home/tfx/tfx-env/lib/python3.6/site-packages/tensorflow_model_analysis/extractors/predict_extractor_v2.py", line 150, in _batch_reducible_process output = {k: v[i].numpy() for k, v in outputs.items()} File "/home/tfx/tfx-env/lib/python3.6/site-packages/tensorflow_model_analysis/extractors/predict_extractor_v2.py", line 150, in <dictcomp> output = {k: v[i].numpy() for k, v in outputs.items()} AttributeError: 'Tensor' object has no attribute 'numpy'

I was able to solve this by adding the following compatibility mode:

tensorflow_model_analysis/extractors/predict_extractor_v2.py with

tf.compat.v1.enable_eager_execution()

as referenced here.

Is this the right solution to this problem?

[jason-brian-anderson]

If it is the right solution, would it be appropriate to create a pull request for the model-analysis project? or is there a better way?

[jiyongjung0]

Thank you for the report and a suggestion. It looks like you are running this in TF 1.X.

If that is the case the binary needs to add following code in main (or equivalent):

tf.compat.v1.enable_v2_behaviour()

And this change is needed only when you are using TF 1.X.

[AnbangZhao]

@jiyongjung0 I've seen a similar error caused by the same issue: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/dataflow_worker/batchworker.py", line 650, in do_work work_executor.execute() File "/usr/local/lib/python3.7/site-packages/dataflow_worker/executor.py", line 176, in execute op.start() File "dataflow_worker/shuffle_operations.py", line 50, in dataflow_worker.shuffle_operations.GroupedShuffleReadOperation.start File "dataflow_worker/shuffle_operations.py", line 51, in dataflow_worker.shuffle_operations.GroupedShuffleReadOperation.start File "dataflow_worker/shuffle_operations.py", line 66, in dataflow_worker.shuffle_operations.GroupedShuffleReadOperation.start File "dataflow_worker/shuffle_operations.py", line 67, in dataflow_worker.shuffle_operations.GroupedShuffleReadOperation.start File "dataflow_worker/shuffle_operations.py", line 71, in dataflow_worker.shuffle_operations.GroupedShuffleReadOperation.start File "apache_beam/runners/worker/operations.py", line 256, in apache_beam.runners.worker.operations.Operation.output File "apache_beam/runners/worker/operations.py", line 143, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive File "dataflow_worker/shuffle_operations.py", line 234, in dataflow_worker.shuffle_operations.BatchGroupAlsoByWindowsOperation.process File "dataflow_worker/shuffle_operations.py", line 241, in dataflow_worker.shuffle_operations.BatchGroupAlsoByWindowsOperation.process File "apache_beam/runners/worker/operations.py", line 256, in apache_beam.runners.worker.operations.Operation.output File "apache_beam/runners/worker/operations.py", line 143, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive File "apache_beam/runners/worker/operations.py", line 753, in apache_beam.runners.worker.operations.CombineOperation.process File "apache_beam/runners/worker/operations.py", line 758, in apache_beam.runners.worker.operations.CombineOperation.process File "/usr/local/lib/python3.7/site-packages/apache_beam/transforms/combiners.py", line 866, in extract_only return self.combine_fn.extract_output(accumulator) File "/Users/anbang.zhao/repo/masterchef-training/venv/lib/python3.7/site-packages/tensorflow_model_analysis/evaluators/metrics_and_plots_evaluator_v2.py", line 350, in extract_output output = c.extract_output(a) File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/metrics/tf_metric_wrapper.py", line 591, in extract_output result[key] = metric.result().numpy() AttributeError: 'Tensor' object has no attribute 'numpy' When executing locally, I can get around this by doing what you suggested in the main function. However, in the dataflow execution mode, I can't find a way to inject code in the setup process to enable eager execution for keras metrics. Any ideas on this?

[jiyongjung0]

I'm sorry for the late follow up. Have you tried something like DoFn.setup()?

https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.setup

[gowthamkpr]

@AnbangZhao Can you please respond to the above comment. Thanks!!

[singhniraj08]

Closing this due to inactivity. Please take a look into the answers provided above, feel free to reopen and post your comments(if you still have queries on this). Thank you!

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No