Test cases failed for training_test, culprit is TestFunctionTracing class

[tarun-bisht]

trafficstars

Test cases failed when running bazel test keras/engine:training_test to test training module.

System information.

• OS Platform and Distribution (Linux Manjaro 21.1):
• Bazel version (4.1.0)

Development environment is setup using CONTRIBUTION.md

Describe the problem.

Test cases failed for training, by tracing logs I found problem resides in class TestFunctionTracing inside keras/engine/training_test.py. These functions are causing error as per logs:

@keras_parameterized.run_all_keras_modes(
      always_skip_v1=True, always_skip_eager=True)
  def test_no_tracing_between_epoch(self):
    model, x, y = self._seq_model_and_data()

    logging.set_verbosity(1)
    with self.assertLogs(level=1) as logs:
      model.fit(x, y, epochs=10, batch_size=5, validation_data=(x, y))

    new_func_graph = 'INFO:absl:Creating new FuncGraph for Python function'
    self.assertEqual(sum(new_func_graph in log for log in logs.output), 9)

@keras_parameterized.run_all_keras_modes(
    always_skip_v1=True, always_skip_eager=True)
def test_evaluate_no_cached_data(self):
  model, x, y = self._seq_model_and_data()

  new_func_graph = 'INFO:absl:Creating new FuncGraph for Python function'
  logging.set_verbosity(1)
  with self.assertLogs(level=1) as eval_logs:
    for _ in range(6):
      model.evaluate(x, y, batch_size=5)
  self.assertEqual(sum(new_func_graph in log for log in eval_logs.output), 20)

Describe the current behavior. 1 test case failed

Describe the expected behavior. All test cases should pass

Contributing.

• Do you want to contribute a PR? (yes/no):
• If yes, please read this page for instructions
• Briefly describe your candidate solution(if contributing):

Source code / logs. Error from Logs

self.assertEqual(sum(new_func_graph in log for log in logs.output), 9)
AssertionError: 0 != 9

self.assertEqual(sum(new_func_graph in log for log in eval_logs.output), 20)
AssertionError: 0 != 20

Error in terminal.

WARNING: The following configs were expanded more than once: [v2]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=136
INFO: Reading rc options for 'test' from /home/starck/Documents/open-source/keras/.bazelrc:
  Inherited 'build' options: --apple_platform_type=macos --define open_source_build=true --define=use_fast_cpp_protos=false --define=tensorflow_enable_mlir_generated_gpu_kernels=0 --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --config=short_logs --config=v2
INFO: Reading rc options for 'test' from /home/starck/Documents/open-source/keras/.bazelrc:
  'test' options: --define open_source_build=true --define=use_fast_cpp_protos=false --config=v2
INFO: Found applicable config definition build:short_logs in file /home/starck/Documents/open-source/keras/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file /home/starck/Documents/open-source/keras/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:v2 in file /home/starck/Documents/open-source/keras/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Analyzed target //keras/engine:training_test (1 packages loaded, 2 targets configured).
INFO: Found 1 test target...
FAIL: //keras/engine:training_test (shard 15 of 20) (see /home/starck/.cache/bazel/_bazel_starck/410419821f283229788d0a5dcee741ab/execroot/org_keras/bazel-out/k8-opt/testlogs/keras/engine/training_test/shard_15_of_20/test.log)
FAIL: //keras/engine:training_test (shard 16 of 20) (see /home/starck/.cache/bazel/_bazel_starck/410419821f283229788d0a5dcee741ab/execroot/org_keras/bazel-out/k8-opt/testlogs/keras/engine/training_test/shard_16_of_20/test.log)

FAILED: //keras/engine:training_test (Summary)
      /home/starck/.cache/bazel/_bazel_starck/410419821f283229788d0a5dcee741ab/execroot/org_keras/bazel-out/k8-opt/testlogs/keras/engine/training_test/shard_15_of_20/test.log
      /home/starck/.cache/bazel/_bazel_starck/410419821f283229788d0a5dcee741ab/execroot/org_keras/bazel-out/k8-opt/testlogs/keras/engine/training_test/shard_16_of_20/test.log
Target //keras/engine:training_test up-to-date:
  bazel-bin/keras/engine/training_test
INFO: Elapsed time: 338.541s, Critical Path: 114.16s
INFO: 218 processes: 3 internal, 215 local.
INFO: Build completed, 1 test FAILED, 218 total actions
//keras/engine:training_test                                             FAILED in 2 out of 20 in 88.3s
  Stats over 20 runs: max = 88.3s, min = 10.3s, avg = 23.5s, dev = 17.5s
  /home/starck/.cache/bazel/_bazel_starck/410419821f283229788d0a5dcee741ab/execroot/org_keras/bazel-out/k8-opt/testlogs/keras/engine/training_test/shard_15_of_20/test.log
  /home/starck/.cache/bazel/_bazel_starck/410419821f283229788d0a5dcee741ab/execroot/org_keras/bazel-out/k8-opt/testlogs/keras/engine/training_test/shard_16_of_20/test.log

Executed 1 out of 1 test: 1 fails locally.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these arINFO: Build completed, 1 test FAILED, 218 total actions

[rchao]

Thanks for reporting. We'll take a further look into this.

[tilakrayal]

@tarun-bisht, The related PR https://github.com/keras-team/keras/pull/15237 which was raised has been merged. Required changes were completed in the below file. https://github.com/keras-team/keras/blob/master/keras/engine/data_adapter.py#L1870 https://github.com/keras-team/keras/blob/master/keras/engine/data_adapter_test.py#L1558

    if isinstance(data, list):
        data = tuple(data)
    if not isinstance(data, tuple):
        return (data, None, None)

Could you please check and confirm the issue was resolved. Thank you!

[tarun-bisht]

@tilakrayal I checked and confirmed that the issue is resolved. I am attaching the screenshot of the run also you can confirm. Thanks.

[tilakrayal]

@tarun-bisht, Glad the issue was resolved. Could you please move this issue to closed status. Thank you!

[tarun-bisht]

Ok, Thanks. I am closing the issue.