[Perf_Analyzer]: took very long to exit, killing perf_analyzer.

[sbmalik]

I tried to run model_navigator on my onnx model and it stops by giving following error..

2022-04-22 18:56:36.187 INFO[perf_analyzer.py:214] perf_analyzer took very long to exit, killing perf_analyzer... 2022-04-22 18:56:41.700 INFO[server_local.py:121] Stopped Triton Server.

Traceback (most recent call last): File "/usr/local/bin/model-analyzer", line 8, in <module> sys.exit(main()) File "/usr/local/lib/python3.8/dist-packages/model_analyzer/entrypoint.py", line 402, in main analyzer.profile(client=client, gpus=gpus) File "/usr/local/lib/python3.8/dist-packages/model_analyzer/analyzer.py", line 125, in profile self._model_manager.run_models(models=[model]) File "/usr/local/lib/python3.8/dist-packages/model_analyzer/model_manager.py", line 79, in run_models while not rcg.is_done() and not self._state_manager.exiting(): File "/usr/local/lib/python3.8/dist-packages/model_analyzer/config/generate/run_config_generator.py", line 60, in is_done return (self._pacg.is_done() and File "/usr/local/lib/python3.8/dist-packages/model_analyzer/config/generate/perf_analyzer_config_generator.py", line 84, in is_done return self._done_walking() or self._last_results_erroneous() File "/usr/local/lib/python3.8/dist-packages/model_analyzer/config/generate/perf_analyzer_config_generator.py", line 176, in _done_walking and self._done_walking_concurrencies() File "/usr/local/lib/python3.8/dist-packages/model_analyzer/config/generate/perf_analyzer_config_generator.py", line 183, in _done_walking_concurrencies 1) or not self._throughput_gain_valid() File "/usr/local/lib/python3.8/dist-packages/model_analyzer/config/generate/perf_analyzer_config_generator.py", line 194, in _throughput_gain_valid valid_gains = [self._calculate_throughput_gain(x) > THROUGHPUT_MINIMUM_GAIN \ File "/usr/local/lib/python3.8/dist-packages/model_analyzer/config/generate/perf_analyzer_config_generator.py", line 194, in <listcomp> valid_gains = [self._calculate_throughput_gain(x) > THROUGHPUT_MINIMUM_GAIN \ File "/usr/local/lib/python3.8/dist-packages/model_analyzer/config/generate/perf_analyzer_config_generator.py", line 212, in _calculate_throughput_gain throughput_after = self._get_throughput(self._all_results[after_index]) File "/usr/local/lib/python3.8/dist-packages/model_analyzer/config/generate/perf_analyzer_config_generator.py", line 217, in _get_throughput return measurement.get_metric_value('perf_throughput') AttributeError: 'NoneType' object has no attribute 'get_metric_value'

Traceback (most recent call last): File "/opt/model-navigator/model_navigator/cli/profile.py", line 130, in profile_cmd checkpoint_path = profiler.run() File "/opt/model-navigator/model_navigator/model_analyzer/profiler.py", line 100, in run analyzer.run(mode=ModelAnalyzerMode.PROFILE, verbose=self._verbose) File "/opt/model-navigator/model_navigator/model_analyzer/model_analyzer.py", line 73, in run raise ModelNavigatorException( model_navigator.exceptions.ModelNavigatorException: Running model-analyzer with ['model-analyzer', 'profile', '-f', '/home/darvis-ml3/darvis_ml/xperiments/model_navigator/navigator_workspace/analyzer/config-profile.yaml'] failed with exit status 1 : None

[sbmalik]

@jkosek kindly look into this..

[jkosek]

@sbmalik this error indicates that Triton Model Analyzer hadn't profiled one of the configurations and was unable to collect results. We are aware of this error and this should be fixed with upcoming release of Triton Model Analyzer. I will update the information once new version is available.

[sbmalik]

Thank you for the details.

@sbmalik this error indicates that Triton Model Analyzer hadn't profiled one of the configurations and was unable to collect results. We are aware of this error and this should be fixed with upcoming release of Triton Model Analyzer. I will update the information once new version is available.

[jkosek]

@sbmalik Please, try the latest version to see if the issues is still appearing.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 7 days with no activity.