Limit datasource spec execution duration

[chenlizhong]

From below log, we can see one datasource spec used more than one hour to complete. We need to have one hard limit for datasource execution time.

2022-10-10 08:59:42,993 DEBUG insights.core.dr insights.specs.Specs.systemctl_pulp_resmg missing requirements All: [] Any: [] 2022-10-10 08:59:42,993 INFO insights.core.dr Trying insights.specs.datasources.system_user_dirs.LocalSpecs.rpm_args 2022-10-10 08:59:42,994 INFO insights.core.dr Trying insights.specs.default.DefaultSpecs.system_user_dirs 2022-10-10 09:10:05,055 DEBUG insights.util.subproc Executing: [['timeout', '-s', '15', '120', 'rpm', '-qa', '--qf=[%{=NAME}; %{FILEMODES:perms}; %{FILEUSERNAME}; %{FILEGROUPNAME}\n]']] 2022-10-10 10:04:39,922 INFO insights.core.dr Trying insights.specs.Specs.system_user_dirs 2022-10-10 10:04:39,926 INFO insights.core.dr Trying insights.specs.Specs.systemctl_pulp_celerybeat

[bfahr]

@chenlizhong there currently is a limit when commands are run and normally a SIGKILL is used to stop the command when time expires. For RPM commands a SIGTERM signal is required to avoid causing issues with RPM state. So the command will only terminate when it processes the signal. In this case I think more research needs to be done to determine why the command is taking so long. Is it caused by problems when executing the command or is it due to the amount of data it processes.

[bfahr]

Adding a link to the specific datasource and local spec.

[bfahr]

While we have limits on specs using simple_command, I think your suggestion to control run-away custom datasouces is a good suggestion and should be investigated.

[jobselko]

This issue is currently being handled in #3573.

[xiangce]

Resolved by #3573