Permission search takes too long

[chris-rock]

Customers reported that the find permission command takes too long. We should find a solution to do this faster.

Running handlers:
[2017-03-16T13:17:19-04:00] INFO: Running report handlers
[2017-03-16T13:17:19-04:00] WARN: Format is json
[2017-03-16T13:17:19-04:00] INFO: Initialize InSpec
[2017-03-16T13:17:20-04:00] WARN: URL target https://github.com/dev-sec/linux-baseline transformed to https://github.com/dev-sec/linux-baseline/archive/master.tar.gz. Consider using the git fetcher
[2017-03-16T13:17:20-04:00] INFO: Running tests from: [{:name=>"linux-baseline", :supermarket=>"dev-sec/linux-baseline"}]
 
 
 
 
 
 
[2017-03-16T13:27:30-04:00] ERROR: Report handler Chef::Handler::AuditReport raised #<Mixlib::ShellOut::CommandTimeout: Command timed out after 600s:
Command exceeded allowed execution time, process terminated
---- Begin output of find / -perm -4000 -o -perm -2000 -type f ! -path '/proc/*' ! -path '/var/lib/lxd/containers/*' -print 2>/dev/null | grep -v '^find:' ----
STDOUT:
STDERR:
---- End output of find / -perm -4000 -o -perm -2000 -type f ! -path '/proc/*' ! -path '/var/lib/lxd/containers/*' -print 2>/dev/null | grep -v '^find:' ----
Ran find / -perm -4000 -o -perm -2000 -type f ! -path '/proc/*' ! -path '/var/lib/lxd/containers/*' -print 2>/dev/null | grep -v '^find:' returned >
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/mixlib-shellout-2.2.7/lib/mixlib/shellout/unix.rb:124:in `run_command'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/mixlib-shellout-2.2.7/lib/mixlib/shellout.rb:259:in `run_command'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/train-0.22.1/lib/train/transports/local.rb:32:in `run_command'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/inspec-1.15.0/lib/resources/command.rb:31:in `result'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/inspec-1.15.0/lib/resources/command.rb:35:in `stdout'
[2017-03-16T13:27:30-04:00] ERROR: linux-baseline-master/controls/os_spec.rb:193:in `block in load_with_context'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/inspec-1.15.0/lib/inspec/rule.rb:51:in `instance_eval'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/inspec-1.15.0/lib/inspec/rule.rb:51:in `initialize'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/inspec-1.15.0/lib/inspec/control_eval_context.rb:71:in `new'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/inspec-1.15.0/lib/inspec/control_eval_context.rb:71:in `block (2 levels) in create'
[2017-03-16T13:27:30-04:00] ERROR: linux-baseline-master/controls/os_spec.rb:187:in `load_with_context'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/inspec-1.15.0/lib/inspec/profile_context.rb:146:in `instance_eval'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/inspec-1.15.0/lib/inspec/profile_context.rb:146:in `load_with_context'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/inspec-1.15.0/lib/inspec/profile_context.rb:130:in `load_control_file'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/inspec-1.15.0/lib/inspec/profile.rb:144:in `block in collect_tests'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/inspec-1.15.0/lib/inspec/profile.rb:141:in `each'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/inspec-1.15.0/lib/inspec/profile.rb:141:in `collect_tests'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/inspec-1.15.0/lib/inspec/runner.rb:90:in `block in load'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/inspec-1.15.0/lib/inspec/runner.rb:79:in `each'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/inspec-1.15.0/lib/inspec/runner.rb:79:in `load'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/inspec-1.15.0/lib/inspec/runner.rb:100:in `run'
[2017-03-16T13:27:30-04:00] ERROR: /var/chef/cache/cookbooks/audit/files/default/handler/audit_report.rb:116:in `call'
[2017-03-16T13:27:30-04:00] ERROR: /var/chef/cache/cookbooks/audit/files/default/handler/audit_report.rb:47:in `block in report'
[2017-03-16T13:27:30-04:00] ERROR: /var/chef/cache/cookbooks/audit/files/default/handler/audit_report.rb:33:in `each'
[2017-03-16T13:27:30-04:00] ERROR: /var/chef/cache/cookbooks/audit/files/default/handler/audit_report.rb:33:in `report'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36/lib/chef/handler.rb:259:in `run_report_unsafe'
[2017-03-16T13:27:30-04:00] ERROR: /var/chef/cache/cookbooks/audit/files/default/handler/audit_report.rb:59:in `run_report_safely'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36/lib/chef/handler.rb:125:in `block in run_report_handlers'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36/lib/chef/handler.rb:123:in `each'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36/lib/chef/handler.rb:123:in `run_report_handlers'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36/lib/chef/handler.rb:135:in `block in <class:Handler>'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36/lib/chef/client.rb:441:in `block in run_completed_successfully'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36/lib/chef/client.rb:440:in `each'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36/lib/chef/client.rb:440:in `run_completed_successfully'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36/lib/chef/client.rb:299:in `run'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36/lib/chef/application.rb:295:in `block in fork_chef_client'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36/lib/chef/application.rb:283:in `fork'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36/lib/chef/application.rb:283:in `fork_chef_client'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36/lib/chef/application.rb:248:in `block in run_chef_client'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36/lib/chef/local_mode.rb:44:in `with_server_connectivity'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36/lib/chef/application.rb:236:in `run_chef_client'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36/lib/chef/application/client.rb:464:in `sleep_then_run_chef_client'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36/lib/chef/application/client.rb:451:in `block in interval_run_chef_client'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36/lib/chef/application/client.rb:450:in `loop'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36/lib/chef/application/client.rb:450:in `interval_run_chef_client'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36/lib/chef/application/client.rb:434:in `run_application'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36/lib/chef/application.rb:59:in `run'
[2017-03-16T13:27:30-04:00] ERROR: /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36/bin/chef-client:26:in `<top (required)>'
[2017-03-16T13:27:30-04:00] ERROR: /bin/chef-client:57:in `load'
[2017-03-16T13:27:30-04:00] ERROR: /bin/chef-client:57:in `<main>'
  - Chef::Handler::AuditReport
Running handlers complete
[2017-03-16T13:27:30-04:00] INFO: Report handlers complete
Chef Client finished, 1/11 resources updated in 10 minutes 26 seconds

[artem-sidorenko]

If it possible to get a bit more information here? My assumption is that this find goes through a filesystem with a huge amount of files (some filesystem with tons of data)

I faced once a similar problem with my first implementations of puppet-os-hardening at P&I years ago and we resolved it this way: exclude filesystems with nosuid, nodev, noexec mount options from a such find and mount all data filesystems with this mount options.

[mcgege]

For the search in os-01 and os-09 I would recommend to limit the the search depth with the option -maxdepth. IMHO 3 levels should be sufficient ... for os-06 I don't see a way to optimize the search.

BTW: Why do you search for .rhosts in os-01 and also in os-09?

[atomic111]

@mcgege thanks for your feedback. i agree with you. it should be something like this:

describe file('/etc/hosts.equiv') do
  it { should_not exist }
end

and to limit the max depth to 3 should be sufficient. Can you create a PR for this?

[mcgege]

@atomic111 Of course! See #77

[mike-stewart]

I'm getting this error when running inspec compliance upload linux-baseline/. Why is that being run during the upload? Any workaround?

I, [2017-07-13T15:14:25.786924 #90832]  INFO -- : Checking profile in ../linux-baseline/
I, [2017-07-13T15:14:25.811868 #90832]  INFO -- : Metadata OK.
/opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/mixlib-shellout-2.2.7/lib/mixlib/shellout/unix.rb:124:in `run_command': Command timed out after 600s: (Mixlib::ShellOut::CommandTimeout)
Command exceeded allowed execution time, process terminated
---- Begin output of find / -perm -4000 -o -perm -2000 -type f ! -path '/proc/*' ! -path '/var/lib/lxd/containers/*' -print 2>/dev/null | grep -v '^find:' ----
STDOUT: 
STDERR: 
---- End output of find / -perm -4000 -o -perm -2000 -type f ! -path '/proc/*' ! -path '/var/lib/lxd/containers/*' -print 2>/dev/null | grep -v '^find:' ----
Ran find / -perm -4000 -o -perm -2000 -type f ! -path '/proc/*' ! -path '/var/lib/lxd/containers/*' -print 2>/dev/null | grep -v '^find:' returned 
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/mixlib-shellout-2.2.7/lib/mixlib/shellout.rb:259:in `run_command'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/train-0.25.0/lib/train/transports/local.rb:32:in `run_command'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/inspec-1.30.0/lib/resources/command.rb:33:in `result'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/inspec-1.30.0/lib/resources/command.rb:37:in `stdout'
	from ../linux-baseline/controls/os_spec.rb:189:in `block in load_with_context'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/inspec-1.30.0/lib/inspec/rule.rb:49:in `instance_eval'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/inspec-1.30.0/lib/inspec/rule.rb:49:in `initialize'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/inspec-1.30.0/lib/inspec/control_eval_context.rb:71:in `new'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/inspec-1.30.0/lib/inspec/control_eval_context.rb:71:in `block (2 levels) in create'
	from ../linux-baseline/controls/os_spec.rb:183:in `load_with_context'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/inspec-1.30.0/lib/inspec/profile_context.rb:146:in `instance_eval'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/inspec-1.30.0/lib/inspec/profile_context.rb:146:in `load_with_context'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/inspec-1.30.0/lib/inspec/profile_context.rb:130:in `load_control_file'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/inspec-1.30.0/lib/inspec/profile.rb:151:in `block in collect_tests'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/inspec-1.30.0/lib/inspec/profile.rb:148:in `each'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/inspec-1.30.0/lib/inspec/profile.rb:148:in `collect_tests'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/inspec-1.30.0/lib/inspec/profile.rb:454:in `load_checks_params'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/inspec-1.30.0/lib/inspec/profile.rb:447:in `load_params'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/inspec-1.30.0/lib/inspec/profile.rb:141:in `params'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/inspec-1.30.0/lib/inspec/profile.rb:307:in `controls_count'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/inspec-1.30.0/lib/inspec/profile.rb:278:in `check'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/inspec-1.30.0/lib/bundles/inspec-compliance/cli.rb:186:in `upload'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/thor-0.19.1/lib/thor/invocation.rb:115:in `invoke'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/thor-0.19.1/lib/thor.rb:235:in `block in subcommand'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
	from /opt/chefdk/embedded/lib/ruby/gems/2.4.0/gems/inspec-1.30.0/bin/inspec:12:in `<top (required)>'
	from /usr/local/bin/inspec:264:in `load'
	from /usr/local/bin/inspec:264:in `<main>'

[atomic111]

@mike-stewart i tested it and it is working. i used the inspec version 1.31.1 and the compliance server 1.10.2

i included you my commands, which i used to upload the linux-baseline.

atomic111:..ooks/linux-baseline ±> inspec compliance login https://192.168.100.40 --insecure --user=creator --refresh-token=<refresh-token>
WARN: Unresolved specs during Gem::Specification.reset:
      thor (~> 0.19)
      rspec (~> 3)
      addressable (~> 2.4)
      winrm (~> 2.0)
      docker-api (~> 1.26)
WARN: Clearing out unresolved specs.
Please report a bug if this causes problems.

API access token verified
atomic111:..ooks/linux-baseline ±> inspec compliance upload ./                                                                                                                                      11d [3303c00]
WARN: Unresolved specs during Gem::Specification.reset:
      thor (~> 0.19)
      rspec (~> 3)
      addressable (~> 2.4)
      winrm (~> 2.0)
      docker-api (~> 1.26)
WARN: Clearing out unresolved specs.
Please report a bug if this causes problems.
Profile is already vendored. Use --overwrite.
I, [2017-07-14T09:09:10.939861 #24338]  INFO -- : Checking profile in ./
I, [2017-07-14T09:09:10.941846 #24338]  INFO -- : Metadata OK.
I, [2017-07-14T09:09:14.976291 #24338]  INFO -- : Found 52 controls.
I, [2017-07-14T09:09:14.976417 #24338]  INFO -- : Control definitions OK.
Profile is valid
Generate temporary profile archive at /tmp/linux-baseline20170714-24338-u9fe.tar.gz
I, [2017-07-14T09:09:15.126673 #24338]  INFO -- : Generate archive /tmp/linux-baseline20170714-24338-u9fe.tar.gz.
I, [2017-07-14T09:09:15.135052 #24338]  INFO -- : Finished archive generation.
Start upload to creator/linux-baseline
Uploading to Chef Compliance
Successfully uploaded profile
atomic111:..ooks/linux-baseline ±> inspec version                                                                                                                                                   11d [3303c00]
WARN: Unresolved specs during Gem::Specification.reset:
      thor (~> 0.19)
      rspec (~> 3)
      addressable (~> 2.4)
      winrm (~> 2.0)
      docker-api (~> 1.26)
WARN: Clearing out unresolved specs.
Please report a bug if this causes problems.
1.31.1

[mike-stewart]

@atomic111 Still doesn't seem to be working for me. Is it possible that it's running the find command on my local machine as part of the upload, and it's failing for me because I have a lot of files on my machine?

[artem-sidorenko]

@mike-stewart it looks like your upload should take some time, can you check in another console with ps axw | grep find or similar if this find is executed really on your system?

[mcgege]

How about this: use locate instead of find if installed --> if you have timeout problems install the (m)locate package on your system This might also solve #78

[mattlqx]

My solution here was to create a wrapper profile with a find that looks for network fs types and excludes those paths from the find. My hosts from 10 minutes to execute the stock profile down to 30 seconds to execute the wrapper profile.

https://gist.github.com/mattlqx/24c6730d7586e78a23a31353066cb31c

This is the best/simplest way I found to override a part of a resource from another profile, feedback welcome if there are better ways. The dynamic classes that InSpec resources are made trying this... interesting.

[mattlqx]

I riffed a bit on my gist from the prior comment and submitted a pull to just get into here. With it, by default, network filesystems (arbitrarily defined) are now ignored in the find of suid_check. Attributes are provided to override the exclude behavior.

[bbigras]

Any progress on this? Or anyway to increase the 600s timeout?

[chris-rock]

Any PR is welcome to improve the situation.

[mattlqx]

I've had a pull open for over a year. 🤷🏻‍♂️

I can rebase it. I'm not entirely sure what comments were meant to be actioned on though.