Unix collector unable to read /proc filesystem

[cgrinds]

This happens on RHEL and Deb systems due to changes in https://github.com/NetApp/harvest/pull/141/ that set the SUID bit for executables in harvest/bin so they always run as the harvest user.

The harvest user does not have permission to read the /proc/ filesystem which means the Unix collector fails to start.

One workaround is to run as root directly or via sudo

[cgrinds]

Log msg

Jun 25 19:10:22 scspo2403645002 harvest[19127]: 7:10PM ERR goharvest2/cmd/collectors/unix/main.go:167 > load metrics error="read file => io: open /proc/19132/io: permission denied" Poller=unix1 collector=Unix:poller stack=[{"func":"New","line":"35","source":"errors.go"},{"func":"(*Process).loadIo","line":"267","so>

[ruanruijuan]

The customer who took the 21.05.3 pre-release build hit this issue too.

[vgratian]

I suspect the problem is that in deb/postinst:

chmod -R u+s /opt/harvest/bin
chmod -R g+s /opt/harvest/bin

changes the set-user-ID to root (current executing user), not to harvest which is what we wanted.

sorry, I am probably wrong, it should set to the user owning the files.

[rahulguptajss]

Tested in ubuntu, works fine when harvest is started through systemctl. It fails and reports same error as above when started through bin/harvest which we don't recommend on rpm/deb

setuid looks fine.

[vgratian]

could you check if the user-ID of the poller is harvest as well when you start them with systemctl?

[rahulguptajss]

yes it is harvest in both cases.

[vgratian]

I reproduced this issue on a Debian 8 with harvest version 21.08.3011-v21.05.0 (commit 19f8f25). I log in as root and when I start a poller, it has the following unix credentials:

• Real User-ID: root
• Real Group-ID: root
• Effective User-ID: harvest
• Effective Group-ID: harvest

We would expect that with these credentials pollers should have access to all files in /proc/self/. This is true for most files, except for /proc/self/io which has special access-control mode. From the proc(5) manual:

Permission to access this file is governed by a ptrace access mode PTRACE_MODE_READ_FSCREDS check; see ptrace(2).

And looking at the ptrace(2) manual, I think this is our problem:

   3. Deny access if neither of the following is true:

    • The real, effective, and saved-set user IDs of the target
      match the caller's user ID, and the real, effective, and
      saved-set group IDs of the target match the caller's group
      ID.

    • The caller has the CAP_SYS_PTRACE capability in the user
      namespace of the target.

This means (as I understand it) that ptrace deems it unsafe for an unprivileged process that has privileged real-user-ID to access certain files. (At any time, the process can change its effective user-ID to root).

We can fix this by changing the real user and group IDs either in bin/harvest or bin/daemonize. I think this is more important than to simply fix the current issue: if we want to make harvest pollers completely safe, we have to make sure they never have (latent) root-privileges.

The reason why we don't get the same error when running the poller with systemctl, is that it runs the pollers as user harvest.

A safe workaround is do the same when you start pollers directly:

sudo -u harvest ./bin/harvest start unix

[cgrinds]

Fixed by #553. Closing