Permissions on /tmp getting changed from 1777 to 0777

[NickJH]

Describe the bug

I have started as a SysAdmin at a new company and am observing permissions on /tmp on a bunch of servers keep getting changed to 0777 from 1777. I have set up a monitor on on of them and I am seeing:

Wed May 24 21:30:01 EDT 2023 drwxrwxrwt 22 root root 4096 May 24 16:32 /tmp
Wed May 24 21:31:01 EDT 2023 drwxrwxrwt 22 root root 4096 May 24 16:32 /tmp
Wed May 24 21:32:01 EDT 2023 drwxrwxrwx 22 root root 4096 May 24 16:32 /tmp
Wed May 24 21:33:01 EDT 2023 drwxrwxrwx 22 root root 4096 May 24 16:32 /tmp

In syslog at that time I see:

May 24 21:30:01 sc1dvapt01 CRON[66187]: (root) CMD (/bin/echo "$(/bin/date) $(/bin/ls -ld1 /tmp)" >> /home/nhowitt/temp_perms)
May 24 21:31:01 sc1dvapt01 CRON[66192]: (root) CMD (/bin/echo "$(/bin/date) $(/bin/ls -ld1 /tmp)" >> /home/nhowitt/temp_perms)
May 24 21:31:31 sc1dvapt01 vmtoolsd[100315]: cat: /tmp/drv_target.lst: No such file or directory
May 24 21:32:01 sc1dvapt01 CRON[66221]: (root) CMD (/bin/echo "$(/bin/date) $(/bin/ls -ld1 /tmp)" >> /home/nhowitt/temp_perms)
May 24 21:33:01 sc1dvapt01 CRON[66226]: (root) CMD (/bin/echo "$(/bin/date) $(/bin/ls -ld1 /tmp)" >> /home/nhowitt/temp_perms)

Authlog gives:

May 24 21:31:01 sc1dvapt01 CRON[66191]: pam_unix(cron:session): session closed for user root
May 24 21:31:28 sc1dvapt01 VGAuth[100315]: vmtoolsd: Username and password successfully validated for 'root'.
May 24 21:31:31 sc1dvapt01 VGAuth[100315]: message repeated 9 times: [ vmtoolsd: Username and password successfully validated for 'root'.]
May 24 21:31:36 sc1dvapt01 VGAuth[100315]: vmtoolsd: Username and password successfully validated for 'root'.
May 24 21:31:37 sc1dvapt01 VGAuth[100315]: message repeated 8 times: [ vmtoolsd: Username and password successfully validated for 'root'.]
May 24 21:32:01 sc1dvapt01 CRON[66220]: pam_unix(cron:session): session opened for user root by (uid=0)

So I am suspecting open-vm-tools but I don't know how to investigate further.

Reproduction steps

1. See above

Expected behavior

I would not expect permissions on /tmp to be changed by anything.

Additional context

This is just on Ubuntu 18 LTS machines running open-vm-tools 2:11.0.5-4ubuntu0.18.04.2 amd64.

If this is not the right place to post, please can you point me to it? The mailing lists on sourceforge look dead.

[rprabhud]

Hi, Thanks for reporting the issue. I don't see this issue happening in my Ubuntu-18.04.2 VM with open-vm-tools 11.0.5. We would want to check the VMTools debug level logs from your machine to check what is actually happening. Can you please set "vmsvc.level = debug" in "/etc/vmware-tools/tools.conf" and reproduce the issue again? And provide logs from the file "/var/log/vmware-vmsvc-root.log".

[lemke1458]

The auth log suggests something is doing guestOperations to the VM (lots of authn checks for 'root'). guestOps allow apps to do file system and process operations in the VM, so it could be some app that's changing the perms.

The Tools debug log might show this (there's a ChangeFileAttruibtes API) though its also very possible that this app is uploading a script that does a chmod(), which we won't see in the logs.

[NickJH]

Yes, it seems to be happening at the time backups (Arcserve) run. I have opened a ticket with them as well, but I have zero experience of vmware-tools so didn't know where to begin troubleshooting. I have set up degugging with open-vm-tools as asked. I've done it on the same machine I reported against but I've also tried updating to the latest upstream release which is a minor point upgrade to open-vm-tools 2:11.0.5-4ubuntu0.18.04.3 amd64, IIRC. Lets see what happens overnight.

[NickJH]

Defeated by log rotation which are rotating the logs every 1MB so often under an hour. I'll try again with 50MB but I won't get to the logs until Tuesday as it is a holiday here on Monday.

[NickJH]

Permissions changed between 21:17 and 21:18 on 26/05. The Arcserve backup job started at 21:13:47 and ran through until 21:23:35. I am attaching the rotated vmware-vmsvc-root.log cut down to that 30min interval and the Arcserve log although I am doubtful of the latter helping. vmware-vmsvc-root.4.log Arcserve_activitylog.csv

I can't see anything directly related to /tmp in the vmware-vmsvc-root.4.log.

[lemke1458]

I don't see any guestOps in the vmsvc log, which is unexpected since your earlier auth log shows guestOps being authenticated.

Is there a timezone issue? The vmsvc logs are using UTC timestamps.

[NickJH]

Good catch. I've attached the correct log snippet this time, I hope. The server is in EDT (UTC-4) as are most of its logs. vmware-vmsvc-root.4.log

There is a syslog entry:

May 26 21:17:01 sc1dvapt01 CRON[93714]: (root) CMD (/bin/echo "$(/bin/date) $(/bin/ls -ld1 /tmp)" >> /home/nhowitt/temp_perms)
May 26 21:17:03 sc1dvapt01 vmtoolsd[79581]: cat: /tmp/drv_target.lst: No such file or directory
May 26 21:18:01 sc1dvapt01 CRON[93742]: (root) CMD (/bin/echo "$(/bin/date) $(/bin/ls -ld1 /tmp)" >> /home/nhowitt/temp_perms)

[rprabhud]

Good catch. I've attached the correct log snippet this time, I hope. The server is in EDT (UTC-4) as are most of its logs. vmware-vmsvc-root.4.log

There is a syslog entry:

May 26 21:17:01 sc1dvapt01 CRON[93714]: (root) CMD (/bin/echo "$(/bin/date) $(/bin/ls -ld1 /tmp)" >> /home/nhowitt/temp_perms)
May 26 21:17:03 sc1dvapt01 vmtoolsd[79581]: cat: /tmp/drv_target.lst: No such file or directory
May 26 21:18:01 sc1dvapt01 CRON[93742]: (root) CMD (/bin/echo "$(/bin/date) $(/bin/ls -ld1 /tmp)" >> /home/nhowitt/temp_perms)

Looks like you attached same log file.

[NickJH]

Bother. It didn't overwrite the old file on my desktop. I'll try again. vmware-vmsvc-root.4.log

[NickJH]

Not working. It is a github issue. It is not handling the same file name correctly. I'll rename it and re-upload it.

[NickJH]

Let's try this: vmware-vmsvc-root.4v2.log

Looking better this time.

[lemke1458]

Still a TZ issue. The vmsvc.log needs to be 4 hours ahead (approx 1:17 may 27), not 4 hours behind.

You should see some log noise of the form:

VixTools_ProcessVixCommand: command

[NickJH]

Sorry. I am normally hot on TZ's as I've worked with them for years. I've seen clowns more able than me today :( vmware-vmsvc-root.4v3.log

[lemke1458]

OK, this has the details:

First, it creates /tmp (making sure to also ask to create any parent dirs)

[2023-05-27T01:17:00.777Z] [ debug] [vix] VixToolsCreateDirectory: User: root dirPathName: /tmp createParent: 1

Then it sets up a file copy [2023-05-27T01:17:00.866Z] [ debug] [vix] VixToolsInitiateFileTransferToGuest: User: root path: /tmp/checkmachine attrs: 3

Then 6 packets of data for the file contents: [2023-05-27T01:17:01.097Z] [ debug] [vmsvc] HgfsServerManager_ProcessPacket: P rocessing Packet for Vix_1_Relayed_Command.

The script is made executable:

[2023-05-27T01:17:01.186Z] [ debug] [vix] VixTools_StartProgram: User: root args: progamPath: '/bin/chmod', arguments: '+x /tmp/checkmachine', workingDir: '' [2023-05-27T01:17:01.186Z] [ debug] [vmsvc] Executing async command: '"/bin/chmod" +x /tmp/checkmachine' in working dir '/root'

Then run [2023-05-27T01:17:03.313Z] [ debug] [vix] VixTools_StartProgram: User: root args: progamPath: '/bin/sh', arguments: '-c ""/tmp/checkmachine -volusage > /tmp/checkmachine.output.txt""', workingDir: ''

List files to see if the output is there [2023-05-27T01:17:08.477Z] [ debug] [vix] VixToolsListFiles: pattern length is 23, value is 'checkmachine.output.txt'

File copy of the output file started [2023-05-27T01:17:08.670Z] [ debug] [vix] VixToolsInitiateFileTransferFromGuest: User: root filePath: /tmp/checkmachine.output.txt

Packets copied out [2023-05-27T01:17:08.785Z] [ debug] [vmsvc] HgfsServerManager_ProcessPacket: Processing Packet for Vix_1_Relayed_Command.

No more guestOps after that.

So if /tmp/checkmachine is still there (and they don't clean up via guestOps) you may find the problem chmod in it

[NickJH]

Thanks a bunch for the analysis. The script is there and long. It is v. late here so I'll have a proper look tomorrow. I don't understand why they are doing a mkdir /tmp without checking for it already existing in your analysis. Later they do something like a mkdir 777 -p $workingdirectory and I think the $workingdirectory is in /tmp and this may stomp on the permissions.

[NickJH]

It is Arcserve's checkmachine script which is the problem and does the chmod. Thanks for the help.

[lemke1458]

Glad you could chase it down.

I'll note that Arcserve is not using guestOps in a secure way. A script with a static name in a world-readable directory is susceptible to having its contents be replaced/modified. So a normal user can get their own commands run as root.

VMware has a KB on best practices https://kb.vmware.com/s/article/70711