Confusion over updating rules and trusted files

[blairconrad]

I'm loath to ask several questions that may just stem from my complete lack of understanding of the product, but I've not been able to find answers in the online documentation.

I've been working on the installer my company's product, an Oracle Linux 8.3 system with fapolicyd enabled. The installer will sometimes add a new fapolicyd.rules entry, and will sometimes add trust entries via fapolicyd-cli. I'm seeing some behaviour that leaves me confused and was hoping I could get some clarity.

Oh, for what it's worth, I've been working with fapolicyd-1.0-3.0.1.el8_3.4.x86_64, supplied to me on an OVA by another department. I could experiment with a newer one if required.

1. After editing fapolicyd.rules, I've been restarting fapolicyd so it picks up the changes. Is this proper? Should I not bother, or instead do something else? I'd found that without this, the new rule was not applied (which was not surprising).

2. After restarting fapolicyd above, would I expect to immediately have access to

   1. files that are allowed by the new rules?
   2. files that are allowed by pre-existing rules?

   I've been getting "Operation not permitted" when trying to run some executables (even sudo and cut, which were allowed out of the box). Introducing a delay (sleeping for a minute) after restarting fapolicyd seemed to help, but not in all cases.

3. After installing LibreOffice RPMs using rpm (because there are neither package nor file digests, and I couldn't trick dnf into installing them) I've been running both fapolicyd-cli --file add and fapolicyd-cli --update to get fapolicyd to trust the new files.

   1. is fapolicyd-cli --file add superfluous? Should the files being added to the rpmdb be good enough?
   2. is fapolicyd-cli --update required? It does seem that fapolicyd will not honour the new files without it

4. Similarly to my question about restarting fapolicyd above, after running fapolicyd-cli --update, should the new (and old) trusted files and rules be in force immediately? Or does the fapolicyd daemon build up the rules and the trusted files all while intercepting requests? I ask because I've been denied access to certain executables (again, not always ones that would be newly-trusted) shortly after running fapolicyd-cli.

5. Potentially a useless question, depending on how 2 and 4 come out, but if fapolicyd takes time to update its in-memory of rules and trusted files, is there a way to tell when it's ready to go?

Again, sorry for the wall of questions. I appreciate anything you can do to unconfuse me.

[stevegrubb]

Hello, this is really too much to answer in one reply. One of the factors may be that Oracle doesn't have all the pieces in place or they have modified permissions or something. I don't know what they do. That said, which set of rules are you using? I'd recommend the known-libs rules. On RHEL 8, this would be /usr/share/fapolicyd/fapolicyd.rules.known-libs. You'd copy that to /etc/fapolicyd/fapolicyd.rules. I wouldn't modify that as it should pretty much stay out of the way as long as files are properly identified.

Fapolicyd has a debug mode that may be helpful to understanding what's happening. It uses the format provided in syslog_format in the config file. I have mine set to rule,dec,perm,uid,pid,exe,:,path,ftype,trust. Getting trust output is important to figuring out what's going on. Run /usr/sbin/fapolicyd --debug as root on the command line. You can exit with Ctl-c.

Next issue, does Oracle ship fapolicyd-dnf-plugin? This needs to be installed in order for fapolicyd to get notifications of rpm updates. Note that this is an area of active development and we are switching to an rpm plugin based solution because dnf doesn't let us see the transaction early enough. This, I think, goes out in the 8.4 update.

Also, we identified and fixed several problems in the file package. An update just went out this week. So, I don't know if Oracle has seen it or understands that the update was to make fapolicyd work correctly as opposed to some mail agent.

You can also dump the trust database to examine what fapolicyd understands to be trusted. Run: fapolicyd-cli --dump-db You should be able to see if libreoffice is there. And if you wanted to jettison all of the updates you added manually, you can just rm -f /etc/fapolicyd/fapolicyd.trust and then touch it to make it an empty file.

Try the debug mode. It should tell you what's going on. If the output is too much and all you want is the denials, use --debug-deny and that should filter it. On a denial, you want to see 2 things. What does it think is the file type and is it trusted. If the file type is wrong, then you might want to let us know. There's not much you can do about that. But also look at the trust output. If the file is something that you yourself trust, add it to the trust db using the cli tool. I don't think the cli tool sends a notification to fapoolicyd that it just changed the database...so maybe that's a bug on our end. We'll look into it.

Let us know what you find.

[stevegrubb]

To notify to fapolicyd that the file trust database was updated, run fapolicyd-cli --update. This will cause it to reload all trust sources.

[blairconrad]

Thanks for your responses, @stevegrubb. I know my questions are too much and too general, and I appreciate the effort you're putting in. I'll gather info and respond and refine to our first comments a little later, but your latest response does touch on a specific concern I had:

To notify to fapolicyd that the file trust database was updated, run fapolicyd-cli --update. This will cause it to reload all trust sources.

Edit: I think you answered this question above, but I didn't register. You said there might be a bug that is blocking the nodification from fapolicyd-cli --file add…: ~~Indeed. So this is required even after running fapolicyd-cli --file add …?~~

And will the change take effect immediately? That is, if I

fapolicyd-cli --file add newfile
fapolicyd-cli --update
command-that-accesses-newfile

Should I expect fapolicyd to grant access to newfile no matter how big the trust sources are, and how quickly I run the last command? In my early testing, it did not seem that it was so. I was denied access to newfile sometimes, and my interpretation was that I was too quick: that I was beating fapolicyd's reload.

[blairconrad]

Hello, this is really too much to answer in one reply.

Agreed. Sorry.

One of the factors may be that Oracle doesn't have all the pieces in place or they have modified permissions or something. I don't know what they do. That said, which set of rules are you using?

We're using an augmented known-libs rules file. Customizations are

allow perm=any all : dir=/opt/agfa : all trust=0

added at the very top. This is the rule I mentioned in my original question. Our installer is overly complicated and adds files piecemeal into /opt/agfa, modifies them, runs them, and so forth. One complicating factor was that some python files are dumped in, then when we exercise them during the installer, they are compiled, so all of a sudden the .pyc files exist and aren't trusted. I iterated using the fapolicyd-cli command to add files, but it kept growing, and so we settled on adding the new rule. I'm not proud of it.

Additional rules appear as follows:

@@ -31,6 +38,9 @@

 # Allow any program to open trusted language files
 allow perm=open all : ftype=%languages trust=1
+allow perm=any all : ftype=application/java-archive all trust=0
+allow perm=any all : ftype=application/x-java-applet all trust=0
+allow perm=any all : ftype=text/x-java all trust=0
 deny_audit perm=any all : ftype=%languages

I did not add them (they showed up with the OVA I was given). They seem permissive, but certainly shouldn't be denying access…

Fapolicyd has a debug mode that may be helpful to understanding what's happening.

Yes. This has been very helpful in setting things up. I iterated with it many many times before settling on the first bonus rule I mentioned above. Unfortunately, the problems we're left with have been very hard to diagnose using this approach, as our installer is performing several automated tasks including adjusting the fapolicyd rules and restarting the service. And in the end, the operations that are being denied are later allowed, leading me to ask all my questions about timing so on. Still, I will keep this approach in my back pocket.

It uses the format provided in syslog_format in the config file. I have mine set to rule,dec,perm,uid,pid,exe,:,path,ftype,trust. Getting trust output is important to figuring out what's going on. Run /usr/sbin/fapolicyd --debug as root on the command line.

Ah, our format has been set to "rule,dec,perm,auid,pid,exe,:,path,ftype". On a related note, I was able to extract some output using ausearch in this format, but I don't think it's sufficient to diagnose what's going on, and the systems that were unhappy with cut are now content. It tells me that cut was denied, but I don't get much more out of it. Still:

----
node=jniews81x1.mitra.com type=PROCTITLE msg=audit(04/19/2021 13:25:22.196:5166) : proctitle=/bin/bash install.sh /opt/agfa/installer/lastInstalledCwpRpms
node=jniews81x1.mitra.com type=PATH msg=audit(04/19/2021 13:25:22.196:5166) : item=1 name=/lib64/ld-linux-x86-64.so.2 inode=25167549 dev=08:02 mode=file,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:ld_so_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
node=jniews81x1.mitra.com type=PATH msg=audit(04/19/2021 13:25:22.196:5166) : item=0 name=/usr/bin/cut inode=16981814 dev=08:02 mode=file,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:bin_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
node=jniews81x1.mitra.com type=CWD msg=audit(04/19/2021 13:25:22.196:5166) : cwd=/opt/agfa/CWP_RPMS/2000.0.59.team_coyote_20210416
node=jniews81x1.mitra.com type=SYSCALL msg=audit(04/19/2021 13:25:22.196:5166) : arch=x86_64 syscall=execve success=no exit=EPERM(Operation not permitted) a0=0x555bd78fe0c0 a1=0x555bd78fb980 a2=0x555bd78fb210 a3=0x0 items=2 ppid=31108 pid=31110 auid=agfaservice uid=agfa gid=agfa euid=agfa suid=agfa fsuid=agfa egid=agfa sgid=agfa fsgid=agfa tty=pts3 ses=4 comm=bash exe=/usr/bin/bash subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
node=jniews81x1.mitra.com type=FANOTIFY msg=audit(04/19/2021 13:25:22.196:5166) : resp=deny
----
node=jniews81x1.mitra.com type=PROCTITLE msg=audit(04/19/2021 13:25:22.206:5167) : proctitle=/bin/bash install.sh /opt/agfa/installer/lastInstalledCwpRpms
node=jniews81x1.mitra.com type=PATH msg=audit(04/19/2021 13:25:22.206:5167) : item=0 name=/usr/bin/cut inode=16981814 dev=08:02 mode=file,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:bin_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
node=jniews81x1.mitra.com type=CWD msg=audit(04/19/2021 13:25:22.206:5167) : cwd=/opt/agfa/CWP_RPMS/2000.0.59.team_coyote_20210416
node=jniews81x1.mitra.com type=SYSCALL msg=audit(04/19/2021 13:25:22.206:5167) : arch=x86_64 syscall=openat success=no exit=EPERM(Operation not permitted) a0=0xffffff9c a1=0x555bd78fe0c0 a2=O_RDONLY a3=0x0 items=1 ppid=31108 pid=31110 auid=agfaservice uid=agfa gid=agfa euid=agfa suid=agfa fsuid=agfa egid=agfa sgid=agfa fsgid=agfa tty=pts3 ses=4 comm=bash exe=/usr/bin/bash subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=unsuccessful-access
node=jniews81x1.mitra.com type=FANOTIFY msg=audit(04/19/2021 13:25:22.206:5167) : resp=deny
----
node=jniews81x1.mitra.com type=PROCTITLE msg=audit(04/19/2021 13:25:22.207:5168) : proctitle=/bin/bash install.sh /opt/agfa/installer/lastInstalledCwpRpms
node=jniews81x1.mitra.com type=PATH msg=audit(04/19/2021 13:25:22.207:5168) : item=0 name=/usr/bin/cut inode=16981814 dev=08:02 mode=file,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:bin_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
node=jniews81x1.mitra.com type=CWD msg=audit(04/19/2021 13:25:22.207:5168) : cwd=/opt/agfa/CWP_RPMS/2000.0.59.team_coyote_20210416
node=jniews81x1.mitra.com type=SYSCALL msg=audit(04/19/2021 13:25:22.207:5168) : arch=x86_64 syscall=openat success=no exit=EPERM(Operation not permitted) a0=0xffffff9c a1=0x555bd78fe0c0 a2=O_RDONLY a3=0x0 items=1 ppid=31108 pid=31110 auid=agfaservice uid=agfa gid=agfa euid=agfa suid=agfa fsuid=agfa egid=agfa sgid=agfa fsgid=agfa tty=pts3 ses=4 comm=bash exe=/usr/bin/bash subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=unsuccessful-access
node=jniews81x1.mitra.com type=FANOTIFY msg=audit(04/19/2021 13:25:22.207:5168) : resp=deny
----

Next issue, does Oracle ship fapolicyd-dnf-plugin?

I think so. I see a python module of that name, and when I run dnf -v, I'm told

Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, fapolicyd, generate_completion_cache, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync

Also, we identified and fixed several problems in the file package. An update just went out this week. So, I don't know if Oracle has seen it or understands that the update was to make fapolicyd work correctly as opposed to some mail agent.

I don't really understand "as opposed to some mail agent", but good to know there's a new release.

You can also dump the trust database to examine what fapolicyd understands to be trusted. Run: fapolicyd-cli --dump-db You should be able to see if libreoffice is there. And if you wanted to jettison all of the updates you added manually, you can just rm -f /etc/fapolicyd/fapolicyd.trust and then touch it to make it an empty file.

After the fact, the trust database generally looks good, it's so far seemed to be all about the timing.

Okay, another huge comment. Sorry about that. Your responses have been very educational.

[stevegrubb]

I spoke with the team about maybe making --update implicit in --file add / --file delete / --file update. The concern is that if there was a script that walked a list of files to add them, we could update the server way too many times.

The answer is yes. It should grant access afterwards. As for how long it takes, you can see it happen in debug mode. Fapolicyd-1.0 is very fast on updates. It should be all done in a second or so. Note that if you do it in debug mode, this slows down the actual update because it has to write to stderr and scroll the screen.

Also, fapolicyd keeps statistics. It writes them to disk on shutdown. You can see how deep the queue got during an update and adjust it's size. There is a a lot of information in the README.md performance section.

[blairconrad]

The concern is that if there was a script that walked a list of files to add them, we could update the server way too many times

That's why I assumed it seemed to be separate. Makes sense to me.

The answer is yes. It should grant access afterwards. As for how long it takes, you can see it happen in debug mode. Fapolicyd-1.0 is very fast on updates. It should be all done in a second or so. […] Also, fapolicyd keeps statistics. It writes them to disk on shutdown. You can see how deep the queue got during an update and adjust it's size. There is a a lot of information in the README.md performance section.

I'll experiment more later to see what's going wrong. Thanks again.

[stevegrubb]

That said, which set of rules are you using?

We're using an augmented known-libs rules file. Customizations are

allow perm=any all : dir=/opt/agfa : all trust=0

The syntax is wrong.

allow perm=any all : dir=/opt/agfa : all trust=0
'=' is missing for field :, in line 8

If you really wanted that rule, it would be better as:

allow perm=any all : dir=/opt/agfa

However, that is a wide open door. It would be better to make the program files trusted. And perhaps using py_compile in the build system and distributing those would make things easier. I'd also consider making a file manifest of some kind and using that for scripting to enroll the files in the trust database.

added at the very top. This is the rule I mentioned in my original question. Our installer is overly complicated and adds files piecemeal into /opt/agfa, modifies them, runs them, and so forth.

You may need to stop fapolicyd during install if its not rpm based and runs freshly installed utilities. There is another open issue about making a way to update the trustdb easier. Maybe we can make installing easier in that issue?

One complicating factor was that some python files are dumped in, then when we exercise them during the installer, they are compiled, so all of a sudden the .pyc files exist and aren't trusted. I iterated using the fapolicyd-cli command to add files, but it kept growing, and so we settled on adding the new rule. I'm not proud of it.

Additional rules appear as follows:

@@ -31,6 +38,9 @@

 # Allow any program to open trusted language files
 allow perm=open all : ftype=%languages trust=1
+allow perm=any all : ftype=application/java-archive all trust=0
+allow perm=any all : ftype=application/x-java-applet all trust=0
+allow perm=any all : ftype=text/x-java all trust=0
 deny_audit perm=any all : ftype=%languages

For starters, all is not needed on the object side. The effect of these rules is to say, allow any java programs to run. This doesn't limit them in any way. I'd limit it to a directory or subject uid or something. And I'd hope that the long term fix is to make them trusted so that this rule is not needed.

On a related note, I was able to extract some output using ausearch in this format, but I don't think it's sufficient to diagnose what's going on, and the systems that were unhappy with cut are now content. It tells me that cut was denied, but I don't get much more out of it.

There is an issue open to add more information to the audit event. Originally the FANOTIFY developers wouldn't allow me to add the rule number. I posted a patch adding this and they asked for a major API change - which I have not had time to do.

Also, we identified and fixed several problems in the file package. An update just went out this week. So, I don't know if Oracle has seen it or understands that the update was to make fapolicyd work correctly as opposed to some mail agent.

I don't really understand "as opposed to some mail agent",

Mail agents use libmagic to classify mail attachments. That is how libmagic is used for the most part. So, a lot of people wouldn't make the connection that it's actually now important to keep updated.

[blairconrad]

allow perm=any all : dir=/opt/agfa : all trust=0

The syntax is wrong. '=' is missing for field :, in line 8

Well that's embarrassing. Thanks. Will amend.

However, that is a wide open door. It would be better to make the program files trusted. And perhaps using py_compile in the build system and distributing those would make things easier. I'd also consider making a file manifest of some kind and using that for scripting to enroll the files in the trust database.

Agree on all fronts.

You may need to stop fapolicyd during install if its not rpm based and runs freshly installed utilities. There is another open issue about making a way to update the trustdb easier. Maybe we can make installing easier in that issue?

For starters, all is not needed on the object side. The effect of these rules is to say, allow any java programs to run. This doesn't limit them in any way. I'd limit it to a directory or subject uid or something. And I'd hope that the long term fix is to make them trusted so that this rule is not needed.

Agree. Will feed back to team that provides these rules.

Just to add more fuel to the fire, a colleague was able to capture debugging info when cut was denied:

rule=4 dec=deny_audit perm=execute auid=1000 pid=45505 exe=/usr/bin/bash : path=/usr/lib64/ld-2.28.so ftype=application/x-sharedlib
rule=4 dec=deny_audit perm=open auid=1000 pid=45505 exe=/usr/bin/bash : path=/usr/bin/cut ftype=application/x-executable
rule=4 dec=deny_audit perm=open auid=1000 pid=45505 exe=/usr/bin/bash : path=/usr/bin/cut ftype=application/x-executable

where rule 4 is here:

1. allow perm=any all : dir=/opt/agfa : all trust=0
-> %languages=application/x-bytecode.ocaml,application/x-bytecode.python,application/java-archive,text/x-java,application/x-java-applet,text/javascript,text/x-awk,text/x-gawk,text/x-lisp,text/x-lua,text/x-m4,text/x-perl,text/x-php,text/x-python,text/x-R,text/x-ruby,text/x-script.guile,text/x-tcl,text/x-luatex,text/x-systemtap
2. allow perm=any uid=0 : dir=/var/tmp/
3. allow perm=any uid=0 trust=1 : all
4. deny_audit perm=any pattern=ld_so : all

and I also harvested the syslogs, but they don't do much more for me than confirm the debugging above:

node=... type=PROCTITLE msg=audit(04/22/2021 07:54:31.012:5092) : proctitle=/bin/bash install.sh /opt/agfa/installer/lastInstalledCwpRpms
node=... type=PATH msg=audit(04/22/2021 07:54:31.012:5092) : item=1 name=/lib64/ld-linux-x86-64.so.2 inode=25167549 dev=08:02 mode=file,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:ld_so_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
node=... type=PATH msg=audit(04/22/2021 07:54:31.012:5092) : item=0 name=/usr/bin/cut inode=16981814 dev=08:02 mode=file,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:bin_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
node=... type=CWD msg=audit(04/22/2021 07:54:31.012:5092) : cwd=/opt/agfa/CWP_RPMS/2100.0.2.team_coyote_20210420
node=... type=SYSCALL msg=audit(04/22/2021 07:54:31.012:5092) : arch=x86_64 syscall=execve success=no exit=EPERM(Operation not permitted) a0=0x55a5ff64a0a0 a1=0x55a5ff647970 a2=0x55a5ff647200 a3=0x0 items=2 ppid=45501 pid=45505 auid=agfaservice uid=agfa gid=agfa euid=agfa suid=agfa fsuid=agfa egid=agfa sgid=agfa fsgid=agfa tty=pts3 ses=5 comm=bash exe=/usr/bin/bash subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
node=... type=FANOTIFY msg=audit(04/22/2021 07:54:31.012:5092) : resp=deny
----
node=... type=PROCTITLE msg=audit(04/22/2021 07:54:31.029:5093) : proctitle=/bin/bash install.sh /opt/agfa/installer/lastInstalledCwpRpms
node=... type=PATH msg=audit(04/22/2021 07:54:31.029:5093) : item=0 name=/usr/bin/cut inode=16981814 dev=08:02 mode=file,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:bin_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
node=... type=CWD msg=audit(04/22/2021 07:54:31.029:5093) : cwd=/opt/agfa/CWP_RPMS/2100.0.2.team_coyote_20210420
node=... type=SYSCALL msg=audit(04/22/2021 07:54:31.029:5093) : arch=x86_64 syscall=openat success=no exit=EPERM(Operation not permitted) a0=0xffffff9c a1=0x55a5ff64a0a0 a2=O_RDONLY a3=0x0 items=1 ppid=45501 pid=45505 auid=agfaservice uid=agfa gid=agfa euid=agfa suid=agfa fsuid=agfa egid=agfa sgid=agfa fsgid=agfa tty=pts3 ses=5 comm=bash exe=/usr/bin/bash subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=unsuccessful-access
node=... type=FANOTIFY msg=audit(04/22/2021 07:54:31.029:5093) : resp=deny
----
node=... type=PROCTITLE msg=audit(04/22/2021 07:54:31.030:5094) : proctitle=/bin/bash install.sh /opt/agfa/installer/lastInstalledCwpRpms
node=... type=PATH msg=audit(04/22/2021 07:54:31.030:5094) : item=0 name=/usr/bin/cut inode=16981814 dev=08:02 mode=file,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:bin_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
node=... type=CWD msg=audit(04/22/2021 07:54:31.030:5094) : cwd=/opt/agfa/CWP_RPMS/2100.0.2.team_coyote_20210420
node=... type=SYSCALL msg=audit(04/22/2021 07:54:31.030:5094) : arch=x86_64 syscall=openat success=no exit=EPERM(Operation not permitted) a0=0xffffff9c a1=0x55a5ff64a0a0 a2=O_RDONLY a3=0x0 items=1 ppid=45501 pid=45505 auid=agfaservice uid=agfa gid=agfa euid=agfa suid=agfa fsuid=agfa egid=agfa sgid=agfa fsgid=agfa tty=pts3 ses=5 comm=bash exe=/usr/bin/bash subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=unsuccessful-access
node=... type=FANOTIFY msg=audit(04/22/2021 07:54:31.030:5094) : resp=deny

The very odd thing is that later, the calls to cut succeed, either when run just from the bash prompt (but as the same effective user) or via the very process that triggered them during this session. And that's without replacing cut, restarting, or otherwise reconfiguring fapolicyd. It's very confusing. Sadly, no record of it happening where we've had presence of mind to capture the "allow" logs later. Will keep an eye out more to try to figure out what's going wrong.

Thanks for listening.

[stevegrubb]

That cut execution is awfully suspicious. The program is not starting up the normal way It's leading off with ld-2.28.so where a normal execution would start with the program and then ld.so. I'd want to dig into that more. There are some notes about execution order here:

https://github.com/linux-application-whitelisting/fapolicyd/blob/master/src/library/rules.c#L968

[blairconrad]

If you've not lost your appetite for this issue, I'm back with slightly enhanced logs and a (I think) more compelling example of what I see as weirdness.

As noted, our installer sometimes fails, indicating that an Operation (is) not permitted when running /usr/bin/cut. This is reported when a java executable runs a shell script "install.sh". The relevant lines in the script are

OS_VERSION=$(cut --fields=5 --delimiter=" " /etc/system-release)
OS_MAJOR_VERSION=$(echo $OS_VERSION | cut --fields=1 --delimiter=.)

I've been able to gather fapolicyd debug output for a run when the first cut did not work:

rule=9  dec=allow      perm=execute auid=1000 pid=9558 exe=/usr/bin/bash : path=/usr/bin/cut ftype=application/x-executable
rule=19 dec=allow      perm=open    auid=1000 pid=9558 exe=/usr/bin/bash : path=/usr/bin/cut ftype=application/x-executable
rule=4  dec=deny_audit perm=execute auid=1000 pid=9558 exe=/usr/bin/bash : path=/usr/lib64/ld-2.28.so ftype=application/x-sharedlib
rule=4  dec=deny_audit perm=open    auid=1000 pid=9558 exe=/usr/bin/bash : path=/usr/bin/cut ftype=application/x-executable
rule=4  dec=deny_audit perm=open    auid=1000 pid=9558 exe=/usr/bin/bash : path=/usr/bin/cut ftype=application/x-executable

As I read it, the bash process is

1. granted access to execute /usr/bin/cut
2. granted access to open /usr/bin/cut (I imagine it needs this to execute it)
3. denied access to execute /usr/lib64/ld-2.28.so
4. denied access to open /usr/bin/cut. This looks exactly like the second line, except for the rule and the decision. (Why, I ask myself? Because of the deny above?)
5. denied access to open /usr/bin/cut. This looks exactly like the second line, except for the rule and the decision. (Why, I ask myself? Because of the deny above?) As before, Rule 4 is deny_audit perm=any pattern=ld_so : all

The audit logs spewed at this time are

node=ol8xwws3.ts.mitra.com type=PROCTITLE msg=audit(05/03/2021 13:44:29.323:6565) : proctitle=/bin/bash install.sh /opt/agfa/installer/lastInstalledCwpRpms 
node=ol8xwws3.ts.mitra.com type=PATH msg=audit(05/03/2021 13:44:29.323:6565) : item=1 name=/lib64/ld-linux-x86-64.so.2 inode=25167549 dev=08:02 mode=file,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:ld_so_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 
node=ol8xwws3.ts.mitra.com type=PATH msg=audit(05/03/2021 13:44:29.323:6565) : item=0 name=/usr/bin/cut inode=16981814 dev=08:02 mode=file,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:bin_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 
node=ol8xwws3.ts.mitra.com type=CWD msg=audit(05/03/2021 13:44:29.323:6565) : cwd=/opt/agfa/CWP_RPMS/2100.0.9.team_coyote_20210429 
node=ol8xwws3.ts.mitra.com type=SYSCALL msg=audit(05/03/2021 13:44:29.323:6565) : arch=x86_64 syscall=execve success=no exit=EPERM(Operation not permitted) a0=0x55813a1d0000 a1=0x55813a1cd900 a2=0x55813a1cd190 a3=0x0 items=2 ppid=9551 pid=9558 auid=agfaservice uid=agfa gid=agfa euid=agfa suid=agfa fsuid=agfa egid=agfa sgid=agfa fsgid=agfa tty=pts3 ses=1 comm=bash exe=/usr/bin/bash subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null) 
node=ol8xwws3.ts.mitra.com type=FANOTIFY msg=audit(05/03/2021 13:44:29.323:6565) : resp=deny 
----
node=ol8xwws3.ts.mitra.com type=PROCTITLE msg=audit(05/03/2021 13:44:29.383:6566) : proctitle=/bin/bash install.sh /opt/agfa/installer/lastInstalledCwpRpms 
node=ol8xwws3.ts.mitra.com type=PATH msg=audit(05/03/2021 13:44:29.383:6566) : item=0 name=/usr/bin/cut inode=16981814 dev=08:02 mode=file,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:bin_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 
node=ol8xwws3.ts.mitra.com type=CWD msg=audit(05/03/2021 13:44:29.383:6566) : cwd=/opt/agfa/CWP_RPMS/2100.0.9.team_coyote_20210429 
node=ol8xwws3.ts.mitra.com type=SYSCALL msg=audit(05/03/2021 13:44:29.383:6566) : arch=x86_64 syscall=openat success=no exit=EPERM(Operation not permitted) a0=0xffffff9c a1=0x55813a1d0000 a2=O_RDONLY a3=0x0 items=1 ppid=9551 pid=9558 auid=agfaservice uid=agfa gid=agfa euid=agfa suid=agfa fsuid=agfa egid=agfa sgid=agfa fsgid=agfa tty=pts3 ses=1 comm=bash exe=/usr/bin/bash subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=unsuccessful-access 
node=ol8xwws3.ts.mitra.com type=FANOTIFY msg=audit(05/03/2021 13:44:29.383:6566) : resp=deny 
----
node=ol8xwws3.ts.mitra.com type=PROCTITLE msg=audit(05/03/2021 13:44:29.383:6567) : proctitle=/bin/bash install.sh /opt/agfa/installer/lastInstalledCwpRpms 
node=ol8xwws3.ts.mitra.com type=PATH msg=audit(05/03/2021 13:44:29.383:6567) : item=0 name=/usr/bin/cut inode=16981814 dev=08:02 mode=file,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:bin_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 
node=ol8xwws3.ts.mitra.com type=CWD msg=audit(05/03/2021 13:44:29.383:6567) : cwd=/opt/agfa/CWP_RPMS/2100.0.9.team_coyote_20210429 
node=ol8xwws3.ts.mitra.com type=SYSCALL msg=audit(05/03/2021 13:44:29.383:6567) : arch=x86_64 syscall=openat success=no exit=EPERM(Operation not permitted) a0=0xffffff9c a1=0x55813a1d0000 a2=O_RDONLY a3=0x0 items=1 ppid=9551 pid=9558 auid=agfaservice uid=agfa gid=agfa euid=agfa suid=agfa fsuid=agfa egid=agfa sgid=agfa fsgid=agfa tty=pts3 ses=1 comm=bash exe=/usr/bin/bash subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=unsuccessful-access 
node=ol8xwws3.ts.mitra.com type=FANOTIFY msg=audit(05/03/2021 13:44:29.383:6567) : resp=deny 

which is similar what we discussed before. @stevegrubb, you'd raised concerns over the first record, saying that normal order of execution would be to start with the program and then ld.so, but I think that's what the fapolicyd output shows, no?

Fun fact: the second call to cut has this fapolicyd output:

rule=9  dec=allow perm=execute auid=1000 pid=9561 exe=/usr/bin/bash : path=/usr/bin/cut ftype=application/x-executable
rule=19 dec=allow perm=open    auid=1000 pid=9561 exe=/usr/bin/bash : path=/usr/bin/cut ftype=application/x-executable
rule=9  dec=allow perm=execute auid=1000 pid=9561 exe=/usr/bin/bash : path=/usr/lib64/ld-2.28.so ftype=application/x-sharedlib
rule=7  dec=allow perm=open    auid=1000 pid=9561 exe=/usr/bin/bash : path=/usr/lib64/ld-2.28.so ftype=application/x-sharedlib
rule=19 dec=allow perm=open    auid=1000 pid=9561 exe=/usr/bin/cut : path=/etc/ld.so.cache ftype=application/octet-stream
rule=7  dec=allow perm=open    auid=1000 pid=9561 exe=/usr/bin/cut : path=/usr/lib64/libc-2.28.so ftype=application/x-sharedlib
rule=19 dec=allow perm=open    auid=1000 pid=9561 exe=/usr/bin/cut : path=/usr/share/locale/locale.alias ftype=text/plain
rule=19 dec=allow perm=open    auid=1000 pid=9561 exe=/usr/bin/cut : path=/usr/lib/locale/en_US.utf8/LC_IDENTIFICATION ftype=application/octet-stream
rule=19 dec=allow perm=open    auid=1000 pid=9561 exe=/usr/bin/cut : path=/usr/lib64/gconv/gconv-modules.cache ftype=application/octet-stream
rule=19 dec=allow perm=open    auid=1000 pid=9561 exe=/usr/bin/cut : path=/usr/lib/locale/en_US.utf8/LC_MEASUREMENT ftype=application/octet-stream
rule=19 dec=allow perm=open    auid=1000 pid=9561 exe=/usr/bin/cut : path=/usr/lib/locale/en_US.utf8/LC_TELEPHONE ftype=application/octet-stream
rule=19 dec=allow perm=open    auid=1000 pid=9561 exe=/usr/bin/cut : path=/usr/lib/locale/en_US.utf8/LC_ADDRESS ftype=application/octet-stream
rule=19 dec=allow perm=open    auid=1000 pid=9561 exe=/usr/bin/cut : path=/usr/lib/locale/en_US.utf8/LC_NAME ftype=application/octet-stream
rule=19 dec=allow perm=open    auid=1000 pid=9561 exe=/usr/bin/cut : path=/usr/lib/locale/en_US.utf8/LC_PAPER ftype=application/octet-stream
rule=19 dec=allow perm=open    auid=1000 pid=9561 exe=/usr/bin/cut : path=/usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES ftype=application/octet-stream
rule=19 dec=allow perm=open    auid=1000 pid=9561 exe=/usr/bin/cut : path=/usr/lib/locale/en_US.utf8/LC_MONETARY ftype=application/octet-stream
rule=19 dec=allow perm=open    auid=1000 pid=9561 exe=/usr/bin/cut : path=/usr/lib/locale/en_US.utf8/LC_COLLATE ftype=application/octet-stream
rule=19 dec=allow perm=open    auid=1000 pid=9561 exe=/usr/bin/cut : path=/usr/lib/locale/en_US.utf8/LC_TIME ftype=application/octet-stream
rule=19 dec=allow perm=open    auid=1000 pid=9561 exe=/usr/bin/cut : path=/usr/lib/locale/en_US.utf8/LC_NUMERIC ftype=application/octet-stream
rule=19 dec=allow perm=open    auid=1000 pid=9561 exe=/usr/bin/cut : path=/usr/lib/locale/en_US.utf8/LC_CTYPE ftype=application/octet-stream

We see things start the same, but on the 3rd line, rule 9 is invoked, allowing access to /usr/lib64/ld-2.28.so instead of denying it due to rule 4. After that, access to cut is allowed, and execution proceeds.

(Rule 9: allow perm=execute all : trust=1)

There was no reconfiguration or restart of the fapolicyd daemon between calls.

On a subsequent run of the installer, invoking the same shell script, with no intervening changes to the fapolicyd process, its configuration, or trust store, the first call to /usr/bin/cut succeeds with fapolicyd debug output analogous to the second run above.

Sorry to treat you like my personal support line, but the changing behaviour between subsequent invocations of the utility makes me feel like there may be something going wrong inside fapolicyd, and I not sure how to address it.

[stevegrubb]

We are still thinking about this issue. For rpm based software, we can make installation do the right thing by hooking rpm internals and telling fapolicyd what's happening. For other software, we'd need to do a similar trick which is really not possible. For the moment, I'd say the best thing is install with fapolicyd stopped, update the trust db afterwards, then restart fapolicyd. As I said, we are talking it over and may have a better plan soon.

[stevegrubb]

I think this issue can be closed. There is an rpm extension that updates the database during package installation.