augenrules --load exit code 1

[mrvantage]

I'm having a similar issue as #355 since August 14th on Arch linux. I'm not sure opening a new issue is the correct action to take, since that issue is already closed.

Audit was upgraded to version 4.0.2:

[2024-08-14T07:42:17+0200] [ALPM] upgraded audit (4.0.1-3 -> 4.0.2-1)

Since then auditd service started failing:

-- Boot 70db635ffe114a9d866e08bdc44101e8 --
aug 14 07:58:19 arch systemd[1]: Dependency failed for Security Audit Logging Service.
aug 14 07:58:19 arch systemd[1]: auditd.service: Job auditd.service/start failed with result 'dependency'.

The failing dependency seems to be the audit-rules service:

[root@arch]# systemctl status audit-rules
× audit-rules.service - Load Audit Rules
     Loaded: loaded (/usr/lib/systemd/system/audit-rules.service; disabled; preset: disabled)
     Active: failed (Result: exit-code) since Wed 2024-08-21 07:31:19 CEST; 1h 7min ago
 Invocation: d6405f7057354513a025c05a96cb4a19
       Docs: man:auditctl(8)
             https://github.com/linux-audit/audit-documentation
    Process: 1397 ExecStart=/usr/bin/augenrules --load (code=exited, status=1/FAILURE)
   Main PID: 1397 (code=exited, status=1/FAILURE)
   Mem peak: 4.3M
        CPU: 14ms

aug 21 07:31:19 arch systemd[1]: Starting Load Audit Rules...
aug 21 07:31:19 arch augenrules[1397]: /usr/bin/augenrules: No change
aug 21 07:31:19 arch systemd[1]: audit-rules.service: Main process exited, code=exited, status=1/FAILURE
aug 21 07:31:19 arch systemd[1]: audit-rules.service: Failed with result 'exit-code'.
aug 21 07:31:19 arch systemd[1]: Failed to start Load Audit Rules.

The issue seems to be that augenrules --load doesn't exit with code 0:

[root@arch]# /usr/bin/augenrules --load
/usr/bin/augenrules: No change
[root@arch]# echo $?
1

[variasdesign]

I have the same issue and I thought I was going crazy. Using Arch Linux too. I've been using the same rules for a while until the upgrade to 4.0.2-1, when they stopped working. I've tried adding them manually using auditctl and it seems the syntax has broke:

# auditctl -a exit,always -F dir=/home -F euid=0 -C auid!=obj_uid -k sudoAbuse
Append rule - bad keyword exit,always

Let me know if I can be of any help. Thanks.

[stevegrubb]

@variasdesign, If you switch the rule to "always,exit", does it work? (This part of the code has not changed in a long, long time.)

[stevegrubb]

@mrvantage are you running in immutable mode? Run auditctl -s and post the enabled line. The reason it was closed is a fix was put in place to detect immutable mode and handle it. If you run augenrules by hand, do you get any better information?

[mrvantage]

@mrvantage are you running in immutable mode? Run auditctl -s and post the enabled line.

[root@arch]#  auditctl -s
enabled 0
....

If you run augenrules by hand, do you get any better information?

That's what I did in the original description of the issue

By the way, im not even sure why I have audit installed. I guess it's a dependency of something else. Also I don't really know what is the effect of this service failing, I haven't noticed anything broken. It's just this line of the failed service that I can see when turning off my laptop.

[variasdesign]

@variasdesign, If you switch the rule to "always,exit", does it work? (This part of the code has not changed in a long, long time.)

Thanks for replying. I get the following output:

# auditctl -a always,exit -F dir=/home -F euid=0 -C auid!=obj_uid -k sudoAbuse
dir can only be used with exit filter list
There was an error while processing parameters

[stevegrubb]

@mrvantage I run augenrules --load and I get the no change message but it exits with 0. I don't know what is happening with your system. Maybe the distro doesn't add default rules? In any event, depending on how your distro packaged the audit software, you should be able to either uninstall the deamon or turn it off if you don't actually need security audit logs. What is required is the audit libraries since a lot of user space is linked to it.

[stevegrubb]

@variasdesign I run your rule on fedora and it has no problems with it.

auditctl -D No rules auditctl -a always,exit -F dir=/home -F euid=0 -C auid!=obj_uid -k sudoAbuse auditctl -l -a always,exit -S all -F dir=/home -F euid=0 -C auid!=obj_uid -F key=sudoAbuse

Maybe you should report it to your distro in case they've modified the source code?

[variasdesign]

@stevegrubb Yep, it seems that is distro-related. Thanks for your help!

[mrvantage]

I see, started to get the feeling it might be. Thanks for the swift support @stevegrubb

[Maryse47]

It still would be nice to investigate what's causing this. Arch didn't do anything other than bumping version from 4.0.1 -> 4.0.2 . Version 4.0.1 still works.

You may see arch build config here, excerpt:

    --enable-gssapi-krb5=yes
    --enable-systemd=yes
    --enable-zos-remote
    --libexecdir=/usr/lib/audit
    --prefix=/usr
    --runstatedir=/run
    --sbindir=/usr/bin
    --sysconfdir=/etc
    --with-apparmor=yes
    --with-io_uring=yes
    --with-libcap-ng=yes
    --with-python3=yes

They use one patch which was also used before regression.

@stevegrubb does anything above look suspicious for you?

[stevegrubb]

Maybe edit augenrules as root and add set -x just below the shebang. This will cause bash to trace the commands. It might have a clue in the output. The only part to look at is after it outputs "No change".

[mrvantage]

I just investigated this on my system. Also note this is impacting only one of my machines. I created an audit rule to troubleshoot another issue some time ago and I totally forgot about it.

Debugging the auditgen script I can see it fails on the command /sbin/auditctl -R /etc/audit/audit.rules, which returns exit code 1

Note, that this one rule is the only one I have, so I think this has to be the culprit. Did something change in regards to syntax, or syntax checking in 4.0.2?

Anyway, the rule is the following:

-a exit,always -F arch=b64 -S execve -F path=/usr/bin/git

Edit:

Passing the rule to auditctl outputs:

auditctl -a exit,always -F arch=b64 -S execve -F path=/usr/bin/git
Append rule - bad keyword exit,always
There was an error while processing parameters

[wereii]

Maybe edit augenrules as root and add set -x just below the shebang. This will cause bash to trace the commands. It might have a clue in the output. The only part to look at is after it outputs "No change". @stevegrubb

A note that this Arch bug is most definitely occurring somewhere in libaudit and not in the rule gen helpers - as I wrote in Archs gitlab issue it does not matter if rules are loaded from files (audit-rules.service, augenrules) or manually ran against auditctl (which as I understand it, is the final place where the rules are truly applied/executed).

Note, that this one rule is the only one I have, so I think this has to be the culprit. Did something change in regards to syntax, or syntax checking in 4.0.2? @mrvantage

From my findings, nothing works except the old file watch (-w) - -a exit,always, -A always,exit, -A exclude,always, ... it all fails

@stevegrubb could we get the current compiler versions for the fedora build of audit-userspace ? I don't expect this has something to do with the compiler but it's an easy one to rule out.

[stevegrubb]

@mrvantage , the code really doesn't care about the order. See: https://github.com/linux-audit/audit-userspace/blob/master/src/auditctl.c#L196 Which makes me wonder why does yours complain when it shouldn't? I notice that arch appears to be building from the github archive where Fedora is using the release tar file from my people.redhat.com page. Which makes it hard to compare SHA512 numbers. I'd almost want to get the two files untarred side by side to compare with diff. Fedora is using gcc-13.3.1 through gcc-14.2.1 depending on which Fedora release we are talking about. I don't think the compiler is the issue.

[stevegrubb]

@wereii Fedora is using gcc-13.3.1 through gcc-14.2.1 depending on which Fedora release we are talking about. I don't think the compiler is the issue. I wonder if there are kernel differences? The code of auditctl does the lookups of the list and action. As pasted in the preceding answer, it doesn't care about the order. I wrote the code this way intentionally because I couldn't remember the order all the time and fixed it so it always worked. The SCAP guys demanded a known order so the scanner could detect bad rules. The compromise was make all sample rules consistent and document the correct order, but otherwise disregard the order.

[wereii]

So, just for posterity, Arch currently has gcc version 14.2.1.

I notice that arch appears to be building from the github archive where Fedora is using the release tar file from my people.redhat.com page.

These two sources do differ according to diff, which is weird to say at least.

I've just edited the arch's package build with the release tar from your people page and there is no difference for me.

# auditctl -a always,exit -F arch=b64 -F path=/etc/shadow -F perm=wa 
path can only be used with exit filter list
There was an error while processing parameters

Note: I've also noticed a somewhat weird handling of cli args:

auditctl any strings here that do_not contain dash -a always,exit -F arch=b64 -F path=/etc/shadow -F perm=wa

which results in the same error as above. I consider that unexpected as I would expect auditctl to fail with unknown arguments/flags first. Though I am guessing that auditctl just parses the args "lazily" and first looks at known args like -a until it exhausts all matching/expected and then errors out (as it does with for example -w which works in this version). Unsure if this is in any way relevant.

(Also sorry about kind of polluting your issue tracker here with something that should probably be handled in Arch's, the issue just seemed to have more traction here).

[Maryse47]

I think two things that can help finding the culprit are:

1. Rebuild previous version (4.0.1) against current Arch toolchain and see if it still works.
2. Play with git bisect to find which commit between 4.0.1-4.0.2 is causing this behavior on Arch.

[stevegrubb]

@wereii Regarding the parsing of command line parameters, auditctl uses getopt_long from glibc. So, if getopt_long takes that kind of text and doesn't error on it, auditctl keeps processing until there is no more. When I run that bad command, Fedora returns: parameter passed without an option given...which is different than what you are getting.

@Maryse47 both options sound good. Maybe there's something different about the C library now? All of my testing is against glibc-2.38.

The patch mentioned above only adjusts some paths for plugins and shouldn't matter. That said, I was curious if the paths in augenrules are correct?

[wereii]

1: Rebuilding older 4.0.1 on my Arch results in the correct behavior of auditctl (thanks for the idea, didn't occur to me as an easy way to discern if something specific to current Arch might be causing this)

I will see if I can pinpoint the commit with bisect.

[wereii]

Bisecting is not stable, three times I arrived to a different commit, but I noticed that if I do a whole clean build of the 4.0.2 audit arch package (test that it still fails, which it does) then do a manual make clean in the source and rerun the packaging process (which does rebuild the source) without tar extraction, install the rebuilt package and I get a fully working auditctl.

E: make distclean instead of clean does not help. It's literally: build it once, then enter the src/audit-userspace-4.0.2, run make clean, force rebuild package without source extraction and install with makepkg -efi, tada it now works.

[stevegrubb]

That is a really interesting find. I wonder what this means? I'm going to move this issue to open so that its more discoverable in case others have the same problem and might offer new clues.

[ss23]

The difference between the first run of makepkg and the one after make clean is in the generated flagtabs.h file. After make clean and in the second run of makepkg with --noextract --force the prepare() function is not called again and so autoreconf is not done but the configure script is called again and it generates flagtabs.h again albeit correctly this time.

[wereii]

Thanks for looking into it @ss23 I am not well versed with configure/autoconf stuff

So, the question is, why on arch, does the first configure not generate flagtabs.h correctly (?)

[stevegrubb]

From a developer perspective, I'd be curious what the diff is between the two files because that might point to a fix. The autoreconf is necessary when the audit source code is taken from github. Something has to turn the configure.ac into configure. The file from my people page already has the conversion.

The audit system uses a lot of lookup tables. Way back around 2008, we noticed that as the tables were getting bigger, it was taking too much time for ausearch to translate things. The answer that Miloslav came up with is to make a program, gen_tables, that compiled the lookup strings into a sorted in-memory btree. The convention was to add an 's' to the header's name to denote the compiled header. So, flag_tab.h becomes flag_tabs.h.

I can't think of a good reason why it's wrong the first time and OK the second unless there is something in the environment during the package build that is not there when run by hand. Maybe a diff between them will give a clue? Rename the first generated flag_tabs.h to flag_tabs.h.orig and then do the make clean && make && diff -u flag_tabs.h.orig flag_tabs.h

[wereii]

--- flagtabs.h.orig	2024-09-02 22:28:06.540685969 +0200
+++ flagtabs.h	2024-09-02 22:28:59.101524936 +0200
@@ -1,10 +1,10 @@
 /* This is a generated file, see Makefile.am for its inputs. */
-static const char flag_strings[] = "access\0atomic\0continue\0create\0directory\0follow\0noalt\0open\0parent";
+static const char flag_strings[] = "exclude\0exit\0filesystem\0io_uring\0task\0user";
 static const unsigned flag_s2i_s[] = {
-	0,7,14,23,30,40,47,53,58,
+	0,8,13,24,33,38,
 };
 static const int flag_s2i_i[] = {
-	1024,64,4,512,2,1,32,256,16,
+	5,4,6,7,1,0,
 };
 static int flag_s2i(const char *s, int *value) {
 	size_t len, i;
@@ -17,15 +17,12 @@
 		copy[i] = GT_ISUPPER(c) ? c - 'A' + 'a' : c;
 	}
 	copy[i] = 0;
-	return s2i__(flag_strings, flag_s2i_s, flag_s2i_i, 9, copy, value);
+	return s2i__(flag_strings, flag_s2i_s, flag_s2i_i, 6, copy, value);
 	}
 }
-static const int flag_i2s_i[] = {
-	1,2,4,16,32,64,256,512,1024,
-};
-static const unsigned flag_i2s_s[] = {
-	40,30,14,58,47,7,53,23,0,
+static const unsigned flag_i2s_direct[] = {
+	38,33,-1u,-1u,8,0,13,24,
 };
 static const char *flag_i2s(int v) {
-	return i2s_bsearch__(flag_strings, flag_i2s_i, flag_i2s_s, 9, v);
+	return i2s_direct__(flag_strings, flag_i2s_direct, 0, 7, v);
 }

considering it generates a valid file (in that it compiles) and even the flag_strings look valid it looks like some weird caching issue or "argument misdirection" (it processes a different file)

[wereii]

So with this diff across the whole src (src-orig is first run ie faulty):

diff -x '*.o' -x '*.so*' -x '*.a' -x '*.log' -ur src-orig/ src/
diff --color -x '*.o' -x '*.so*' -x '*.a' -x '*.log' -ur src-orig/audit-userspace-4.0.2/auparse/flagtabs.h src/audit-userspace-4.0.2/auparse/flagtabs.h
--- src-orig/audit-userspace-4.0.2/auparse/flagtabs.h	2024-09-02 23:34:56.777729039 +0200
+++ src/audit-userspace-4.0.2/auparse/flagtabs.h	2024-09-02 23:41:51.781076576 +0200
@@ -1,6 +1,6 @@
 /* This is a generated file, see Makefile.am for its inputs. */
-static const char flag_strings[] = "access\0atomic\0continue\0create\0directory\0follow\0noalt\0open\0parent";
+static const char flag_strings[] = "exclude\0exit\0filesystem\0io_uring\0task\0user";
 static const struct transtab flag_table[] = {
-	{1,40},{2,30},{4,14},{16,58},{32,47},{64,7},{256,53},{512,23},{1024,0},
+	{1,33},{4,8},{0,38},{5,0},{6,13},{7,24},
 };
 #define FLAG_NUM_ENTRIES (sizeof(flag_table) / sizeof(*flag_table))
Binary files src-orig/audit-userspace-4.0.2/auparse/gen_flagtabs_h and src/audit-userspace-4.0.2/auparse/gen_flagtabs_h differ
Only in src-orig/audit-userspace-4.0.2: autom4te.cache
diff --color -x '*.o' -x '*.so*' -x '*.a' -x '*.log' -ur src-orig/audit-userspace-4.0.2/lib/.deps/gen_flagtabs_h-gen_tables.Po src/audit-userspace-4.0.2/lib/.deps/gen_flagtabs_h-gen_tables.Po
--- src-orig/audit-userspace-4.0.2/lib/.deps/gen_flagtabs_h-gen_tables.Po	2024-09-02 23:34:56.784395815 +0200
+++ src/audit-userspace-4.0.2/lib/.deps/gen_flagtabs_h-gen_tables.Po	2024-09-02 23:41:50.431054917 +0200
@@ -1,16 +1,16 @@
-../lib/gen_flagtabs_h-gen_tables.o: ../lib/gen_tables.c \
- /usr/include/stdc-predef.h ../config.h /usr/include/assert.h \
- /usr/include/features.h /usr/include/features-time64.h \
- /usr/include/bits/wordsize.h /usr/include/bits/timesize.h \
- /usr/include/sys/cdefs.h /usr/include/bits/long-double.h \
- /usr/include/gnu/stubs.h /usr/include/gnu/stubs-64.h \
- /usr/include/ctype.h /usr/include/bits/types.h \
- /usr/include/bits/typesizes.h /usr/include/bits/time64.h \
- /usr/include/bits/endian.h /usr/include/bits/endianness.h \
- /usr/include/bits/types/locale_t.h /usr/include/bits/types/__locale_t.h \
- /usr/include/errno.h /usr/include/bits/errno.h \
- /usr/include/linux/errno.h /usr/include/asm/errno.h \
- /usr/include/asm-generic/errno.h /usr/include/asm-generic/errno-base.h \
+gen_flagtabs_h-gen_tables.o: gen_tables.c /usr/include/stdc-predef.h \
+ ../config.h /usr/include/assert.h /usr/include/features.h \
+ /usr/include/features-time64.h /usr/include/bits/wordsize.h \
+ /usr/include/bits/timesize.h /usr/include/sys/cdefs.h \
+ /usr/include/bits/long-double.h /usr/include/gnu/stubs.h \
+ /usr/include/gnu/stubs-64.h /usr/include/ctype.h \
+ /usr/include/bits/types.h /usr/include/bits/typesizes.h \
+ /usr/include/bits/time64.h /usr/include/bits/endian.h \
+ /usr/include/bits/endianness.h /usr/include/bits/types/locale_t.h \
+ /usr/include/bits/types/__locale_t.h /usr/include/errno.h \
+ /usr/include/bits/errno.h /usr/include/linux/errno.h \
+ /usr/include/asm/errno.h /usr/include/asm-generic/errno.h \
+ /usr/include/asm-generic/errno-base.h \
  /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/include/limits.h \
  /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/include/syslimits.h \
  /usr/include/limits.h /usr/include/bits/libc-header-start.h \
@@ -65,17 +65,17 @@
  /usr/include/sys/ttydefaults.h /usr/include/linux/mount.h \
  /usr/include/linux/types.h /usr/include/asm/types.h \
  /usr/include/asm-generic/types.h /usr/include/asm-generic/int-ll64.h \
- ../lib/gen_tables.h ../lib/libaudit.h /usr/include/sys/socket.h \
+ gen_tables.h libaudit.h /usr/include/sys/socket.h \
  /usr/include/bits/types/struct_iovec.h /usr/include/bits/socket.h \
  /usr/include/bits/socket_type.h /usr/include/bits/sockaddr.h \
  /usr/include/bits/types/struct_osockaddr.h /usr/include/linux/netlink.h \
  /usr/include/linux/const.h /usr/include/linux/audit.h \
  /usr/include/linux/elf-em.h /usr/include/syslog.h \
  /usr/include/sys/syslog.h /usr/include/bits/syslog-path.h \
- ../lib/audit_logging.h ../lib/audit-records.h auparse-defs.h \
+ audit_logging.h audit-records.h ../auparse/auparse-defs.h \
  /usr/include/time.h /usr/include/bits/time.h \
  /usr/include/bits/types/struct_tm.h \
- /usr/include/bits/types/struct_itimerspec.h ../lib/../auparse/flagtab.h
+ /usr/include/bits/types/struct_itimerspec.h flagtab.h
 /usr/include/stdc-predef.h:
 ../config.h:
 /usr/include/assert.h:
@@ -195,8 +195,8 @@
 /usr/include/asm/types.h:
 /usr/include/asm-generic/types.h:
 /usr/include/asm-generic/int-ll64.h:
-../lib/gen_tables.h:
-../lib/libaudit.h:
+gen_tables.h:
+libaudit.h:
 /usr/include/sys/socket.h:
 /usr/include/bits/types/struct_iovec.h:
 /usr/include/bits/socket.h:
@@ -210,11 +210,11 @@
 /usr/include/syslog.h:
 /usr/include/sys/syslog.h:
 /usr/include/bits/syslog-path.h:
-../lib/audit_logging.h:
-../lib/audit-records.h:
-auparse-defs.h:
+audit_logging.h:
+audit-records.h:
+../auparse/auparse-defs.h:
 /usr/include/time.h:
 /usr/include/bits/time.h:
 /usr/include/bits/types/struct_tm.h:
 /usr/include/bits/types/struct_itimerspec.h:
-../lib/../auparse/flagtab.h:
+flagtab.h:
diff --color -x '*.o' -x '*.so*' -x '*.a' -x '*.log' -ur src-orig/audit-userspace-4.0.2/lib/flagtabs.h src/audit-userspace-4.0.2/lib/flagtabs.h
--- src-orig/audit-userspace-4.0.2/lib/flagtabs.h	2024-09-02 23:34:56.784395815 +0200
+++ src/audit-userspace-4.0.2/lib/flagtabs.h	2024-09-02 23:41:50.481055718 +0200
@@ -1,10 +1,10 @@
 /* This is a generated file, see Makefile.am for its inputs. */
-static const char flag_strings[] = "access\0atomic\0continue\0create\0directory\0follow\0noalt\0open\0parent";
+static const char flag_strings[] = "exclude\0exit\0filesystem\0io_uring\0task\0user";
 static const unsigned flag_s2i_s[] = {
-	0,7,14,23,30,40,47,53,58,
+	0,8,13,24,33,38,
 };
 static const int flag_s2i_i[] = {
-	1024,64,4,512,2,1,32,256,16,
+	5,4,6,7,1,0,
 };
 static int flag_s2i(const char *s, int *value) {
 	size_t len, i;
@@ -17,15 +17,12 @@
 		copy[i] = GT_ISUPPER(c) ? c - 'A' + 'a' : c;
 	}
 	copy[i] = 0;
-	return s2i__(flag_strings, flag_s2i_s, flag_s2i_i, 9, copy, value);
+	return s2i__(flag_strings, flag_s2i_s, flag_s2i_i, 6, copy, value);
 	}
 }
-static const int flag_i2s_i[] = {
-	1,2,4,16,32,64,256,512,1024,
-};
-static const unsigned flag_i2s_s[] = {
-	40,30,14,58,47,7,53,23,0,
+static const unsigned flag_i2s_direct[] = {
+	38,33,-1u,-1u,8,0,13,24,
 };
 static const char *flag_i2s(int v) {
-	return i2s_bsearch__(flag_strings, flag_i2s_i, flag_i2s_s, 9, v);
+	return i2s_direct__(flag_strings, flag_i2s_direct, 0, 7, v);
 }
Binary files src-orig/audit-userspace-4.0.2/lib/gen_flagtabs_h and src/audit-userspace-4.0.2/lib/gen_flagtabs_h differ

I can see that the first run often has extra ../lib/ in some paths. The most prominent is the existence of another flagtab.h in auparse/ (which has the keywords of the faulty flagtabs.h) which shows up in this diff multiple times:

- /usr/include/bits/types/struct_itimerspec.h ../lib/../auparse/flagtab.h
+ /usr/include/bits/types/struct_itimerspec.h flagtab.h

-../lib/../auparse/flagtab.h:
+flagtab.h:

[stevegrubb]

Given that auparse/flagtabs.h is for RHEL4, maybe it's safe to just drop that file at this point. I suppose RHEL4 has been out of support for a while now. If that does the trick, maybe I can do a new release soonish.

Any chance ccache is involved?

[wereii]

Any chance ccache is involved?

not on my machine

[Foxboron]

I can't think of a good reason why it's wrong the first time and OK the second unless there is something in the environment during the package build that is not there when run by hand.

Wild guess; dependency issue? All Arch packages are run with -j$(nproc) while running this by hand probably just does -j1 implicitly?

[wereii]

Even with unset MAKEFLAGS and/or make -j1 the package still generates the wrong lib/flagtabs.h