Unique transaction ID is not unique

[git-madresc]

The Unique transaction ID as described in ModSecurity-2-Data-Formats is not unique.

My goal is to send the SecAuditLog data chunks (audit log parts A-F and Z) to a database for further analysis. This works so far but I run into a problem here. I can not reassemble every sigle request, because Unique transaction IDs are not unique.

To demonstrate this, I have set 'SecAuditLogType Concurrent' and 'SecAuditLogStorageDir /opt/modsecurity/var/audit/'. Now I run a bunch of test URLs, which will trigger the SecAuditEngine. As a Result I get this:

wfs-proxy:/opt/modsecurity/var/audit/wwwrun # grep -re '^--' ./
./20230503/20230503-1333/20230503-133313-ZFJGec8R3U_ldaMTbxzC3wAAAAM:--e3bdce7c-A--
./20230503/20230503-1333/20230503-133313-ZFJGec8R3U_ldaMTbxzC3wAAAAM:--e3bdce7c-B--
./20230503/20230503-1333/20230503-133313-ZFJGec8R3U_ldaMTbxzC3wAAAAM:--e3bdce7c-F--
./20230503/20230503-1333/20230503-133313-ZFJGec8R3U_ldaMTbxzC3wAAAAM:--e3bdce7c-E--
./20230503/20230503-1333/20230503-133313-ZFJGec8R3U_ldaMTbxzC3wAAAAM:--e3bdce7c-Z--
./20230503/20230503-1333/20230503-133313-ZFJGeQ-tIwVAaX7zG6cCrAAAABc:--7253ec42-A--
./20230503/20230503-1333/20230503-133313-ZFJGeQ-tIwVAaX7zG6cCrAAAABc:--7253ec42-B--
./20230503/20230503-1333/20230503-133313-ZFJGeQ-tIwVAaX7zG6cCrAAAABc:--7253ec42-F--
./20230503/20230503-1333/20230503-133313-ZFJGeQ-tIwVAaX7zG6cCrAAAABc:--7253ec42-E--
./20230503/20230503-1333/20230503-133313-ZFJGeQ-tIwVAaX7zG6cCrAAAABc:--7253ec42-Z--
./20230503/20230503-1333/20230503-133313-ZFJGeWUtrU7v5kvWR0JxxwAAABE:--7253ec42-A--
./20230503/20230503-1333/20230503-133313-ZFJGeWUtrU7v5kvWR0JxxwAAABE:--7253ec42-B--
./20230503/20230503-1333/20230503-133313-ZFJGeWUtrU7v5kvWR0JxxwAAABE:--7253ec42-F--
./20230503/20230503-1333/20230503-133313-ZFJGeWUtrU7v5kvWR0JxxwAAABE:--7253ec42-E--
./20230503/20230503-1333/20230503-133313-ZFJGeWUtrU7v5kvWR0JxxwAAABE:--7253ec42-Z--
./20230503/20230503-1333/20230503-133313-ZFJGeTmD8lkS7lgL73cABAAAAAE:--e3bdce7c-A--
./20230503/20230503-1333/20230503-133313-ZFJGeTmD8lkS7lgL73cABAAAAAE:--e3bdce7c-B--
./20230503/20230503-1333/20230503-133313-ZFJGeTmD8lkS7lgL73cABAAAAAE:--e3bdce7c-F--
./20230503/20230503-1333/20230503-133313-ZFJGeTmD8lkS7lgL73cABAAAAAE:--e3bdce7c-E--
./20230503/20230503-1333/20230503-133313-ZFJGeTmD8lkS7lgL73cABAAAAAE:--e3bdce7c-Z--
./20230503/20230503-1333/20230503-133313-ZFJGeVOLgwIBaW4q2ekUFAAAABU:--7253ec42-A--
./20230503/20230503-1333/20230503-133313-ZFJGeVOLgwIBaW4q2ekUFAAAABU:--7253ec42-B--
./20230503/20230503-1333/20230503-133313-ZFJGeVOLgwIBaW4q2ekUFAAAABU:--7253ec42-F--
./20230503/20230503-1333/20230503-133313-ZFJGeVOLgwIBaW4q2ekUFAAAABU:--7253ec42-E--
./20230503/20230503-1333/20230503-133313-ZFJGeVOLgwIBaW4q2ekUFAAAABU:--7253ec42-Z--
./20230503/20230503-1333/20230503-133313-ZFJGeSWi0RqhZpm4t4h8LgAAABg:--23b0d761-A--
./20230503/20230503-1333/20230503-133313-ZFJGeSWi0RqhZpm4t4h8LgAAABg:--23b0d761-B--
./20230503/20230503-1333/20230503-133313-ZFJGeSWi0RqhZpm4t4h8LgAAABg:--23b0d761-F--
./20230503/20230503-1333/20230503-133313-ZFJGeSWi0RqhZpm4t4h8LgAAABg:--23b0d761-E--
./20230503/20230503-1333/20230503-133313-ZFJGeSWi0RqhZpm4t4h8LgAAABg:--23b0d761-Z--
./20230503/20230503-1333/20230503-133313-ZFJGefyFgPo-zywsXm9qygAAAAw:--c96b7719-A--
./20230503/20230503-1333/20230503-133313-ZFJGefyFgPo-zywsXm9qygAAAAw:--c96b7719-B--
./20230503/20230503-1333/20230503-133313-ZFJGefyFgPo-zywsXm9qygAAAAw:--c96b7719-F--
./20230503/20230503-1333/20230503-133313-ZFJGefyFgPo-zywsXm9qygAAAAw:--c96b7719-E--
./20230503/20230503-1333/20230503-133313-ZFJGefyFgPo-zywsXm9qygAAAAw:--c96b7719-Z--
./20230503/20230503-1333/20230503-133313-ZFJGeWkv4RhdUr82XzP3MQAAAAc:--e3bdce7c-A--
./20230503/20230503-1333/20230503-133313-ZFJGeWkv4RhdUr82XzP3MQAAAAc:--e3bdce7c-B--
./20230503/20230503-1333/20230503-133313-ZFJGeWkv4RhdUr82XzP3MQAAAAc:--e3bdce7c-F--
./20230503/20230503-1333/20230503-133313-ZFJGeWkv4RhdUr82XzP3MQAAAAc:--e3bdce7c-E--
./20230503/20230503-1333/20230503-133313-ZFJGeWkv4RhdUr82XzP3MQAAAAc:--e3bdce7c-Z--
./20230503/20230503-1333/20230503-133313-ZFJGeSAkZTeubWGVgOyg-wAAAA8:--a09a2874-A--
./20230503/20230503-1333/20230503-133313-ZFJGeSAkZTeubWGVgOyg-wAAAA8:--a09a2874-B--
./20230503/20230503-1333/20230503-133313-ZFJGeSAkZTeubWGVgOyg-wAAAA8:--a09a2874-F--
./20230503/20230503-1333/20230503-133313-ZFJGeSAkZTeubWGVgOyg-wAAAA8:--a09a2874-E--
./20230503/20230503-1333/20230503-133313-ZFJGeSAkZTeubWGVgOyg-wAAAA8:--a09a2874-Z--
./20230503/20230503-1333/20230503-133313-ZFJGefHBvyBvLWBOcUuWaAAAAAs:--112c5e06-A--
./20230503/20230503-1333/20230503-133313-ZFJGefHBvyBvLWBOcUuWaAAAAAs:--112c5e06-B--
./20230503/20230503-1333/20230503-133313-ZFJGefHBvyBvLWBOcUuWaAAAAAs:--112c5e06-F--
./20230503/20230503-1333/20230503-133313-ZFJGefHBvyBvLWBOcUuWaAAAAAs:--112c5e06-E--
./20230503/20230503-1333/20230503-133313-ZFJGefHBvyBvLWBOcUuWaAAAAAs:--112c5e06-Z--

As you can see, 'e3bdce7c' and '7253ec42' are the UIDs for three different requests each. May be I did something wrong or misunderstood something, but that is, what I see.

This is a mod_security2 module for a Apache HTTP Server 2.4 on a Suse system, rpm version is apache2-mod_security2-2.9.4-150400.3.6.1.x86_64.

Oh, one more thing: I suspect, this UID may may somehow stick to the httpd worker child. This was tested with MPM models 'prefork' and 'event'.

[martinhsv]

Hello @git-madresc ,

The lines that delimit the different parts ('A', 'B', etc.) of the native-format audit log do indeed contain an 8-character string of hexadecimal digits. However, those 8-character strings are not the unique transaction id.

In the native format audit log the unique transaction id is the longer string of alphanumeric characters that appears within the body of 'part A'. If you are using the JSON log format, it is the item with the name 'transaction_id'.

(The 8-character string in the boundary is also intended to be quasi-random (and hence quasi-unique) but it is dependent on how rand() is seeded ... and the current functionality uses time (to the second) and pid. I.e. I would expect audit logs produced in the same second by the same process will indeed have duplicate values.)

[git-madresc]

Granted, I mixed up the 'Unique transaction ID' (generated by mod_unique_id, I guess?) and the 'Unique boundary' from the sections separators mentioned in https://github.com/SpiderLabs/ModSecurity/wiki/ModSecurity-2-Data-Formats#audit-log. But I understand, my suspicion was right, that this ID sticks in some way on the httpd's worker thread, which helps me a lot. May I suggest you add the detail from above to the documentation?

Thanks a lot, Martin