Improve Transformers -> Allow them to interact with parameters that are not in "kparams"

[cyohg]

Hello ! Fibratus helps me a lot for my current project. However, I've tried using transformers and it seems to only works for editing parameters that are inside the "kparams" parameter.

For example, my Json output is formated like :

{
"seq": 9714518,
"pid": 4,
"tid": 4128,
"cpu": 1,
"name": "CreateFile",
"category": "file",
"description": "Creates or opens a file or I/O device",
"host": "DESKTOP-J1PAUV7","timestamp": "2025-05-12T16:27:23.8979653+02:00",
"kparams":
{
"attributes": "NORMAL",
"create_disposition": "CREATE",
"create_options": "SYNCHRONOUS_IO_NONALERT",
"file_name": "C:\\Windows\\system32\\Logfiles\\WMI\\RtBackup\\EtwRTNT Kernel Logger.etl",
"file_object": "ffffe50c40d059e0",
"irp": "ffffe50c3dc7fc48",
"share_mask": "READ|DELETE",
"status": "Success",
"tid": 4128,
"type": "File"
},
"meta": {},
"ps":
{
"pid": 4,
"ppid": 0,
"name": "System",
"cmdline": "",
"exe": "System",
"cwd": "",
"sid": "S-1-5-18",
"args": [],
"sessionid": 4294967295
}
}

I managed to remove the "tid" event inside the "kparams" parameter, but I can't remove the "tid" event that is outside it. Is there a way around that ?

If there isn't, I would like that putting a code like that in the fibratus.yml file would work :

transformers:
  # Remove transformer deletes provided event parameters.
  remove:
    # Indicates if the remove transformer is enabled
    enabled: true
    field:
      - tid

Would that be a posibility ?

[rabbitstack]

Hi @cyohg ,

As the documentation clearly states, the remove transformer operates on event parameters. It can't mutate other event attributes. I'm curious about your use case. The top-level tid attribute represents the identifier of the thread generating the syscall/event, and as such, it is among the canonical event fields.

[cyohg]

Hi, thank you for your quick answer !

For my use case, I have huge captures which when converted into Json give me files of several gigabytes. Thus I would like to add only attributes that I need unlike the tid (and a few others) to reduce as much as possible the file size.

I could give it post treatment (that's actually what I'm doing right know), but it takes a lot of time.

[rabbitstack]

Oh, I understand now. As I already explained previously, currently it is not possible to mutate JSON fields other than event parameters. Implementing such a feature would require controlling the construction of the JSON payload. By the way, have you considered Fibratus captures? They are quite efficient in terms of storage because they leverage ZSTD compression.

More here> https://www.fibratus.io/#/captures/introduction

[cyohg]

I'm currently using the Fibratus captures but when I need to analyse them I have to convert them into Json. Otherwise it wouldn't be possible to access the nth line without browsing all of the beginning of the file because it's a flux (or I missed something ?).

And why is it impossible to add a feature which controls the construction of the Json payload ? Isn't it Fibratus that does that ?

[rabbitstack]

And why is it impossible to add a feature which controls the construction of the Json payload ?

I didn't mean it is impossible to deliver such a feature. If you're willing to give it a try, I can provide guidance. It boils down to controlling the manual JSON composition here. I reckon it might be a bit tricky to carry the transformers config into the event structure and make it accessible inside the MarshalJSON method.

[cyohg]

Wow. I dindn't dev in go since a long time and the Fibratus source code looks substancial but if you tell me that it is doable why not ! I won't be around for the next 7 days but if you can explain how the code works I will read you, and I will tell you if I think it looks manageable for me or not.

[rabbitstack]

Sorry for the delay. I've been deeply reflecting on the ROI that the following functionality could have, and I think the gains in terms of storage reduction are negligible. The most substantial JSON content lives in process, modules, environment variables, and a few other attributes. Aside from the process object, the rest of the aforementioned attributes are skipped by default. So, in my opinion, if you really want to experience significant gains, a better approach entails controlling the serialization of the process section. We could expose a config parameter to achieve this.

[cyohg]

For my use case, it's not only the tid that I would like to remove, but like 8 other attributes. I just made a few tests and it appears that it halves the size of the Json file, which helps me a lot for what I'm doing. Also during that time, I've studied your code and I want to give it a try !

[cyohg]

Hello again, I've tried to install the dev environment, but I keep getting this error while trying to build with the make.bat with the cap option :

github.com/valyala/gozstd
..\..\go\pkg\mod\github.com\valyala\[email protected]\stream.go:14:48: undefined: DefaultCompressionLevel
..\..\go\pkg\mod\github.com\valyala\[email protected]\stream.go:31:59: undefined: CDict
..\..\go\pkg\mod\github.com\valyala\[email protected]\stream.go:35:64: undefined: CDict
..\..\go\pkg\mod\github.com\valyala\[email protected]\stream.go:47:20: undefined: Writer
..\..\go\pkg\mod\github.com\valyala\[email protected]\stream.go:56:22: undefined: NewWriterLevel
..\..\go\pkg\mod\github.com\valyala\[email protected]\stream.go:101:61: undefined: DDict
..\..\go\pkg\mod\github.com\valyala\[email protected]\stream.go:110:6: undefined: Reader
..\..\go\pkg\mod\github.com\valyala\[email protected]\stream.go:117:8: undefined: NewReader
Failed with error #1.

I guess this has to do with the gozstd library, is that a known problem or did I missed something ?

(Without the cap option I can get the .msi and install fibratus but I can't get a capture because the option isn't available.)

[rabbitstack]

@cyohg Please refer to Building from source. The key is to set up the MinGW compiler toolchain. You can, of course, ignore libyara and python build options if not intended to employ.

[cyohg]

I've just tried that and follow the instructions in the contributing.md file, but it keeps telling me either the above error when I'm compiling with cap or : Error: fibratus was compiled without cap support. Please compile with the 'cap' build flag when I'm not adding the cap flag (or kcap I don't know, I'm trying both each time) I'm currently on Windows 10, maybe it could be the source of my problem ? If I understand correctly, the points 2 and 3 of the "Installing dependencies" section are optional for my installation ?

[rabbitstack]

It looks like the cap build flag is not set correctly. If you're running the build from the PowerShell-based terminal, you must use a different command to set the environment variables:

$env:TAGS="cap"

[cyohg]

When I'm doing that, I have the error above : https://github.com/rabbitstack/fibratus/issues/506#issuecomment-2958969442

[rabbitstack]

It seems like you haven't set up the C compiler correctly. Just in case, try to enable CGo.

$env:CGO_ENABLED="1"

[cyohg]

Oh, indeed I forgot that. I just installed gcc compiler (which I uninstalled a few days ago for some reason) with Mingw but I can't get past this error :

C:\Users\me\go\pkg\mod\golang.org\[email protected]\pkg\tool\windows_amd64\link.exe: running gcc failed: exit status 1
C:\msys64\ucrt64\bin\gcc.exe -m64 -s -mconsole -Wl,--tsaware -Wl,--nxcompat -Wl,--major-os-version=6 -Wl,--minor-os-version=1 -Wl,--major-subsystem-version=6 -Wl,--minor-subsystem-version=1 -Wl,--dynamicbase -Wl,--high-entropy-va -o $WORK\b001\exe\a.out.exe -Wl,--no-insert-timestamp C:\Users\me~1\AppData\Local\Temp\go-link-1237681482\go.o C:\Users\me~1\AppData\Local\Temp\go-link-1237681482\000000.o C:\Users\me~1\AppData\Local\Temp\go-link-1237681482\000001.o C:\Users\me~1\AppData\Local\Temp\go-link-1237681482\000002.o C:\Users\me~1\AppData\Local\Temp\go-link-1237681482\000003.o C:\Users\me~1\AppData\Local\Temp\go-link-1237681482\000004.o C:\Users\me~1\AppData\Local\Temp\go-link-1237681482\000005.o C:\Users\me~1\AppData\Local\Temp\go-link-1237681482\000006.o C:\Users\me~1\AppData\Local\Temp\go-link-1237681482\000007.o C:\Users\me~1\AppData\Local\Temp\go-link-1237681482\000008.o C:\Users\me~1\AppData\Local\Temp\go-link-1237681482\000009.o C:\Users\me~1\AppData\Local\Temp\go-link-1237681482\000010.o C:\Users\me~1\AppData\Local\Temp\go-link-1237681482\000011.o C:\Users\me~1\AppData\Local\Temp\go-link-1237681482\000012.o C:\Users\me~1\AppData\Local\Temp\go-link-1237681482\000013.o -O2 -g C:/Users/me/go/pkg/mod/github.com/valyala/[email protected]/libzstd_windows_amd64.a -O2 -g -Wl,-T,C:\Users\me~1\AppData\Local\Temp\go-link-1237681482\fix_debug_gdb_scripts.ld -Wl,--start-group -lmingwex -lmingw32 -Wl,--end-group -lkernel32
C:/msys64/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/14.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/Users/me/go/pkg/mod/github.com/valyala/[email protected]/libzstd_windows_amd64.a(zdict.o):zdict.c:(.text+0x872): undefined reference to `__imp___iob_func'
C:/msys64/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/14.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/Users/mego/pkg/mod/github.com/valyala/[email protected]/libzstd_windows_amd64.a(zdict.o):zdict.c:(.text+0x94d): undefined reference to `__imp___iob_func'
C:/msys64/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/14.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/Users/me/go/pkg/mod/github.com/valyala/[email protected]/libzstd_windows_amd64.a(zdict.o):zdict.c:(.text+0xed6): undefined reference to `__imp___iob_func'
C:/msys64/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/14.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/Users/me/go/pkg/mod/github.com/valyala/[email protected]/libzstd_windows_amd64.a(zdict.o):zdict.c:(.text+0xf8d): undefined reference to `__imp___iob_func'
C:/msys64/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/14.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/Users/me/go/pkg/mod/github.com/valyala/[email protected]/libzstd_windows_amd64.a(zdict.o):zdict.c:(.text+0xfa6): undefined reference to `__imp___iob_func'
C:/msys64/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/14.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/Users/me/go/pkg/mod/github.com/valyala/[email protected]/libzstd_windows_amd64.a(zdict.o):zdict.c:(.text+0x1231): more undefined references to `__imp___iob_func' follow
collect2.exe: error: ld returned 1 exit status

Failed with error #1.

I tried to look at the files mentionned in the error but it seems weird that the problem would come from them.

[rabbitstack]

Go can be sensitive about the C compiler. This may help:

• https://github.com/golang/go/issues/57455
• https://github.com/anjannath/crc/commit/6733174441456ce55f0894b2af0a12c569c227a6

[cyohg]

choco install mingw --version 11.2.0 --allow-downgrade was the solution for me ! (anjannath/crc/commit/6733174) I finally managed to build fibratus with cap tag, let's dev

[cyohg]

While exploring Fibratus code, I just discovered how powerful filaments were. I'm currently deving a filament and I think that the feature I was looking for is kinda already existing 🤔 (or really easy to dev with filaments)

[rabbitstack]

Indeed, filaments have a lot of potential. Sadly, they feel like a bit abandoned, but I'll invest some time to make them more useful