libz.so.1 and libpython3.8.so.1.0 created in /tmp/ at runtime

[nnmmaaoo]

Sorry to bother you with something thats not really a bug. I noticed, that files libz.so.1 and libpython3.8.so.1.0 are being created in /tmp/ and used at runtime after zui launch. They are not kept, so do not pose a problem. Could you, please, tell me what elements of zui are responsible for those libraries? And is there a way to avoid them, maybe by substituting some of the brimcap components? Or maybe its prossible to plug them using LD_PRELOAD?

[philrz]

Hello @nnmmaaoo. Thanks for your interest in Zui!

Yes, I think I can explain what you're seeing. First I'll post a video of reproducing the problem myself just to make sure we're talking about the same thing. Here I've got an Ubuntu 20.04 VM running and in one window I'm running watch -d -n 1 ls -lR _ME* from /tmp and in the other I launch GA Zui v1.5.0. It does indeed show the two libraries you speak of (and others) appearing and then disappearing from within a temp directory with a name starting with the characters _ME.

https://github.com/brimdata/zui/assets/5934157/8f1dcccb-92ea-4da4-83fd-d14d4907094c

Here's what's going on. One of Zui's use cases is for search/analysis of network security logs, and I sense you may be familiar with this since you mentioned Brimcap, which is bundled with Zui for creating summary logs from packet captures. One tool bundled with Brimcap is Suricata, and a key piece of Suricata is suricata-update which downloads the latest rule sets so you can catch many current threats when analyzing pcaps, and Zui typically runs suricata-update once whenever the app is launched. In its original form, suricata-update is typically distributed as a Python script, and this is reasonable given that Suricata is most often used in server environments where it's reasonable to require an installed Python distribution. However, as Zui is desktop software, we did not want to require all users to have Python installed nor did we want the bulk of including a full Python distribution with Zui. Therefore we use PyInstaller to turn suricata-update into an executable that includes its dependencies such as these libraries. My PyInstaller knowledge is not encyclopedic, but my understanding that it unpacking its dependencies in a temporary location is all part of how it typically works.

With that established, if you feel strongly about avoiding them, I could make some suggestions based on your use case. Some questions:

1. Could you explain how they're currently causing a problem for you?
2. Is network security a large part of what you use Zui for?
3. If so, is having the updated Suricata rules important to you?

There's a few directions we could go based on the answers to these questions.

[nnmmaaoo]

Hey, thank you so much for your explanation. About our usecase: we plan on running zui in an air-gapped environment with strict SELinux enforcement policies, that require all of the ELF files running on the system to be "signed". It is usually done using or after building of a .deb package. But that doesnt cover files, that are being created and executed at runtime. Thats why i was asking about LD_PRELOAD - we can simply provide all of the needed libraries already signed inside the package or as a separate one.

Zui is being used for network security purposes. Mainly for parsing provided pcap files.

For us suricata rule updates are not important, since we are planning on providing our own suricata rules. So if its possible to cleanly disable the suricata-update module - that would be ideal. If that is the only module, that is packaged this way.

[philrz]

Thanks for the additional detail.

Based on what you described, I can see a couple ways to get the PyInstaller'd suricata-update out of your way. There's unfortunately not currently a user-friendly checkbox-style config to disable it (though it's something I'll discuss with our dev team, as based on your experience I can see how this would be desirable), but if you just delete suricata-update from the Zui's Application Binaries directory on Linux (/opt/Zui/resources/app.asar.unpacked/zdeps/suricata/bin) Zui will quietly attempt to run it when it always does but it will fail silently. This means that the Suricata rules that the app shipped with will get used each time you load a pcap (e.g., the ones in /opt/Zui/resources/app.asar.unpacked/zdeps/suricata/var/lib/suricata/rules). Since you said you were planning on providing your own Suricata rules, perhaps you were already planning to replace those manually.

In any case, I'd normally not see this as a super clean solution because if a user updates their app to a newer version then that'll end up installing the PyInstaller'd suricata-update again and be right back in the same spot. However, it sounds like you're tightly controlling the software installations in your environment so perhaps this would be workable.

Since you mentioned providing your own Suricata rules, I wanted to make sure you're aware of the Custom Brimcap Config article that describes how to use wholly separate Suricata and/or Zeek installations for pcap processing in place of the ones that ship with Zui. If someone says to me they want to use their own Suricata rules, that's the supported technique I'd point them at, since it gives them total control not only of what rules are used and when they're updated but it also gives them the freedom to use a newer/customized Suricata if they choose, which I expect would be desirable for some since we've not been able to update the Suricata version that ships with Brimcap/Zui in some time. Something to consider.

I'll hold this issue open in the event you have additional questions, and please do let us know what you come up with for a final config. If you'd like more interactive assistance, you may want to join the Brim community Slack. You can speak up in one of the public channels or message me directly @Phil.

[nnmmaaoo]

Hey, thank you for this suggestion. We will start with removing the suricata-update from the package (by repacking deb after its built maybe?) and checking if thats gonna be enough. Ill try to keep you updated.

[nnmmaaoo]

I ended up with this solution: rebuilding zeek and suricata using the instructions in brimdata/build-${package} repos, but on the target system. Stubbing suricata-update script in zui to avoid autoupdates, replacing zdeps contents with rebuilt zeek and suricata, prior to invoking yarn package-zui. Thank you very much for the instructions and clarifications!