AFLplusplus
Timeout for LLMP Clients
LLMP, short for LowLevel Message Passing, has the concept of multiple clients, which are all connected to one broker.
The broker broadcasts each new message it receives from a client (over an individual shared) to all other clients via one ...shared... shared map.
So far, LLMP Clients never get unregistered.
It would be good to keep track of the last time a client sent a message, and unregister them accordingly.
This could be done in the once method of the broker (which is called in a loop in the broker)
https://github.com/AFLplusplus/LibAFL/blob/6e4c6a9b6b0db909853a19a68ba769bb2516f1bf/libafl/src/bolts/llmp.rs#L1349
On top, we may want to support a good-bye/unregister message from clients when they know they are done. This could be called inside a Drop crate automatically.
So far, LLMP Clients never get unregistered. It would be good to keep track of the last time a client sent a message, and unregister them accordingly.
what do you mean by unregister and what point should they be removed? is it like removing client from llmp_client vec? will last_message_offset work to keep track or should I use last_message_sent.
Hey @aga7hokakological , yes they should be removed from the broker's vec after the broker did not receive a new message from them for n seconds. We'll need to keep the last recv time for each client.
handle_new_msgs could return the amount of new messages handled, for example.
I am bit confused here. say if I am adding some time function in LlmpMsg struct to get that time instance and then in once method I am calculating if broker received message or not and then removing it from vec. That's how it should be right? or maybe changing register_client function to get time of last_msg_recvd variable and then comparing it in once function that it haven't received message might work. just want to know what you think.
The timing should be done locally in the broker IMHO, it should just keep track of when it received the last new message from this client. Less likely to break things, too
For example, you can have a last_msg_time_for_client: Vec<Duration> vec, and after this line:
https://github.com/AFLplusplus/LibAFL/blob/6e4c6a9b6b0db909853a19a68ba769bb2516f1bf/libafl/src/bolts/llmp.rs#L1356
check if you got new messages, then update the current client's time, else check if it timed out.
Well the problem here is that I am only able to see 2 clients on my machines. Also whenever I am receiving new test cases they are only from client 0. In broker only client is there so the message is passed to broker and then to the other clients. Also I guess I need to use std::time because only core::time doesn't work as I need to have instace there to calculate the elapsed time. But I am not getting how to check for new message as it is a struct and not any other data type such as bool.
You get as many clients as you spawn; if you run more than two clients, you should see more than two clients (?) else we have a bug somewhere... Yes, the time can only be std
How should I spawn more than 2 clients?
In any real-world scenario, you should use taskset to pin each client to an empty CPU core, the lib does not pick an empty core automatically (yet). as this line states that it should be done manually.
But here,
RUST_BACKTRACE=full taskset -c 1 ./.libfuzzer_test.elf 2>/dev/null taskset -c 1is binding one CPU.
and I see no other option to bind cpu. If there is another way plz tell.
But I am not getting how to check for new message as it is a struct and not any other data type such as bool.
Also this is quite confusing as it cannot be used directly and I think I'll have to write whole code again in the functions to check new message.
or is using message_id fine of the llmp_msg cause then I can keep eye on message with the sender so that way it can be done.
-c 1 is binding to CPU 1. Just use -c 2, -c 3, ... for other CPUs. The test script is just there for some quick tests, don't be afraid to run the binary manually.
So, build the fuzzer and then run it from the target directory multiple times.
WRT new messages, inside handle_new_msgs, you know which client you are dealing with, and in this line:
https://github.com/AFLplusplus/LibAFL/blob/6e4c6a9b6b0db909853a19a68ba769bb2516f1bf/libafl/src/bolts/llmp.rs#L1588
you know that a new message arrived, just add the current time to some datastructure if this is the case. Or prune the client, if None was returned.
Hope this helps :)
-c 1 is binding to CPU 1. Just use -c 2, -c 3, ... for other CPUs. The test script is just there for some quick tests, don't be afraid to run the binary manually.
I tried doing this and building but the clients number remain the same.
@aga7hokakological gave you access, please push to a branch and open a pr to dev :)
Hey @domenukk, sorry for disturbing again but target/release/example does not work for me. I have tried on arch-linux as well as on ubuntu. I was able to run once but then it is not working. I tried the way you told me to run it but It just freezes at this.
./libfuzzer_libpng Workdir: "/home/myname/Desktop/LibAFL/target/release/examples" [libafl/src/bolts/llmp.rs:409] "We're the broker" = "We're the broker" Doing broker things. Run this tool again to start fuzzing in a client.
Though I have written code but without testing I cannot be able to make a pull request. And with ./test.sh from fuzzer/libfuzzer_libpng it works but because it just spawns 2 clients I am not able to test it. I am trying to resolve it from my side if I am missing anything plz tell.
Have you tried running it a second time, or multiple times? The first instance of the tool is the broker, the next ones then do the actual fuzzing.
Yes, I have tried running it multiple times. Almost 10-15 times I tried running but it just freezes.
It was giving problem because of this panic.
panic!("Fuzzer-respawner: Storing state in crashed fuzzer instance did not work, no point to spawn the next client!");
I solved it. Now working fine and clients are spawning perfectly.
WRT new messages, inside
handle_new_msgs, you know which client you are dealing with, and in this line:https://github.com/AFLplusplus/LibAFL/blob/6e4c6a9b6b0db909853a19a68ba769bb2516f1bf/libafl/src/bolts/llmp.rs#L1588
you know that a new message arrived, just add the current time to some datastructure if this is the case. Or prune the client, if None was returned.
Here the function handle_new_msgs returns result. So I tried this code but because of the timeout is not perfectly working so it is unable to remove clients.
let mut last_time_msg_sent_for_client = HashMap::new(); compiler_fence(Ordering::SeqCst); for i in 0..self.llmp_clients.len() { let time = Instant::now(); if let Ok(_) = self.handle_new_msgs(i as u32, on_new_msg) { if time.elapsed().as_millis() < duration::new(0, 1).as_millis() { last_time_msg_sent_for_client.insert( self.llmp_out.id, SystemTime::now(), ); } } else { //remove the client if message not sent self.llmp_clients.retain(|client| client.id != i as u32); } }
maybe this line is not necessary
if time.elapsed().as_millis() < duration::new(0, 1).as_millis()
It was giving problem because of this panic.
panic!("Fuzzer-respawner: Storing state in crashed fuzzer instance did not work, no point to spawn the next client!");
I solved it. Now working fine and clients are spawning perfectly.
How did you solve this?
Hey @s1341
How did you solve this?
This was the full error:
[libafl/src/bolts/llmp.rs:416] "We're the client" = "We're the client" [libafl/src/bolts/llmp.rs:416] e = Os { code: 98, kind: AddrInUse, message: "Address already in use", } Connected to port 1337 [libafl/src/events/llmp.rs:553] "Spawning next client (id {})" = "Spawning next client (id {})" [libafl/src/events/llmp.rs:553] ctr = 0 We're a client, let's fuzz :) First run. Let's set it all up We're a client, let's fuzz :) thread 'main' panicked at 'Failed to load initial corpus at ["./corpus"]: File(Os { code: 2, kind: Not Found, message: "No such file or directory" })', fuzzers/libfuzzer_libpng/./src/fuzzer.rs:176:14 note: run with
RUST_BACKTRACE=1environment variable to display a backtrace thread 'main' panicked at 'Fuzzer-respawner: Storing state in crashed fuzzer instance did not work, no point to spawn the next client!', /home/aga7hokakological/Saurabh/LibAFL/libafl/src/events/llmp.rs:56 8:21 note: run withRUST_BACKTRACE=1environment variable to display a backtrace
It was because I was running it from wrong directory, also there was error where it wasn't able to find libpng.tar.xz so I downloaded it manually and put it in the folder.
WRT new messages, inside
handle_new_msgs, you know which client you are dealing with, and in this line: https://github.com/AFLplusplus/LibAFL/blob/6e4c6a9b6b0db909853a19a68ba769bb2516f1bf/libafl/src/bolts/llmp.rs#L1588you know that a new message arrived, just add the current time to some datastructure if this is the case. Or prune the client, if None was returned.
Here the function
handle_new_msgsreturns result. So I tried this code but because of the timeout is not perfectly working so it is unable to remove clients.let mut last_time_msg_sent_for_client = HashMap::new(); compiler_fence(Ordering::SeqCst); for i in 0..self.llmp_clients.len() { let time = Instant::now(); if let Ok(_) = self.handle_new_msgs(i as u32, on_new_msg) { if time.elapsed().as_millis() < duration::new(0, 1).as_millis() { last_time_msg_sent_for_client.insert( self.llmp_out.id, SystemTime::now(), ); } } else { //remove the client if message not sent self.llmp_clients.retain(|client| client.id != i as u32); } }
maybe this line is not necessary
if time.elapsed().as_millis() < duration::new(0, 1).as_millis()
a) there is no need for a hashmap here, all clients are stored in a vec with fixed ids, so a vec will work here, too.
b) you create a new map each time this function gets called. Instead, store the state in self so that it survives across calls.
a) there is no need for a hashmap here, all clients are stored in a vec with fixed ids, so a vec will work here, too. b) you create a new map each time this function gets called. Instead, store the state in
selfso that it survives across calls.
I did but even if handle_new_msg is result the client.recv() is Option as I need some(msg) => msg but I cannot access it outside the function. But to say it contains Ok(()) of the handle_new_msg it means it has been handled so it should be pruned. My code ends as soon as I spawn clients.
Sorry, I don't really understand what you try to do here? Is it roughly what I described in https://github.com/AFLplusplus/LibAFL/issues/36#issuecomment-808882592 ?
I have declared another should_unregister variable in broker as bool.
Also added this code with some changes (just checking if self.should_unregister is true or false) to above code to check if it is some or none in handle_new_msg below msg and it works but when it is none means message have arrived it means it should add time to the vec rather it removes the client. that's the only problem remaining. I mean it works but in opposite way.
if msg.as_ref().is_none() { self.should_unregister = true; } else { self.should_unregister = false; }
if I interchange the true/false then it removes when msg is none but that should not be the case right?
Hello.
I have made few changes to LlmpBroker struct to add Vec<Duration> so that the data is stored in self itself.
For testing the changes, i built the libfuzzer_libpng example as mentioned. But the following make command is throwing an error.
make CC="$(pwd)/../target/release/libafl_cc" CXX="$(pwd)/../target/release/libafl_cxx" -j ``nproc `
The error message is thread 'main' panicked at 'Failed to run the wrapped compiler: Io(Os { code: 2, kind: NotFound, message: "No such file or directory" })', src/bin/libafl_cc.rs:29:14
I found that "$(pwd)/../target/release/libafl_cc" is a binary but not a directory. The exact path of binary is - fuzzers/libfuzzer_libpng/target/release/libafl_cc.
As per the instructions, I think I am building it in the correct directory. - fuzzers/libfuzzer_libpng/libpng-1.6.37.
Can you help me regarding this?
Actually I used that command first. Then to check where the build breaks, I followed instructions from README.md
Yet I tried it again today. It is still showing the same error.