Queue Watcher

qwatcher is designed to help monitor TCP connections and diagnose buffer and connectivity issues on Linux machines related to input and output queues.

Table of Contents

• Queue Watcher
  • Table of Contents
  • Why
  • How
    • Usage
    • Available flags
  • What
  • Nimble Directory
  • Tested on
  • Sponsor me
  • Contact me
  • License

Why

The idea of writing this program came to me after reading a few blog posts discussing how they faced odd connectivity issues and how it ended up being related to the send and receive queues.

Had they had this tool, they would have been able to find the root cause much faster.

How

1. Download and extract the file from this URL:

   wget https://github.com/pouriyajamshidi/qwatcher/releases/latest/download/qwatcher.tar.gz
   tar xvf qwatcher.tar.gz
   

   Optionally, you can use Nimble for installation https://nimble.directory/pkg/qwatcher and skip to step 3.

   nimble install qwatcher
   

2. Make it executable and if you prefer, move it to your $PATH:

   chmod +x qwatcher
   sudo cp qwatcher /usr/local/bin
   

3. Run it. There are two modes to run qwatcher.

   1. Write mode. Provides two logging methods:
      1. Using --db_path to log to a database. Default path: /var/log/qwatcher.db
      2. Using --log_path to log to a text file. Default path: /var/log/qwatcher.log
   2. Monitor mode (default). Prints the output to the console

:warning: Make sure to not feed a higher number than your current buffer size to the program.

In order to get the current read and write buffer sizes, run the following commands:

cat /proc/sys/net/ipv4/tcp_rmem
cat /proc/sys/net/ipv4/tcp_wmem

Usage

Let's explore all modes:

1. check every 5 seconds and log connections that surpass 100 kilobytes in send or receive queues to a database located at /var/log/qwatcher.db:

   qwatcher --recv_q=100000 --send_q=100000 --refresh=5 --db_path=/var/log/qwatcher.db
   

   Then you can check the database. For instance, connections with receive queue bigger than zero:

   sqlite3 needs to be installed.

   # Open the database file:
   sqlite3 /var/log/qwatcher.db
   # Beautify sqlite output:
   .mode line
   .headers on
   .separator ROW "\n"
   # Run your query:
   sqlite> SELECT * FROM qwatcher WHERE receiveQ > 0;
   

2. Should you prefer to log the stats in a log file instead of a database as shown in step 1 use --log_path:

   qwatcher --recv_q=100000 --send_q=100000 --refresh=5 --log_path=/var/log/qwatcher.log
   

   Then you can use tail to check the file contents:

   tail -f /var/log/qwatcher.log
   

3. Default mode. If you want the output to be shown on the console and not log to disk, use the commands above without --log_path or --db_path options:

   qwatcher --recv_q=100000 --send_q=100000
   

   The default refresh interval is 10 seconds.

The sample output can be seen here

Additionally, you can use the accompanying systemd service to run qwatcher in the background and not worry about system restarts:

sudo cp qwatcher.service /etc/systemd/system/qwatcher.service
sudo systemctl enable qwatcher.service
sudo systemctl start qwatcher.service

Available flags

  -h, --help        : show help
  --recv_q,         : Minimum receive-Q to trigger alert in bytes (default: 10000)
  --send_q,         : Minimum send-Q to trigger alert in bytes (default: 10000)
  --refresh,        : Refresh interval in seconds (default: 10)
  --db_path,        : Path to SQLite database to log reports (default: /var/log/qwatcher.db)
  --log_path,       : Path to log file to log reports (default: /var/log/qwatcher.log)
  -v, --version,    : Show version

Please note that you cannot use --db_path and --log_path at the same time.

What

Below output depicts the provided information for a connection:

Apart from the send and receive queues, there are additional information that can be useful to diagnose connectivity issues such as congestion window, mss, retransmits and more.

Nimble Directory

This project is also hosted on Nimble.

Tested on

Ubuntu server 22.04.

Sponsor me

Contact me

License