BBRF Listener Timeout Error

[jakre1234]

Hello devs, my bbrf listener keeps giving this timeout error after few minutes

bbrf listner &

[ERROR] HTTPSConnectionPool(host='72.XX.XX.XX', port=443): Read timed out. (read timeout=90)

I have tried running the bbrf listener in bot background and a separate tmux session as well.

[jakre1234]

This timeout error keep happening every now and then. I am running bbrf server and client on a linode VPS. While looking at the listen output I find some other errors too [Error] Can not import BBRFClient. Sorry for the issues spam

[jakre1234]

Also I noticed some inconsistency on the bbrf client. I test bbrf on two different VPS, as we know the correct syntax for bbrf to view resolved/unresolved domains is

bbrf domains --view resolved/unresolved

but on one of the VPS the following command was also working the same and producing same output as above

bbrf domains resolved

while on the other VPS, this command does not produce results and gave a help menu.

[honoki]

Regarding your last point, can you verify you have the last version of BBRF installed on all your servers:

bbrf -v

If necessary, update using pip install --upgrade bbrf

[jakre1234]

Hey @honoki , Thanks for the quick response. I just did a fresh install on the first VPS and now the commands seems to work fine and no inconsistency is being observed but I still face those TimeOut Error.

The timeout error especially occurs when I add a new program with command bbrf new prog , I was wondering if there's any flag I can use to specify timeouts? P.s I have not set up and domain or certs and have changed the ignore ssl errors to true in config file as well.

Thanks,

[honoki]

Do you get any timeout errors when talking to your server directly via curl?

curl  "$(jq -r '.couchdb' ~/.bbrf/config.json)/_changes?feed=continuous;include_docs=true;since=now" -u bbrf:$(jq -r .password ~/.bbrf/config.json)

[jakre1234]

Hey @honoki , Thanks, As you asked I create a connection to the server via curl in one tmux session and normal bbrf listen in another. After starting a new program and adding inscope items, I checkd on both tmux sessions and both seems to have dropped connection. I am attaching the output bellow

Tmux session1:

curl  "$(jq -r '.couchdb' ~/.bbrf/config.json)/_changes?feed=continuous;include_docs=true;since=now" -u bbrf:$(jq -r .password ~/.bbrf/config.json) -k

Tmux session 2:

bbrf listen

[honoki]

This is likely an issue with the BBRF server configuration. I will move this issue there.

Did you deploy the server according to the documentation of this repo?

Do you seen any relevant errors when you run docker logs --tail 100 bbrf-server_proxy_1 on the couchdb server?

[honoki]

I ran some local tests and am also seeing the timeout via curl sometimes, but not consistently. It looks like you found a configuration issue.

I'm looking into https://docs.couchdb.org/en/stable/api/database/changes.html#db-changes and it seems like maybe these changes will be needed:

• Explicitly add a timeout parameter to the _changes URL, default is 1 minute (60000);
• Add a proxy_read_timeout of 60s to the BBRF server Nginx config, default should be 1 minute;

Need to investigate this more to pinpoint the issue.

[honoki]

Additionally, it looks like disabling the proxy_buffering directive will be necessary to improve listening to changes.

I'm not sure how that will play with gzip compression.

[jakre1234]

Hey @honoki , I am sorry for missing our your messages, I am not sure what needs to be done on my end to resolve this or perhaps you will be pushing a universal fix anytime soon? I have zero experience with couchDB but I will try to go through the attached documentation once.

Thanks and Regards,

[jakre1234]

This is likely an issue with the BBRF server configuration. I will move this issue there.

Did you deploy the server according to the documentation of this repo?

Do you seen any relevant errors when you run docker logs --tail 100 bbrf-server_proxy_1 on the couchdb server?

Yes, On running the command I can see several connection timeout errors in between. I have attached the output below

root@linode:$ docker logs --tail 100 bbrf-server-proxy-1|egrep 'timeout'
2023/03/21 01:17:15 [error] 13#13: *21359 upstream timed out (110: Connection timed out) while reading upstream, client: X.X.X.X, server: , request: "GET /bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT HTTP/1.1", upstream: "http://X.X.X.X:5984/bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT", host: "X.X.X.X"
2023/03/21 01:23:15 [error] 13#13: *21375 upstream timed out (110: Connection timed out) while reading upstream, client: X.X.X.X, server: , request: "GET /bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT HTTP/1.1", upstream: "http://X.X.X.X:5984/bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT", host: "X.X.X.X"
2023/03/21 01:26:15 [error] 13#13: *21382 upstream timed out (110: Connection timed out) while reading upstream, client: X.X.X.X, server: , request: "GET /bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT HTTP/1.1", upstream: "http://X.X.X.X:5984/bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT", host: "X.X.X.X"
2023/03/21 01:27:15 [error] 13#13: *21386 upstream timed out (110: Connection timed out) while reading upstream, client: X.X.X.X, server: , request: "GET /bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT HTTP/1.1", upstream: "http://X.X.X.X:5984/bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT", host: "X.X.X.X"
2023/03/21 01:31:15 [error] 13#13: *21388 upstream timed out (110: Connection timed out) while reading upstream, client: X.X.X.X, server: , request: "GET /bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT HTTP/1.1", upstream: "http://X.X.X.X:5984/bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT", host: "X.X.X.X"
2023/03/21 01:32:15 [error] 13#13: *21393 upstream timed out (110: Connection timed out) while reading upstream, client: X.X.X.X, server: , request: "GET /bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT HTTP/1.1", upstream: "http://X.X.X.X:5984/bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT", host: "X.X.X.X"
2023/03/21 01:37:15 [error] 13#13: *21395 upstream timed out (110: Connection timed out) while reading upstream, client: X.X.X.X, server: , request: "GET /bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT HTTP/1.1", upstream: "http://X.X.X.X:5984/bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT", host: "X.X.X.X"
2023/03/21 01:50:16 [error] 13#13: *21401 upstream timed out (110: Connection timed out) while reading upstream, client: X.X.X.X, server: , request: "GET /bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT HTTP/1.1", upstream: "http://X.X.X.X:5984/bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT", host: "X.X.X.X"
2023/03/21 01:51:16 [error] 13#13: *21415 upstream timed out (110: Connection timed out) while reading upstream, client: X.X.X.X, server: , request: "GET /bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT HTTP/1.1", upstream: "http://X.X.X.X:5984/bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT", host: "X.X.X.X"
2023/03/21 02:01:16 [error] 13#13: *21417 upstream timed out (110: Connection timed out) while reading upstream, client: X.X.X.X, server: , request: "GET /bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT HTTP/1.1", upstream: "http://X.X.X.X:5984/bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT", host: "X.X.X.X"
2023/03/21 02:02:16 [error] 13#13: *21428 upstream timed out (110: Connection timed out) while reading upstream, client: X.X.X.X, server: , request: "GET /bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT HTTP/1.1", upstream: "http://X.X.X.X:5984/bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT", host: "X.X.X.X"
2023/03/21 02:16:16 [error] 13#13: *21430 upstream timed out (110: Connection timed out) while reading upstream, client: X.X.X.X, server: , request: "GET /bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT HTTP/1.1", upstream: "http://X.X.X.X:5984/bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT", host: "X.X.X.X"
2023/03/21 02:20:16 [error] 13#13: *21448 upstream timed out (110: Connection timed out) while reading upstream, client: X.X.X.X, server: , request: "GET /bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT HTTP/1.1", upstream: "http://X.X.X.X:5984/bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT", host: "X.X.X.X"
2023/03/21 02:24:16 [error] 13#13: *21453 upstream timed out (110: Connection timed out) while reading upstream, client: X.X.X.X, server: , request: "GET /bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT HTTP/1.1", upstream: "http://X.X.X.X:5984/bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT", host: "X.X.X.X"
2023/03/21 02:27:16 [error] 13#13: *21458 upstream timed out (110: Connection timed out) while reading upstream, client: X.X.X.X, server: , request: "GET /bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT HTTP/1.1", upstream: "http://X.X.X.X:5984/bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT", host: "X.X.X.X"
2023/03/21 02:28:16 [error] 13#13: *21462 upstream timed out (110: Connection timed out) while reading upstream, client: X.X.X.X, server: , request: "GET /bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT HTTP/1.1", upstream: "http://X.X.X.X:5984/bbrf/_changes?feed=continuous;include_docs=true;since=428-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCScyJNX___8_K4M58XUuUIDdwsI40dLcEF0xDu15LECSoQFI_YeachBiSrJFkkWaKbqeLACpICwT", host: "X.X.X.X"

[jakre1234]

Hey @honoki , Do we have any updates on this ??