SQL Error 1002: ClickHouse exception; Connection pool shutdown

[avinovarov]

Hi, not entirely certain if the question fits, but here is our setup - a single CHProxy VM in front of our ClickHouse clusters, clients are IDE users running SQL queries among others.

Sometimes our users encounter SQL Error [1002]: ClickHouse exception, host: <chproxyURL>, port 9090; Connection pool shut down errors. They say that 'firing up a new SQL editor' helps from time to time though.

Google didn't really answer my question, found no 'max concurrent connections' options neither in CHProxy, nor in ClickHouse settings.

I tried to monitor TCP connections on both CHProxy machine and ClickHouse nodes but neither of them seems to hit any meaningful spike on Grafana, nor I don't see an unusually long list of established connections in netstat outputs.

Does the error has something to do with CHProxy? Or should we dig deeper into ClickHouse itself? Some advice on what and how to monitor is greatly appreciated too =)

Thank you!

[avinovarov]

The issue appeared to be in chproxy config, which in our case had concurrent queries limited for certain user because god knows why, like this:

- name: "username"
  max_concurrent_queries: 10

After increasing the number to 20 and restarting chproxy service the issue's gone.

Still we're suggesting a bit more descriptive error message, because 'connection pool shutdown' hardly has something to do with concurrent queries.

[gontarzpawel]

Hello @avinovarov, thank you for your update. In order to localise the issue, could you provide how to reproduce it?

[avinovarov]

Hi @Garnek20, to be honest it's a bit tough to recall the details almost a year later =)

But essentially I assume that we had a single chproxy user for multiple people using it simultaneously, and historically that user had max_concurrent_queries: 10 setting in chproxy config. So I'd try to reproduce it exactly like this - create one chproxy user, limit concurrent connections, run over 10 concurrent queries.

Overall that doesn't really look like a big issue, just the error we got sounds a bit misleading.

Thanks!