Random wrong password on Clickhouse pod

[vasicvuk]

Bug description

I get the following error on Clickhouse pod:

2023.01.27 07:34:25.035880 [ 685 ] {} <Error> DynamicQueryHandler: Code: 516. DB::Exception: clickhouse_operator: Authentication failed: password is incorrect or there is no user with such name. (AUTHENTICATION_FAILED), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xa3ef75a in /usr/bin/clickhouse
1. DB::AccessControl::authenticate(DB::Credentials const&, Poco::Net::IPAddress const&) const @ 0x13cfb3b6 in /usr/bin/clickhouse
2. DB::Session::authenticate(DB::Credentials const&, Poco::Net::SocketAddress const&) @ 0x14f79c95 in /usr/bin/clickhouse
3. DB::HTTPHandler::authenticateUser(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&) @ 0x15bf1f1c in /usr/bin/clickhouse
4. DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::__1::optional<DB::CurrentThread::QueryScope>&) @ 0x15bf3899 in /usr/bin/clickhouse
5. DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&) @ 0x15bfb9f4 in /usr/bin/clickhouse
6. DB::HTTPServerConnection::run() @ 0x15c774a2 in /usr/bin/clickhouse
7. Poco::Net::TCPServerConnection::start() @ 0x189db793 in /usr/bin/clickhouse
8. Poco::Net::TCPServerDispatcher::run() @ 0x189dcaed in /usr/bin/clickhouse
9. Poco::PooledThread::run() @ 0x18b9f449 in /usr/bin/clickhouse
10. Poco::ThreadImpl::runnableEntry(void*) @ 0x18b9cd02 in /usr/bin/clickhouse
11. ? @ 0x7f2868610609 in ?
12. __clone @ 0x7f2868535133 in ?
 (version 22.8.8.3 (official build))
2023.01.27 07:34:25.043175 [ 685 ] {} <Error> Access(user directories): from: ::ffff:10.42.19.125, user: clickhouse_operator: Authentication failed: Code: 195. DB::Exception: Connections from ::ffff:10.42.19.125 are not allowed. (IP_ADDRESS_NOT_ALLOWED), Stack trace (when copying this message, always include the lines below):

The error appears every 5 seconds a few times.

Seems like the operator has no privilege to access the click house database.

Expected behavior

No errors

How to reproduce

1. Install signoz using helm chart
2. Use it for a while
3. Error appears

Version information

• Signoz version: 0.14.0
• Browser version: Chrome latest
• Your OS and version: Running on Kubernetes
• Your CPU Architecture(ARM/Intel): Intel

/etc/clickhouse-server/users.d/chop-generated-users.xml contents

<yandex>
    <users>
        <admin>
            <networks>
                <host_regexp>(chi-signoz-test-clickhouse-[^.]+\d+-\d+|clickhouse\-signoz-test-clickhouse)\.signoz\.svc\.cluster\.local$</host_regexp>
                <ip>::1</ip>
                <ip>127.0.0.1</ip>
                <ip>10.0.0.0/8</ip>
                <ip>100.64.0.0/10</ip>
                <ip>172.16.0.0/12</ip>
                <ip>192.0.0.0/24</ip>
                <ip>198.18.0.0/15</ip>
                <ip>192.168.0.0/16</ip>
            </networks>
            <password_sha256_hex>xxxxx</password_sha256_hex>
            <profile>default</profile>
            <quota>default</quota>
        </admin>
        <clickhouse_operator>
            <networks>
                <ip>10.42.19.135</ip>
            </networks>
        </clickhouse_operator>
        <default>
            <networks>
                <host_regexp>(chi-signoz-test-clickhouse-[^.]+\d+-\d+|clickhouse\-signoz-test-clickhouse)\.signoz\.svc\.cluster\.local$</host_regexp>
                <ip>::1</ip>
                <ip>127.0.0.1</ip>
                <ip>10.42.19.141</ip>
            </networks>
            <profile>default</profile>
            <quota>default</quota>
        </default>
    </users>
</yandex>

[vasicvuk]

I think I found what the issue is. I have two installations of SigNoz both having an operator installed. It seems that one operator will try to access the instance of another click house operator. Can we limit somehow the operator to a namespace using the helm chart?

[srikanthccv]

Your own resource definition may help, but I will let @prashant-shahi chime in here.