clickhouse-operator icon indicating copy to clipboard operation
clickhouse-operator copied to clipboard
verified publisher icon Altinity

DB::Exception: defau lt: Authentication failed: password is incorrect or there is no user with such name. Stack trace:DB::Exception: defau lt: Authentication failed: password is incorrect or there is no user with such name. Stack trace:

Open czhfe opened this issue 4 years ago • 8 comments

A running clickhouse instance occasionally throws an exception with an incorrect username or password, logged as follows:

2021.09.26 10:58:38.329282 [ 121 ] {e90f2fe0-20c3-4a9a-a6a8-49bba12cc67d} <Error> TCPHandler: Code: 516, e.displayText() = DB::Exception: Received from chi-clickhouse-woxshare-0-1:9000. DB::Exception: defau lt: Authentication failed: password is incorrect or there is no user with such name. Stack trace:

  1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8b6cbba in /usr/bin/clickhouse

  2. DB::IAccessStorage::throwCannotAuthenticate(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&) @ 0xf0cb81f in /usr/bin/clickhouse

  3. DB::IAccessStorage::login(DB::Credentials const&, Poco::Net::IPAddress const&, DB::ExternalAuthenticators const&, bool) const @ 0xf0cb662 in /usr/bin/clickhouse

  4. DB::Context::setUser(DB::Credentials const&, Poco::Net::SocketAddress const&) @ 0xf5191b9 in /usr/bin/clickhouse

  5. DB::TCPHandler::receiveHello() @ 0x10434aac in /usr/bin/clickhouse

  6. DB::TCPHandler::runImpl() @ 0x1042de74 in /usr/bin/clickhouse

  7. DB::TCPHandler::run() @ 0x10441839 in /usr/bin/clickhouse

  8. Poco::Net::TCPServerConnection::start() @ 0x12a3fd4f in /usr/bin/clickhouse

  9. Poco::Net::TCPServerDispatcher::run() @ 0x12a417da in /usr/bin/clickhouse

  10. Poco::PooledThread::run() @ 0x12b7ab39 in /usr/bin/clickhouse

  11. Poco::ThreadImpl::runnableEntry(void*) @ 0x12b76b2a in /usr/bin/clickhouse

  12. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so

  13. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so : While executing Remote, Stack trace:

  14. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8b6cbba in /usr/bin/clickhouse

  15. DB::readException(DB::ReadBuffer&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, bool) @ 0x8bc8fd4 in /usr/bin/clickhouse

  16. DB::Connection::receiveException() const @ 0x10352e82 in /usr/bin/clickhouse

  17. DB::Connection::receiveHello() @ 0x10352a1f in /usr/bin/clickhouse

  18. DB::Connection::connect(DB::ConnectionTimeouts const&) @ 0x1035026c in /usr/bin/clickhouse

  19. DB::Connection::forceConnected(DB::ConnectionTimeouts const&) @ 0x10353b91 in /usr/bin/clickhouse

  20. DB::ConnectionEstablisher::run(PoolWithFailoverBaseDB::IConnectionPool::TryResult&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >&) @ 0x1036a1a9 in /usr/bin/clic khouse

  21. ? @ 0x10366277 in /usr/bin/clickhouse

  22. PoolWithFailoverBaseDB::IConnectionPool::getMany(unsigned long, unsigned long, unsigned long, unsigned long, bool, std::__1::function<PoolWithFailoverBaseDB::IConnectionPool::TryResult (DB::IConnecti onPool&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >&)> const&, std::__1::function<unsigned long (unsigned long)> const&) @ 0x10363b25 in /usr/bin/clickhouse

  23. DB::ConnectionPoolWithFailover::getManyImpl(DB::Settings const*, DB::PoolMode, std::__1::function<PoolWithFailoverBaseDB::IConnectionPool::TryResult (DB::IConnectionPool&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >&)> const&) @ 0x103633ed in /usr/bin/clickhouse

  24. DB::ConnectionPoolWithFailover::getManyChecked(DB::ConnectionTimeouts const&, DB::Settings const*, DB::PoolMode, DB::QualifiedTableName const&) @ 0x10363726 in /usr/bin/clickhouse

  25. ? @ 0xf32bcef in /usr/bin/clickhouse

  26. DB::RemoteQueryExecutor::sendQuery() @ 0xf3229af in /usr/bin/clickhouse

  27. DB::RemoteSource::tryGenerate() @ 0x10628d47 in /usr/bin/clickhouse

  28. DB::ISource::work() @ 0x1046f13a in /usr/bin/clickhouse

  29. DB::SourceWithProgress::work() @ 0x1062ee2a in /usr/bin/clickhouse

  30. ? @ 0x104aa3dd in /usr/bin/clickhouse

  31. DB::PipelineExecutor::executeStepImpl(unsigned long, unsigned long, std::__1::atomic*) @ 0x104a6f71 in /usr/bin/clickhouse

  32. ? @ 0x104ab9f6 in /usr/bin/clickhouse

  33. ThreadPoolImplstd::__1::thread::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x8bacedf in /usr/bin/clickhouse

  34. ? @ 0x8bb0403 in /usr/bin/clickhouse

  35. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so

  36. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so

2021.09.26 10:58:40.268422 [ 196 ] {} <Error> ServerErrorHandler: Code: 210, e.displayText() = DB::NetException: Connection reset by peer, while reading from socket (10.16.0.74:38048), Stack trace (when cop ying this message, always include the lines below):

  1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8b6cbba in /usr/bin/clickhouse

  2. DB::ReadBufferFromPocoSocket::nextImpl() @ 0xf5d2b53 in /usr/bin/clickhouse

  3. DB::TCPHandler::runImpl() @ 0x1042e1d4 in /usr/bin/clickhouse

  4. DB::TCPHandler::run() @ 0x10441839 in /usr/bin/clickhouse

  5. Poco::Net::TCPServerConnection::start() @ 0x12a3fd4f in /usr/bin/clickhouse

  6. Poco::Net::TCPServerDispatcher::run() @ 0x12a417da in /usr/bin/clickhouse

  7. Poco::PooledThread::run() @ 0x12b7ab39 in /usr/bin/clickhouse

  8. Poco::ThreadImpl::runnableEntry(void*) @ 0x12b76b2a in /usr/bin/clickhouse

  9. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so

  10. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so (version 21.6.5.37 (official build)) 2021.09.26 10:58:40.289323 [ 121 ] {1568d01d-e4a2-4b9a-9556-ba6cf8687e9c} <Error> executeQuery: Code: 516, e.displayText() = DB::Exception: Received from chi-clickhouse-woxshare-0-1:9000. DB::Exception: def ault: Authentication failed: password is incorrect or there is no user with such name. Stack trace:

  11. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8b6cbba in /usr/bin/clickhouse

  12. DB::IAccessStorage::throwCannotAuthenticate(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&) @ 0xf0cb81f in /usr/bin/clickhouse

  13. DB::IAccessStorage::login(DB::Credentials const&, Poco::Net::IPAddress const&, DB::ExternalAuthenticators const&, bool) const @ 0xf0cb662 in /usr/bin/clickhouse

  14. DB::Context::setUser(DB::Credentials const&, Poco::Net::SocketAddress const&) @ 0xf5191b9 in /usr/bin/clickhouse

  15. DB::TCPHandler::receiveHello() @ 0x10434aac in /usr/bin/clickhouse

  16. DB::TCPHandler::runImpl() @ 0x1042de74 in /usr/bin/clickhouse

  17. DB::TCPHandler::run() @ 0x10441839 in /usr/bin/clickhouse

  18. Poco::Net::TCPServerConnection::start() @ 0x12a3fd4f in /usr/bin/clickhouse

  19. Poco::Net::TCPServerDispatcher::run() @ 0x12a417da in /usr/bin/clickhouse

  20. Poco::PooledThread::run() @ 0x12b7ab39 in /usr/bin/clickhouse

  21. Poco::ThreadImpl::runnableEntry(void*) @ 0x12b76b2a in /usr/bin/clickhouse

  22. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so

  23. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so : While executing Remote (version 21.6.5.37 (official build)) (from 10.16.0.81:56210) (in query: SELECT service_name, lang, error_num, slow_api_num FROM (select service_name, lang, sum(error_num) error_num, sum(slow_api_num) slow_api_num from ((SELECT service_name, tracker_lang as lang, 0 as error_num, count(distinct path(JSONExtractString(tags, 'url'))) AS slow_api_num FROM 446847926040268800.log_detail_a ll WHERE (1=1 and (event_time >= toDateTime('2021-09-20 00:00:00', 'Asia/Shanghai') and event_time < toDateTime('2021-09-26 23:59:59', 'Asia/Shanghai')+1 )) AND span_type=0 AND duration > 1000 GROU P BY tracker_lang, service_name) union all (SELECT service_name, tracker_lang as lang, count(distinct fingerprint) error_num, 0 as slow_api_num FROM 446847926040268800.log_mv_trace_errstat_all WHERE 1=1 and (event_time >= toDateTime('2021-09-20 00:00:00', 'Asia/Shanghai') and event_time < toDateTime('2021-09-26 23:59:59', 'Asia/Shanghai')+1 ) GROUP BY tracker_lang, service_name)) as t group by lan g, service_name) as tt ORDER BY error_num desc LIMIT 100), Stack trace (when copying this message, always include the lines below):

  24. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8b6cbba in /usr/bin/clickhouse

  25. DB::readException(DB::ReadBuffer&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, bool) @ 0x8bc8fd4 in /usr/bin/clickhouse

  26. DB::Connection::receiveException() const @ 0x10352e82 in /usr/bin/clickhouse

  27. DB::Connection::receiveHello() @ 0x10352a1f in /usr/bin/clickhouse

  28. DB::Connection::connect(DB::ConnectionTimeouts const&) @ 0x1035026c in /usr/bin/clickhouse

  29. DB::ConnectionEstablisher::run(PoolWithFailoverBaseDB::IConnectionPool::TryResult&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >&) @ 0x1036a0d7 in /usr/bin/clic khouse

  30. ? @ 0x10366277 in /usr/bin/clickhouse

  31. PoolWithFailoverBaseDB::IConnectionPool::getMany(unsigned long, unsigned long, unsigned long, unsigned long, bool, std::__1::function<PoolWithFailoverBaseDB::IConnectionPool::TryResult (DB::IConnecti onPool&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >&)> const&, std::__1::function<unsigned long (unsigned long)> const&) @ 0x10363b25 in /usr/bin/clickhouse

  32. DB::ConnectionPoolWithFailover::getManyImpl(DB::Settings const*, DB::PoolMode, std::__1::function<PoolWithFailoverBaseDB::IConnectionPool::TryResult (DB::IConnectionPool&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >&)> const&) @ 0x103633ed in /usr/bin/clickhouse

  33. DB::ConnectionPoolWithFailover::getManyChecked(DB::ConnectionTimeouts const&, DB::Settings const*, DB::PoolMode, DB::QualifiedTableName const&) @ 0x10363726 in /usr/bin/clickhouse

  34. ? @ 0xf32bcef in /usr/bin/clickhouse

  35. DB::RemoteQueryExecutor::sendQuery() @ 0xf3229af in /usr/bin/clickhouse

  36. DB::RemoteSource::tryGenerate() @ 0x10628d47 in /usr/bin/clickhouse

  37. DB::ISource::work() @ 0x1046f13a in /usr/bin/clickhouse

  38. DB::SourceWithProgress::work() @ 0x1062ee2a in /usr/bin/clickhouse

  39. ? @ 0x104aa3dd in /usr/bin/clickhouse

  40. DB::PipelineExecutor::executeStepImpl(unsigned long, unsigned long, std::__1::atomic*) @ 0x104a6f71 in /usr/bin/clickhouse

  41. ? @ 0x104ab9f6 in /usr/bin/clickhouse

  42. ThreadPoolImplstd::__1::thread::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x8bacedf in /usr/bin/clickhouse

  43. ? @ 0x8bb0403 in /usr/bin/clickhouse

  44. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so

  45. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so

2021.09.26 10:58:40.289392 [ 121 ] {1568d01d-e4a2-4b9a-9556-ba6cf8687e9c} <Error> TCPHandler: Code: 516, e.displayText() = DB::Exception: Received from chi-clickhouse-woxshare-0-1:9000. DB::Exception: defau lt: Authentication failed: password is incorrect or there is no user with such name. Stack trace:

  1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8b6cbba in /usr/bin/clickhouse

  2. DB::IAccessStorage::throwCannotAuthenticate(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&) @ 0xf0cb81f in /usr/bin/clickhouse

  3. DB::IAccessStorage::login(DB::Credentials const&, Poco::Net::IPAddress const&, DB::ExternalAuthenticators const&, bool) const @ 0xf0cb662 in /usr/bin/clickhouse

  4. DB::Context::setUser(DB::Credentials const&, Poco::Net::SocketAddress const&) @ 0xf5191b9 in /usr/bin/clickhouse

  5. DB::TCPHandler::receiveHello() @ 0x10434aac in /usr/bin/clickhouse

  6. DB::TCPHandler::runImpl() @ 0x1042de74 in /usr/bin/clickhouse

  7. DB::TCPHandler::run() @ 0x10441839 in /usr/bin/clickhouse

  8. Poco::Net::TCPServerConnection::start() @ 0x12a3fd4f in /usr/bin/clickhouse

  9. Poco::Net::TCPServerDispatcher::run() @ 0x12a417da in /usr/bin/clickhouse

  10. Poco::PooledThread::run() @ 0x12b7ab39 in /usr/bin/clickhouse

  11. Poco::ThreadImpl::runnableEntry(void*) @ 0x12b76b2a in /usr/bin/clickhouse

  12. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so

  13. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so : While executing Remote, Stack trace:

  14. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8b6cbba in /usr/bin/clickhouse

  15. DB::readException(DB::ReadBuffer&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, bool) @ 0x8bc8fd4 in /usr/bin/clickhouse

  16. DB::Connection::receiveException() const @ 0x10352e82 in /usr/bin/clickhouse

  17. DB::Connection::receiveHello() @ 0x10352a1f in /usr/bin/clickhouse

  18. DB::Connection::connect(DB::ConnectionTimeouts const&) @ 0x1035026c in /usr/bin/clickhouse

  19. DB::ConnectionEstablisher::run(PoolWithFailoverBaseDB::IConnectionPool::TryResult&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >&) @ 0x1036a0d7 in /usr/bin/clic khouse

  20. ? @ 0x10366277 in /usr/bin/clickhouse

  21. PoolWithFailoverBaseDB::IConnectionPool::getMany(unsigned long, unsigned long, unsigned long, unsigned long, bool, std::__1::function<PoolWithFailoverBaseDB::IConnectionPool::TryResult (DB::IConnecti onPool&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >&)> const&, std::__1::function<unsigned long (unsigned long)> const&) @ 0x10363b25 in /usr/bin/clickhouse

  22. DB::ConnectionPoolWithFailover::getManyImpl(DB::Settings const*, DB::PoolMode, std::__1::function<PoolWithFailoverBaseDB::IConnectionPool::TryResult (DB::IConnectionPool&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >&)> const&) @ 0x103633ed in /usr/bin/clickhouse

  23. DB::ConnectionPoolWithFailover::getManyChecked(DB::ConnectionTimeouts const&, DB::Settings const*, DB::PoolMode, DB::QualifiedTableName const&) @ 0x10363726 in /usr/bin/clickhouse

  24. ? @ 0xf32bcef in /usr/bin/clickhouse

  25. DB::RemoteQueryExecutor::sendQuery() @ 0xf3229af in /usr/bin/clickhouse

  26. DB::RemoteSource::tryGenerate() @ 0x10628d47 in /usr/bin/clickhouse

  27. DB::ISource::work() @ 0x1046f13a in /usr/bin/clickhouse

  28. DB::SourceWithProgress::work() @ 0x1062ee2a in /usr/bin/clickhouse

  29. ? @ 0x104aa3dd in /usr/bin/clickhouse

  30. DB::PipelineExecutor::executeStepImpl(unsigned long, unsigned long, std::__1::atomic*) @ 0x104a6f71 in /usr/bin/clickhouse

  31. ? @ 0x104ab9f6 in /usr/bin/clickhouse

  32. ThreadPoolImplstd::__1::thread::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x8bacedf in /usr/bin/clickhouse

  33. ? @ 0x8bb0403 in /usr/bin/clickhouse

  34. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so

  35. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so

2021.09.26 10:58:40.438361 [ 196 ] {} <Error> Access(user directories): default: Authentication failed: Code: 195, e.displayText() = DB::Exception: Connections from 10.16.0.74 are not allowed, Stack trace ( when copying this message, always include the lines below):

  1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8b6cbba in /usr/bin/clickhouse

  2. DB::IAccessStorage::throwAddressNotAllowed(Poco::Net::IPAddress const&) @ 0xf0cbb83 in /usr/bin/clickhouse

  3. DB::IAccessStorage::loginImpl(DB::Credentials const&, Poco::Net::IPAddress const&, DB::ExternalAuthenticators const&) const @ 0xf0cba96 in /usr/bin/clickhouse

  4. DB::MultipleAccessStorage::loginImpl(DB::Credentials const&, Poco::Net::IPAddress const&, DB::ExternalAuthenticators const&) const @ 0xf0ef998 in /usr/bin/clickhouse

  5. DB::IAccessStorage::login(DB::Credentials const&, Poco::Net::IPAddress const&, DB::ExternalAuthenticators const&, bool) const @ 0xf0cb5dc in /usr/bin/clickhouse

  6. DB::Context::setUser(DB::Credentials const&, Poco::Net::SocketAddress const&) @ 0xf5191b9 in /usr/bin/clickhouse

  7. DB::TCPHandler::receiveHello() @ 0x10434aac in /usr/bin/clickhouse

  8. DB::TCPHandler::runImpl() @ 0x1042de74 in /usr/bin/clickhouse

  9. DB::TCPHandler::run() @ 0x10441839 in /usr/bin/clickhouse

  10. Poco::Net::TCPServerConnection::start() @ 0x12a3fd4f in /usr/bin/clickhouse

  11. Poco::Net::TCPServerDispatcher::run() @ 0x12a417da in /usr/bin/clickhouse

  12. Poco::PooledThread::run() @ 0x12b7ab39 in /usr/bin/clickhouse

  13. Poco::ThreadImpl::runnableEntry(void*) @ 0x12b76b2a in /usr/bin/clickhouse

  14. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so

  15. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so (version 21.6.5.37 (official build)) 2021.09.26 10:58:40.438537 [ 196 ] {} <Error> ServerErrorHandler: Code: 516, e.displayText() = DB::Exception: default: Authentication failed: password is incorrect or there is no user with such name, Stack trace (when copying this message, always include the lines below):

  16. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8b6cbba in /usr/bin/clickhouse

  17. DB::IAccessStorage::throwCannotAuthenticate(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&) @ 0xf0cb81f in /usr/bin/clickhouse

  18. DB::IAccessStorage::login(DB::Credentials const&, Poco::Net::IPAddress const&, DB::ExternalAuthenticators const&, bool) const @ 0xf0cb662 in /usr/bin/clickhouse

  19. DB::Context::setUser(DB::Credentials const&, Poco::Net::SocketAddress const&) @ 0xf5191b9 in /usr/bin/clickhouse

  20. DB::TCPHandler::receiveHello() @ 0x10434aac in /usr/bin/clickhouse

  21. DB::TCPHandler::runImpl() @ 0x1042de74 in /usr/bin/clickhouse

  22. DB::TCPHandler::run() @ 0x10441839 in /usr/bin/clickhouse

  23. Poco::Net::TCPServerConnection::start() @ 0x12a3fd4f in /usr/bin/clickhouse

  24. Poco::Net::TCPServerDispatcher::run() @ 0x12a417da in /usr/bin/clickhouse

  25. Poco::PooledThread::run() @ 0x12b7ab39 in /usr/bin/clickhouse

  26. Poco::ThreadImpl::runnableEntry(void*) @ 0x12b76b2a in /usr/bin/clickhouse

  27. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so

  28. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so (version 21.6.5.37 (official build)) 2021.09.26 10:58:44.638217 [ 188 ] {fcaef4b6-0816-4264-859b-ed84efe3e8b9} <Warning> ConnectionPoolWithFailover: Connection failed at try №1, reason: Code: 32, e.displayText() = DB::Exception: Attempt to rea d after eof (version 21.6.5.37 (official build))

czhfe avatar Sep 26 '21 05:09 czhfe

Try to use dns cache, the k8s dns have some problem.

Sallery-X avatar Sep 26 '21 06:09 Sallery-X

Try to use dns cache, the k8s dns have some problem.

Is this caused by kubernetes dns resolution?

czhfe avatar Sep 27 '21 06:09 czhfe

@czhfe, do you use chCredentialsSecretName in your etc-clickhouse-operator-files ConfigMap?

Slach avatar Sep 28 '21 14:09 Slach

chCredentialsSecretName

Not used

czhfe avatar Sep 29 '21 10:09 czhfe

2021.10.14 14:13:09.834562 [ 243 ] {} <Error> Access(user directories): default: Authentication failed: Code: 195, e.displayText() = DB::Exception: Connections from 10.16.1.32 are not allowed, Stack trace (when copying this message, always include the lines below):

  1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8b6cbba in /usr/bin/clickhouse

  2. DB::IAccessStorage::throwAddressNotAllowed(Poco::Net::IPAddress const&) @ 0xf0cbb83 in /usr/bin/clickhouse

  3. DB::IAccessStorage::loginImpl(DB::Credentials const&, Poco::Net::IPAddress const&, DB::ExternalAuthenticators const&) const @ 0xf0cba96 in /usr/bin/clickhouse

  4. DB::MultipleAccessStorage::loginImpl(DB::Credentials const&, Poco::Net::IPAddress const&, DB::ExternalAuthenticators const&) const @ 0xf0ef998 in /usr/bin/clickhouse

  5. DB::IAccessStorage::login(DB::Credentials const&, Poco::Net::IPAddress const&, DB::ExternalAuthenticators const&, bool) const @ 0xf0cb5dc in /usr/bin/clickhouse

  6. DB::Context::setUser(DB::Credentials const&, Poco::Net::SocketAddress const&) @ 0xf5191b9 in /usr/bin/clickhouse

  7. DB::TCPHandler::receiveHello() @ 0x10434aac in /usr/bin/clickhouse

  8. DB::TCPHandler::runImpl() @ 0x1042de74 in /usr/bin/clickhouse

  9. DB::TCPHandler::run() @ 0x10441839 in /usr/bin/clickhouse

  10. Poco::Net::TCPServerConnection::start() @ 0x12a3fd4f in /usr/bin/clickhouse

  11. Poco::Net::TCPServerDispatcher::run() @ 0x12a417da in /usr/bin/clickhouse

  12. Poco::PooledThread::run() @ 0x12b7ab39 in /usr/bin/clickhouse

  13. Poco::ThreadImpl::runnableEntry(void*) @ 0x12b76b2a in /usr/bin/clickhouse

  14. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so

  15. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so (version 21.6.5.37 (official build)) 2021.10.14 14:13:09.834745 [ 243 ] {} <Error> ServerErrorHandler: Code: 516, e.displayText() = DB::Exception: default: Authentication failed: password is incorrect or there is no user with such name, Stack trace (when copying this message, always include the lines below):

  16. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8b6cbba in /usr/bin/clickhouse

  17. DB::IAccessStorage::throwCannotAuthenticate(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&) @ 0xf0cb81f in /usr/bin/clickhouse

  18. DB::IAccessStorage::login(DB::Credentials const&, Poco::Net::IPAddress const&, DB::ExternalAuthenticators const&, bool) const @ 0xf0cb662 in /usr/bin/clickhouse

  19. DB::Context::setUser(DB::Credentials const&, Poco::Net::SocketAddress const&) @ 0xf5191b9 in /usr/bin/clickhouse

  20. DB::TCPHandler::receiveHello() @ 0x10434aac in /usr/bin/clickhouse

  21. DB::TCPHandler::runImpl() @ 0x1042de74 in /usr/bin/clickhouse

  22. DB::TCPHandler::run() @ 0x10441839 in /usr/bin/clickhouse

  23. Poco::Net::TCPServerConnection::start() @ 0x12a3fd4f in /usr/bin/clickhouse

  24. Poco::Net::TCPServerDispatcher::run() @ 0x12a417da in /usr/bin/clickhouse

  25. Poco::PooledThread::run() @ 0x12b7ab39 in /usr/bin/clickhouse

  26. Poco::ThreadImpl::runnableEntry(void*) @ 0x12b76b2a in /usr/bin/clickhouse

  27. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so

  28. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so (version 21.6.5.37 (official build)) 2021.10.14 14:13:09.886759 [ 239 ] {} <Error> ServerErrorHandler: Code: 210, e.displayText() = DB::NetException: Connection reset by peer, while reading from socket (10.16.0.79:36340), Stack trace (when copying this message, always include the lines below):

  29. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8b6cbba in /usr/bin/clickhouse

  30. DB::ReadBufferFromPocoSocket::nextImpl() @ 0xf5d2b53 in /usr/bin/clickhouse

  31. DB::TCPHandler::runImpl() @ 0x1042e1d4 in /usr/bin/clickhouse

  32. DB::TCPHandler::run() @ 0x10441839 in /usr/bin/clickhouse

  33. Poco::Net::TCPServerConnection::start() @ 0x12a3fd4f in /usr/bin/clickhouse

  34. Poco::Net::TCPServerDispatcher::run() @ 0x12a417da in /usr/bin/clickhouse

  35. Poco::PooledThread::run() @ 0x12b7ab39 in /usr/bin/clickhouse

  36. Poco::ThreadImpl::runnableEntry(void*) @ 0x12b76b2a in /usr/bin/clickhouse

  37. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so

  38. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so (version 21.6.5.37 (official build)) 2021.10.14 14:13:10.011699 [ 242 ] {} <Error> ServerErrorHandler: Code: 210, e.displayText() = DB::NetException: Connection reset by peer, while reading from socket (10.16.1.40:39208), Stack trace (when copying this message, always include the lines below):

  39. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8b6cbba in /usr/bin/clickhouse

  40. DB::ReadBufferFromPocoSocket::nextImpl() @ 0xf5d2b53 in /usr/bin/clickhouse

  41. DB::TCPHandler::runImpl() @ 0x1042e1d4 in /usr/bin/clickhouse

  42. DB::TCPHandler::run() @ 0x10441839 in /usr/bin/clickhouse

  43. Poco::Net::TCPServerConnection::start() @ 0x12a3fd4f in /usr/bin/clickhouse

  44. Poco::Net::TCPServerDispatcher::run() @ 0x12a417da in /usr/bin/clickhouse

  45. Poco::PooledThread::run() @ 0x12b7ab39 in /usr/bin/clickhouse

  46. Poco::ThreadImpl::runnableEntry(void*) @ 0x12b76b2a in /usr/bin/clickhouse

  47. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so

  48. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so (version 21.6.5.37 (official build)) 2021.10.14 14:13:10.074208 [ 242 ] {} <Error> Access(user directories): default: Authentication failed: Code: 195, e.displayText() = DB::Exception: Connections from 10.16.1.32 are not allowed, Stack trace (when copying this message, always include the lines below):

  49. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8b6cbba in /usr/bin/clickhouse

  50. DB::IAccessStorage::throwAddressNotAllowed(Poco::Net::IPAddress const&) @ 0xf0cbb83 in /usr/bin/clickhouse

  51. DB::IAccessStorage::loginImpl(DB::Credentials const&, Poco::Net::IPAddress const&, DB::ExternalAuthenticators const&) const @ 0xf0cba96 in /usr/bin/clickhouse

  52. DB::MultipleAccessStorage::loginImpl(DB::Credentials const&, Poco::Net::IPAddress const&, DB::ExternalAuthenticators const&) const @ 0xf0ef998 in /usr/bin/clickhouse

  53. DB::IAccessStorage::login(DB::Credentials const&, Poco::Net::IPAddress const&, DB::ExternalAuthenticators const&, bool) const @ 0xf0cb5dc in /usr/bin/clickhouse

  54. DB::Context::setUser(DB::Credentials const&, Poco::Net::SocketAddress const&) @ 0xf5191b9 in /usr/bin/clickhouse

  55. DB::TCPHandler::receiveHello() @ 0x10434aac in /usr/bin/clickhouse

  56. DB::TCPHandler::runImpl() @ 0x1042de74 in /usr/bin/clickhouse

  57. DB::TCPHandler::run() @ 0x10441839 in /usr/bin/clickhouse

  58. Poco::Net::TCPServerConnection::start() @ 0x12a3fd4f in /usr/bin/clickhouse

  59. Poco::Net::TCPServerDispatcher::run() @ 0x12a417da in /usr/bin/clickhouse

  60. Poco::PooledThread::run() @ 0x12b7ab39 in /usr/bin/clickhouse

  61. Poco::ThreadImpl::runnableEntry(void*) @ 0x12b76b2a in /usr/bin/clickhouse

  62. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so

  63. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so (version 21.6.5.37 (official build)) 2021.10.14 14:13:10.074389 [ 242 ] {} <Error> ServerErrorHandler: Code: 516, e.displayText() = DB::Exception: default: Authentication failed: password is incorrect or there is no user with such name, Stack trace (when copying this message, always include the lines below):

  64. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8b6cbba in /usr/bin/clickhouse

  65. DB::IAccessStorage::throwCannotAuthenticate(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&) @ 0xf0cb81f in /usr/bin/clickhouse

  66. DB::IAccessStorage::login(DB::Credentials const&, Poco::Net::IPAddress const&, DB::ExternalAuthenticators const&, bool) const @ 0xf0cb662 in /usr/bin/clickhouse

  67. DB::Context::setUser(DB::Credentials const&, Poco::Net::SocketAddress const&) @ 0xf5191b9 in /usr/bin/clickhouse

  68. DB::TCPHandler::receiveHello() @ 0x10434aac in /usr/bin/clickhouse

  69. DB::TCPHandler::runImpl() @ 0x1042de74 in /usr/bin/clickhouse

  70. DB::TCPHandler::run() @ 0x10441839 in /usr/bin/clickhouse

  71. Poco::Net::TCPServerConnection::start() @ 0x12a3fd4f in /usr/bin/clickhouse

  72. Poco::Net::TCPServerDispatcher::run() @ 0x12a417da in /usr/bin/clickhouse

  73. Poco::PooledThread::run() @ 0x12b7ab39 in /usr/bin/clickhouse

  74. Poco::ThreadImpl::runnableEntry(void*) @ 0x12b76b2a in /usr/bin/clickhouse

  75. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so

  76. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so (version 21.6.5.37 (official build)) 2021.10.14 14:13:10.180054 [ 240 ] {} <Error> ServerErrorHandler: Code: 210, e.displayText() = DB::NetException: Connection reset by peer, while reading from socket (10.16.0.79:36430), Stack trace (when copying this message, always include the lines below):

  77. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8b6cbba in /usr/bin/clickhouse

  78. DB::ReadBufferFromPocoSocket::nextImpl() @ 0xf5d2b53 in /usr/bin/clickhouse

  79. DB::TCPHandler::runImpl() @ 0x1042e1d4 in /usr/bin/clickhouse

  80. DB::TCPHandler::run() @ 0x10441839 in /usr/bin/clickhouse

  81. Poco::Net::TCPServerConnection::start() @ 0x12a3fd4f in /usr/bin/clickhouse

  82. Poco::Net::TCPServerDispatcher::run() @ 0x12a417da in /usr/bin/clickhouse

  83. Poco::PooledThread::run() @ 0x12b7ab39 in /usr/bin/clickhouse

  84. Poco::ThreadImpl::runnableEntry(void*) @ 0x12b76b2a in /usr/bin/clickhouse

  85. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so

  86. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so (version 21.6.5.37 (official build)) 2021.10.14 14:13:10.180175 [ 238 ] {} <Error> ServerErrorHandler: Code: 210, e.displayText() = DB::NetException: Connection reset by peer, while reading from socket (10.16.0.79:36408), Stack trace (when copying this message, always include the lines below):

  87. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8b6cbba in /usr/bin/clickhouse

  88. DB::ReadBufferFromPocoSocket::nextImpl() @ 0xf5d2b53 in /usr/bin/clickhouse

  89. DB::TCPHandler::runImpl() @ 0x1042e1d4 in /usr/bin/clickhouse

  90. DB::TCPHandler::run() @ 0x10441839 in /usr/bin/clickhouse

  91. Poco::Net::TCPServerConnection::start() @ 0x12a3fd4f in /usr/bin/clickhouse

  92. Poco::Net::TCPServerDispatcher::run() @ 0x12a417da in /usr/bin/clickhouse

  93. Poco::PooledThread::run() @ 0x12b7ab39 in /usr/bin/clickhouse

  94. Poco::ThreadImpl::runnableEntry(void*) @ 0x12b76b2a in /usr/bin/clickhouse

  95. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so

  96. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so (version 21.6.5.37 (official build))

This should be a problem with the clickhouse whitelist. Because clickhouse-operator deploys clickhouse with default user networks set to allow only local and nodes in the same cluster to access, when a copy of clickhouse changes IP after reboot for some reason, this time the node resumes after DNS resolution out of IP may exist cache In this case, the IP resolved by the node after recovery may be cached and may not be consistent with the IP resolved by the host in the whitelist, thus being specifically accessed. I have tried to set disable_internal_dns_cache to 1, but this did not alleviate the problem

czhfe avatar Oct 15 '21 09:10 czhfe

2021.10.14 14:13:09.834562 [ 243 ] {} Access(user directories): default: Authentication failed: Code: 195, e.displayText() = DB::Exception: Connections from 10.16.1.32 are not allowed, Stack trace (when copying this message, always include the lines below):

  1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8b6cbba in /usr/bin/clickhouse
  2. DB::IAccessStorage::throwAddressNotAllowed(Poco::Net::IPAddress const&) @ 0xf0cbb83 in /usr/bin/clickhouse
  3. DB::IAccessStorage::loginImpl(DB::Credentials const&, Poco::Net::IPAddress const&, DB::ExternalAuthenticators const&) const @ 0xf0cba96 in /usr/bin/clickhouse
  4. DB::MultipleAccessStorage::loginImpl(DB::Credentials const&, Poco::Net::IPAddress const&, DB::ExternalAuthenticators const&) const @ 0xf0ef998 in /usr/bin/clickhouse
  5. DB::IAccessStorage::login(DB::Credentials const&, Poco::Net::IPAddress const&, DB::ExternalAuthenticators const&, bool) const @ 0xf0cb5dc in /usr/bin/clickhouse
  6. DB::Context::setUser(DB::Credentials const&, Poco::Net::SocketAddress const&) @ 0xf5191b9 in /usr/bin/clickhouse
  7. DB::TCPHandler::receiveHello() @ 0x10434aac in /usr/bin/clickhouse
  8. DB::TCPHandler::runImpl() @ 0x1042de74 in /usr/bin/clickhouse
  9. DB::TCPHandler::run() @ 0x10441839 in /usr/bin/clickhouse
  10. Poco::Net::TCPServerConnection::start() @ 0x12a3fd4f in /usr/bin/clickhouse
  11. Poco::Net::TCPServerDispatcher::run() @ 0x12a417da in /usr/bin/clickhouse
  12. Poco::PooledThread::run() @ 0x12b7ab39 in /usr/bin/clickhouse
  13. Poco::ThreadImpl::runnableEntry(void*) @ 0x12b76b2a in /usr/bin/clickhouse
  14. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
  15. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so (version 21.6.5.37 (official build)) 2021.10.14 14:13:09.834745 [ 243 ] {} ServerErrorHandler: Code: 516, e.displayText() = DB::Exception: default: Authentication failed: password is incorrect or there is no user with such name, Stack trace (when copying this message, always include the lines below):
  16. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8b6cbba in /usr/bin/clickhouse
  17. DB::IAccessStorage::throwCannotAuthenticate(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&) @ 0xf0cb81f in /usr/bin/clickhouse
  18. DB::IAccessStorage::login(DB::Credentials const&, Poco::Net::IPAddress const&, DB::ExternalAuthenticators const&, bool) const @ 0xf0cb662 in /usr/bin/clickhouse
  19. DB::Context::setUser(DB::Credentials const&, Poco::Net::SocketAddress const&) @ 0xf5191b9 in /usr/bin/clickhouse
  20. DB::TCPHandler::receiveHello() @ 0x10434aac in /usr/bin/clickhouse
  21. DB::TCPHandler::runImpl() @ 0x1042de74 in /usr/bin/clickhouse
  22. DB::TCPHandler::run() @ 0x10441839 in /usr/bin/clickhouse
  23. Poco::Net::TCPServerConnection::start() @ 0x12a3fd4f in /usr/bin/clickhouse
  24. Poco::Net::TCPServerDispatcher::run() @ 0x12a417da in /usr/bin/clickhouse
  25. Poco::PooledThread::run() @ 0x12b7ab39 in /usr/bin/clickhouse
  26. Poco::ThreadImpl::runnableEntry(void*) @ 0x12b76b2a in /usr/bin/clickhouse
  27. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
  28. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so (version 21.6.5.37 (official build)) 2021.10.14 14:13:09.886759 [ 239 ] {} ServerErrorHandler: Code: 210, e.displayText() = DB::NetException: Connection reset by peer, while reading from socket (10.16.0.79:36340), Stack trace (when copying this message, always include the lines below):
  29. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8b6cbba in /usr/bin/clickhouse
  30. DB::ReadBufferFromPocoSocket::nextImpl() @ 0xf5d2b53 in /usr/bin/clickhouse
  31. DB::TCPHandler::runImpl() @ 0x1042e1d4 in /usr/bin/clickhouse
  32. DB::TCPHandler::run() @ 0x10441839 in /usr/bin/clickhouse
  33. Poco::Net::TCPServerConnection::start() @ 0x12a3fd4f in /usr/bin/clickhouse
  34. Poco::Net::TCPServerDispatcher::run() @ 0x12a417da in /usr/bin/clickhouse
  35. Poco::PooledThread::run() @ 0x12b7ab39 in /usr/bin/clickhouse
  36. Poco::ThreadImpl::runnableEntry(void*) @ 0x12b76b2a in /usr/bin/clickhouse
  37. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
  38. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so (version 21.6.5.37 (official build)) 2021.10.14 14:13:10.011699 [ 242 ] {} ServerErrorHandler: Code: 210, e.displayText() = DB::NetException: Connection reset by peer, while reading from socket (10.16.1.40:39208), Stack trace (when copying this message, always include the lines below):
  39. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8b6cbba in /usr/bin/clickhouse
  40. DB::ReadBufferFromPocoSocket::nextImpl() @ 0xf5d2b53 in /usr/bin/clickhouse
  41. DB::TCPHandler::runImpl() @ 0x1042e1d4 in /usr/bin/clickhouse
  42. DB::TCPHandler::run() @ 0x10441839 in /usr/bin/clickhouse
  43. Poco::Net::TCPServerConnection::start() @ 0x12a3fd4f in /usr/bin/clickhouse
  44. Poco::Net::TCPServerDispatcher::run() @ 0x12a417da in /usr/bin/clickhouse
  45. Poco::PooledThread::run() @ 0x12b7ab39 in /usr/bin/clickhouse
  46. Poco::ThreadImpl::runnableEntry(void*) @ 0x12b76b2a in /usr/bin/clickhouse
  47. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
  48. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so (version 21.6.5.37 (official build)) 2021.10.14 14:13:10.074208 [ 242 ] {} Access(user directories): default: Authentication failed: Code: 195, e.displayText() = DB::Exception: Connections from 10.16.1.32 are not allowed, Stack trace (when copying this message, always include the lines below):
  49. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8b6cbba in /usr/bin/clickhouse
  50. DB::IAccessStorage::throwAddressNotAllowed(Poco::Net::IPAddress const&) @ 0xf0cbb83 in /usr/bin/clickhouse
  51. DB::IAccessStorage::loginImpl(DB::Credentials const&, Poco::Net::IPAddress const&, DB::ExternalAuthenticators const&) const @ 0xf0cba96 in /usr/bin/clickhouse
  52. DB::MultipleAccessStorage::loginImpl(DB::Credentials const&, Poco::Net::IPAddress const&, DB::ExternalAuthenticators const&) const @ 0xf0ef998 in /usr/bin/clickhouse
  53. DB::IAccessStorage::login(DB::Credentials const&, Poco::Net::IPAddress const&, DB::ExternalAuthenticators const&, bool) const @ 0xf0cb5dc in /usr/bin/clickhouse
  54. DB::Context::setUser(DB::Credentials const&, Poco::Net::SocketAddress const&) @ 0xf5191b9 in /usr/bin/clickhouse
  55. DB::TCPHandler::receiveHello() @ 0x10434aac in /usr/bin/clickhouse
  56. DB::TCPHandler::runImpl() @ 0x1042de74 in /usr/bin/clickhouse
  57. DB::TCPHandler::run() @ 0x10441839 in /usr/bin/clickhouse
  58. Poco::Net::TCPServerConnection::start() @ 0x12a3fd4f in /usr/bin/clickhouse
  59. Poco::Net::TCPServerDispatcher::run() @ 0x12a417da in /usr/bin/clickhouse
  60. Poco::PooledThread::run() @ 0x12b7ab39 in /usr/bin/clickhouse
  61. Poco::ThreadImpl::runnableEntry(void*) @ 0x12b76b2a in /usr/bin/clickhouse
  62. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
  63. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so (version 21.6.5.37 (official build)) 2021.10.14 14:13:10.074389 [ 242 ] {} ServerErrorHandler: Code: 516, e.displayText() = DB::Exception: default: Authentication failed: password is incorrect or there is no user with such name, Stack trace (when copying this message, always include the lines below):
  64. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8b6cbba in /usr/bin/clickhouse
  65. DB::IAccessStorage::throwCannotAuthenticate(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&) @ 0xf0cb81f in /usr/bin/clickhouse
  66. DB::IAccessStorage::login(DB::Credentials const&, Poco::Net::IPAddress const&, DB::ExternalAuthenticators const&, bool) const @ 0xf0cb662 in /usr/bin/clickhouse
  67. DB::Context::setUser(DB::Credentials const&, Poco::Net::SocketAddress const&) @ 0xf5191b9 in /usr/bin/clickhouse
  68. DB::TCPHandler::receiveHello() @ 0x10434aac in /usr/bin/clickhouse
  69. DB::TCPHandler::runImpl() @ 0x1042de74 in /usr/bin/clickhouse
  70. DB::TCPHandler::run() @ 0x10441839 in /usr/bin/clickhouse
  71. Poco::Net::TCPServerConnection::start() @ 0x12a3fd4f in /usr/bin/clickhouse
  72. Poco::Net::TCPServerDispatcher::run() @ 0x12a417da in /usr/bin/clickhouse
  73. Poco::PooledThread::run() @ 0x12b7ab39 in /usr/bin/clickhouse
  74. Poco::ThreadImpl::runnableEntry(void*) @ 0x12b76b2a in /usr/bin/clickhouse
  75. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
  76. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so (version 21.6.5.37 (official build)) 2021.10.14 14:13:10.180054 [ 240 ] {} ServerErrorHandler: Code: 210, e.displayText() = DB::NetException: Connection reset by peer, while reading from socket (10.16.0.79:36430), Stack trace (when copying this message, always include the lines below):
  77. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8b6cbba in /usr/bin/clickhouse
  78. DB::ReadBufferFromPocoSocket::nextImpl() @ 0xf5d2b53 in /usr/bin/clickhouse
  79. DB::TCPHandler::runImpl() @ 0x1042e1d4 in /usr/bin/clickhouse
  80. DB::TCPHandler::run() @ 0x10441839 in /usr/bin/clickhouse
  81. Poco::Net::TCPServerConnection::start() @ 0x12a3fd4f in /usr/bin/clickhouse
  82. Poco::Net::TCPServerDispatcher::run() @ 0x12a417da in /usr/bin/clickhouse
  83. Poco::PooledThread::run() @ 0x12b7ab39 in /usr/bin/clickhouse
  84. Poco::ThreadImpl::runnableEntry(void*) @ 0x12b76b2a in /usr/bin/clickhouse
  85. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
  86. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so (version 21.6.5.37 (official build)) 2021.10.14 14:13:10.180175 [ 238 ] {} ServerErrorHandler: Code: 210, e.displayText() = DB::NetException: Connection reset by peer, while reading from socket (10.16.0.79:36408), Stack trace (when copying this message, always include the lines below):
  87. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int, bool) @ 0x8b6cbba in /usr/bin/clickhouse
  88. DB::ReadBufferFromPocoSocket::nextImpl() @ 0xf5d2b53 in /usr/bin/clickhouse
  89. DB::TCPHandler::runImpl() @ 0x1042e1d4 in /usr/bin/clickhouse
  90. DB::TCPHandler::run() @ 0x10441839 in /usr/bin/clickhouse
  91. Poco::Net::TCPServerConnection::start() @ 0x12a3fd4f in /usr/bin/clickhouse
  92. Poco::Net::TCPServerDispatcher::run() @ 0x12a417da in /usr/bin/clickhouse
  93. Poco::PooledThread::run() @ 0x12b7ab39 in /usr/bin/clickhouse
  94. Poco::ThreadImpl::runnableEntry(void*) @ 0x12b76b2a in /usr/bin/clickhouse
  95. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
  96. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so (version 21.6.5.37 (official build))

This should be a problem with the clickhouse whitelist. Because clickhouse-operator deploys clickhouse with default user networks set to allow only local and nodes in the same cluster to access, when a copy of clickhouse changes IP after reboot for some reason, this time the node resumes after DNS resolution out of IP may exist cache In this case, the IP resolved by the node after recovery may be cached and may not be consistent with the IP resolved by the host in the whitelist, thus being specifically accessed. I have tried to set disable_internal_dns_cache to 1, but this did not alleviate the problem

This is the conclusion of my analysis, and the following errors are misleading. 2021.10.14 14:13:09.834745 [ 243 ] {} <Error> ServerErrorHandler: Code: 516, e.displayText() = DB::Exception: default: Authentication failed: password is incorrect or there is no user with such name, Stack trace (when copying this message, always include the lines below):

I don't know if my analysis is right

czhfe avatar Oct 15 '21 09:10 czhfe 

2021.10.26 15:29:41.794500 [ 130 ] {} <Error> cdn_data.http_access_data.DirectoryMonitor: Code: 516, e.displayText() = DB::Exception: Received from chi-vxclickhouse-clickhouse-1-0:9000. DB::Exception: default: Authentication failed: password is incorrect or there is no user with such name. Stack trace:

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x123fd060 in /usr/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0xa23c3fd in /usr/bin/clickhouse
2. ? @ 0xef774e4 in /usr/bin/clickhouse
3. DB::TCPHandler::receiveHello() @ 0xfb55d8a in /usr/bin/clickhouse
4. DB::TCPHandler::runImpl() @ 0xfb598b9 in /usr/bin/clickhouse
5. DB::TCPHandler::run() @ 0xfb5af00 in /usr/bin/clickhouse
6. Poco::Net::TCPServerConnection::start() @ 0x1231ae8b in /usr/bin/clickhouse
7. Poco::Net::TCPServerDispatcher::run() @ 0x1231b31b in /usr/bin/clickhouse
8. Poco::PooledThread::run() @ 0x12499e46 in /usr/bin/clickhouse
9. Poco::ThreadImpl::runnableEntry(void*) @ 0x12495240 in /usr/bin/clickhouse
10. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
11. __clone @ 0x122103 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
, Stack trace (when copying this message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x123fd060 in /usr/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0xa23c3fd in /usr/bin/clickhouse
2. DB::readException(DB::ReadBuffer&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0xa27c199 in /usr/bin/clickhouse
3. DB::Connection::receiveException() @ 0xfab87cf in /usr/bin/clickhouse
4. DB::Connection::receiveHello() @ 0xfab8ad6 in /usr/bin/clickhouse
5. DB::Connection::connect(DB::ConnectionTimeouts const&) @ 0xfabbdb8 in /usr/bin/clickhouse
6. DB::Connection::forceConnected(DB::ConnectionTimeouts const&) @ 0xfabc4cb in /usr/bin/clickhouse
7. DB::ConnectionPool::get(DB::ConnectionTimeouts const&, DB::Settings const*, bool) @ 0xa36eb82 in /usr/bin/clickhouse
8. DB::StorageDistributedDirectoryMonitor::processFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, CurrentMetrics::Increment&) @ 0xf82d416 in /usr/bin/clickhouse
9. DB::StorageDistributedDirectoryMonitor::run() @ 0xf8307c5 in /usr/bin/clickhouse
10. DB::BackgroundSchedulePoolTaskInfo::execute() @ 0xef92cf9 in /usr/bin/clickhouse
11. DB::BackgroundSchedulePool::threadFunction() @ 0xef93182 in /usr/bin/clickhouse
12. ? @ 0xef932b2 in /usr/bin/clickhouse
13. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xa26afd7 in /usr/bin/clickhouse
14. ? @ 0xa269513 in /usr/bin/clickhouse
15. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
16. __clone @ 0x122103 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
 (version 20.6.7.4 (official build))

Not sure if this is related to your error, but the message i get is same.

I am getting the above error, when executing queries on clickhouse client. the application is also not able to query data because of this error.

I started with clickhouse:21.3 then tried using with latest clickhouse image, and then saw some change which wraps over the exact message so went back to clickhouse:20.6.7.4 which was the version before that change was introduced, to see if i could get a better exception in the log, but no changes in error.

I have two environments the only difference between them is one has firewall enabled and other has firewall disabled.

the firewall disabled environment works fine, but the environment with firewall enabled always returns this error.

I have opened the following ports in the firewall (obtained from kubectl get svc)

  • 8123
  • 9000
  • 9009
  • 9010
  • 9440
  • 8888

but still things did not work. I am getting the error.

If I disable the firewall then things work.

If I disable firewall and re-enable firewall after a few mins then things work. I am guessing, things are cached somewhere.
Its already late here, will check again tomorrow if the cache expires with firewall still enabled.

So, what can be done to make it work in the environment with firewall.

Am I missing any other port to be enabled in firewall.

or is this something entirely different?

anabarasan avatar Oct 26 '21 16:10 anabarasan

Try to use dns cache, the k8s dns have some problem.

Is this caused by kubernetes dns resolution?

yes,you can try

Sallery-X avatar Nov 30 '21 08:11 Sallery-X