userver icon indicating copy to clipboard operation
userver copied to clipboard
verified publisher icon userver-framework

При появлении исключения TryHandleError в http сервисе объем используемой памяти увеличивается на сотни МБ

Open taoga opened this issue 5 months ago • 3 comments

Версия userver: 2.8 Сервис создан на базе примера hello_service. ОС ubuntu 22.04. Сборка через cmake.

cmake .. make

Фрагмент static_config.yaml: ... http-client: # Component to do HTTP requests fs-task-processor: fs-task-processor #user-agent: 'fbs-service 1.0' # Set 'User-Agent' header to 'fbs-service 1.0'.

dns-client: # Asynchronous DNS component fs-task-processor: fs-task-processor ... server: listener: port: $server-port task_processor: main-task-processor ... default_task_processor: main-task-processor # Task processor in which components start.

task_processors: grpc-blocking-task-processor: # For blocking gRPC channel creation worker_threads: 8 thread_name: grpc-worker main-task-processor: # For non-blocking operations worker_threads: 16 fs-task-processor: # For blocking filesystem operations worker_threads: 8

В процессе тестирования вызывается ручка, описанная в конфиге как: handler-cpa-documents-imports: path: /cpa/documents/imports # Registering handler by URL method: GET # It will only reply to POST requests. task_processor: main-task-processor # Run it on CPU bound task processor Время ответа на запрос от 0.2 до 200 мс.

Фрагмент теста на python: # get_imports_res = requests.get(main_adress + "/cpa/documents/imports", headers={"X-Access-Token":login_res.json()["client"]["accessToken"]}) print('get_imports_res.status_code:', get_imports_res.status_code )

Тест выполняется в цикле. Если тест не прерывать, объем используемой памяти остается примерно на одном уровне 490Мб.

Сообщения при прерывании теста: CTraceback (most recent call last): File "/media/bolsb/4E03FF6C13B51B5D/work_front/temp/test_rest_cl_ntfy3.py", line 655, in test_add_del_notifications() File "/media/bolsb/4E03FF6C13B51B5D/work_front/temp/test_rest_cl_ntfy3.py", line 521, in test_add_del_notifications get_imports_res = requests.get(main_adress + "/cpa/documents/imports", File "/usr/lib/python3/dist-packages/requests/api.py", line 76, in get return request('get', url, params=params, **kwargs) File "/usr/lib/python3/dist-packages/requests/api.py", line 61, in request return session.request(method=method, url=url, **kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 544, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 657, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 700, in urlopen httplib_response = self._make_request( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 446, in _make_request six.raise_from(e, None) File "", line 3, in raise_from File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 441, in _make_request httplib_response = conn.getresponse() File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse response.begin() File "/usr/lib/python3.10/http/client.py", line 318, in begin version, status, reason = self._read_status() File "/usr/lib/python3.10/http/client.py", line 279, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/usr/lib/python3.10/socket.py", line 705, in readinto return self._sock.recv_into(b) KeyboardInterrupt

Сообщение в логе сервиса:

tskv timestamp=2025-07-02T13:58:57.158508 level=INFO module=TryHandleError<const char [12], const userver::v2_8::engine::io::Sockaddr> ( 3rdparty/userver/core/src/engine/io/fd_control.hpp:195 ) task_id=7636AE818000 thread_id=0x00007636B67FF400 stacktrace= 0# userver::v2_8::utils::(anonymous namespace)::CollectTrace(userver::v2_8::utils::TracefulExceptionBase::TraceMode) at /media/bolsb/4E03FF6C13B51B5D/work_front/client-backend/3rdparty/userver/universal/src/utils/traceful_exception.cpp:19\n 1# userver::v2_8::utils::TracefulExceptionBase::Impl::Impl(userver::v2_8::utils::TracefulExceptionBase::TraceMode) at /media/bolsb/4E03FF6C13B51B5D/work_front/client-backend/3rdparty/userver/universal/src/utils/traceful_exception.cpp:25\n 2# userver::v2_8::utils::FastPimpl<userver::v2_8::utils::TracefulExceptionBase::Impl, 160ul, 8ul, false>::FastPimpluserver::v2_8::utils::TracefulExceptionBase::TraceMode&(userver::v2_8::utils::TracefulExceptionBase::TraceMode&) at /media/bolsb/4E03FF6C13B51B5D/work_front/client-backend/3rdparty/userver/universal/include/userver/utils/fast_pimpl.hpp:72\n 3# userver::v2_8::utils::TracefulExceptionBase::TracefulExceptionBase(userver::v2_8::utils::TracefulExceptionBase::TraceMode) at /media/bolsb/4E03FF6C13B51B5D/work_front/client-backend/3rdparty/userver/universal/src/utils/traceful_exception.cpp:59\n 4# userver::v2_8::utils::TracefulException::TracefulExceptionBase(userver::v2_8::utils::TracefulExceptionBase::TraceMode) at /media/bolsb/4E03FF6C13B51B5D/work_front/client-backend/3rdparty/userver/universal/include/userver/utils/traceful_exception.hpp:73\n 5# userver::v2_8::engine::io::IoException::IoException(std::basic_string_view<char, std::char_traits >) at /media/bolsb/4E03FF6C13B51B5D/work_front/client-backend/3rdparty/userver/core/src/engine/io/exception.cpp:15\n 6# userver::v2_8::engine::io::IoSystemError::IoSystemError(std::error_code, std::basic_string_view<char, std::char_traits >) at /media/bolsb/4E03FF6C13B51B5D/work_front/client-backend/3rdparty/userver/core/src/engine/io/exception.cpp:34\n 7# userver::v2_8::engine::io::IoSystemError::IoSystemError(int, std::basic_string_view<char, std::char_traits >) at /media/bolsb/4E03FF6C13B51B5D/work_front/client-backend/3rdparty/userver/core/src/engine/io/exception.cpp:31\n 8# userver::v2_8::engine::io::impl::ErrorMode userver::v2_8::engine::io::impl::Direction::TryHandleError<char const [12], userver::v2_8::engine::io::Sockaddr const>(int, unsigned long, userver::v2_8::engine::io::impl::TransferMode, userver::v2_8::engine::Deadline, char const (&) [12], userver::v2_8::engine::io::Sockaddr const&) at /media/bolsb/4E03FF6C13B51B5D/work_front/client-backend/3rdparty/userver/core/src/engine/io/fd_control.hpp:188\n 9# unsigned long userver::v2_8::engine::io::impl::Direction::PerformIoV<long ()(int, iovec const, int), char [12], userver::v2_8::engine::io::Sockaddr>(userver::v2_8::engine::io::impl::Direction::SingleUserGuard&, long (&&)(int, iovec const, int), iovec*, unsigned long, userver::v2_8::engine::io::impl::TransferMode, userver::v2_8::engine::Deadline, char const (&) [12], userver::v2_8::engine::io::Sockaddr const&) at /media/bolsb/4E03FF6C13B51B5D/work_front/client-backend/3rdparty/userver/core/src/engine/io/fd_control.hpp:239\n10# userver::v2_8::engine::io::Socket::SendAll(iovec const*, unsigned long, userver::v2_8::engine::Deadline) in ./build/client-backend\n11# userver::v2_8::engine::io::Socket::SendAll(userver::v2_8::engine::io::IoData const*, unsigned long, userver::v2_8::engine::Deadline) in ./build/client-backend\n12# userver::v2_8::engine::io::Socket::SendAll(std::initializer_listuserver::v2_8::engine::io::IoData, userver::v2_8::engine::Deadline) in ./build/client-backend\n13# userver::v2_8::engine::io::Socket::WriteAll(std::initializer_listuserver::v2_8::engine::io::IoData, userver::v2_8::engine::Deadline) at /media/bolsb/4E03FF6C13B51B5D/work_front/client-backend/3rdparty/userver/core/include/userver/engine/io/socket.hpp:101\n14# userver::v2_8::server::http::HttpResponse::SetBodyNotStreamed(userver::v2_8::engine::io::RwBase&, userver::v2_8::utils::SmallString<1024ul>&) at /media/bolsb/4E03FF6C13B51B5D/work_front/client-backend/3rdparty/userver/core/src/server/http/http_response.cpp:322\n15# userver::v2_8::server::http::HttpResponse::SendResponse(userver::v2_8::engine::io::RwBase&) at /media/bolsb/4E03FF6C13B51B5D/work_front/client-backend/3rdparty/userver/core/src/server/http/http_response.cpp:296\n16# userver::v2_8::server::net::Connection::SendResponse(userver::v2_8::server::http::HttpRequest&) at /media/bolsb/4E03FF6C13B51B5D/work_front/client-backend/3rdparty/userver/core/src/server/net/connection.cpp:349\n17# userver::v2_8::server::net::Connection::ProcessRequest(std::shared_ptruserver::v2_8::server::http::HttpRequest&&) at /media/bolsb/4E03FF6C13B51B5D/work_front/client-backend/3rdparty/userver/core/src/server/net/connection.cpp:216\n18# userver::v2_8::server::net::Connection::ListenForRequests() at /media/bolsb/4E03FF6C13B51B5D/work_front/client-backend/3rdparty/userver/core/src/server/net/connection.cpp:127\n19# userver::v2_8::server::net::Connection::Process() at /media/bolsb/4E03FF6C13B51B5D/work_front/client-backend/3rdparty/userver/core/src/server/net/connection.cpp:63\n20# userver::v2_8::server::net::ListenerImpl::ProcessConnection(userver::v2_8::engine::io::Socket, userver::v2_8::server::net::PortConfig const&) in ./build/client-backend\n21# auto userver::v2_8::server::net::ListenerImpl::AcceptConnection(userver::v2_8::engine::io::Socket&, userver::v2_8::server::net::PortConfig const&)::{lambda(auto:1, auto:2)#5}::operator()<userver::v2_8::engine::io::Socket, userver::v2_8::utils::FastScopeGuard<userver::v2_8::server::net::ListenerImpl::AcceptConnection(userver::v2_8::engine::io::Socket&, userver::v2_8::server::net::PortConfig const&)::{lambda()#1}> >(userver::v2_8::engine::io::Socket, userver::v2_8::utils::FastScopeGuard<userver::v2_8::server::net::ListenerImpl::AcceptConnection(userver::v2_8::engine::io::Socket&, userver::v2_8::server::net::PortConfig const&)::{lambda()#1}>) const at /media/bolsb/4E03FF6C13B51B5D/work_front/client-backend/3rdparty/userver/core/src/server/net/listener_impl.cpp:89\n22# void std::__invoke_impl<void, userver::v2_8::server::net::ListenerImpl::AcceptConnection(userver::v2_8::engine::io::Socket&, userver::v2_8::server::net::PortConfig const&)::{lambda(auto:1, auto:2)#5}, userver::v2_8::engine::io::Socket, userver::v2_8::utils::FastScopeGuard<userver::v2_8::server::net::ListenerImpl::AcceptConnection(userver::v2_8::engine::io::Socket&, userver::v2_8::server::net::PortConfig const&)::{lambda()#1}> >(std::__invoke_other, userver::v2_8::server::net::ListenerImpl::AcceptConnection(userver::v2_8::engine::io::Socket&, userver::v2_8::server::net::PortConfig const&)::{lambda(auto:1, auto:2)#5}&&, userver::v2_8::engine::io::Socket&&, userver::v2_8::utils::FastScopeGuard<userver::v2_8::server::net::ListenerImpl::AcceptConnection(userver::v2_8::engine::io::Socket&, userver::v2_8::server::net::PortConfig const&)::{lambda()#1}>&&) at /usr/include/c++/11/bits/invoke.h:61\n23# std::__invoke_result<userver::v2_8::server::net::ListenerImpl::AcceptConnection(userver::v2_8::engine::io::Socket&, userver::v2_8::server::net::PortConfig const&)::{lambda(auto:1, auto:2)#5}, userver::v2_8::engine::io::Socket, userver::v2_8::utils::FastScopeGuard<userver::v2_8::server::net::ListenerImpl::AcceptConnection(userver::v2_8::engine::io::Socket&, userver::v2_8::server::net::PortConfig const&)::{lambda()#1}> >::type std::__invoke<userver::v2_8::server::net::ListenerImpl::AcceptConnection(userver::v2_8::engine::io::Socket&, userver::v2_8::server::net::PortConfig const&)::{lambda(auto:1, auto:2)#5}, userver::v2_8::engine::io::Socket, userver::v2_8::utils::FastScopeGuard<userver::v2_8::server::net::ListenerImpl::AcceptConnection(userver::v2_8::engine::io::Socket&, userver::v2_8::server::net::PortConfig const&)::{lambda()#1}> >(userver::v2_8::server::net::ListenerImpl::AcceptConnection(userver::v2_8::engine::io::Socket&, userver::v2_8::server::net::PortConfig const&)::{lambda(auto:1, auto:2)#5}&&, userver::v2_8::engine::io::Socket&&, userver::v2_8::utils::FastScopeGuard<userver::v2_8::server::net::ListenerImpl::AcceptConnection(userver::v2_8::engine::io::Socket&, userver::v2_8::server::net::PortConfig const&)::{lambda()#1}>&&) at /usr/include/c++/11/bits/invoke.h:97\n24# decltype(auto) std::__apply_impl<userver::v2_8::server::net::ListenerImpl::AcceptConnection(userver::v2_8::engine::io::Socket&, userver::v2_8::server::net::PortConfig const&)::{lambda(auto:1, auto:2)#5}, std::tuple<userver::v2_8::engine::io::Socket, userver::v2_8::utils::FastScopeGuard<userver::v2_8::server::net::ListenerImpl::AcceptConnection(userver::v2_8::engine::io::Socket&, userver::v2_8::server::net::PortConfig const&)::{lambda()#1}> >, 0ul, 1ul>(userver::v2_8::server::net::ListenerImpl::AcceptConnection(userver::v2_8::engine::io::Socket&, userver::v2_8::server::net::PortConfig const&)::{lambda(auto:1, auto:2)#5}&&, std::tuple<userver::v2_8::engine::io::Socket, userver::v2_8::utils::FastScopeGuard<userver::v2_8::server::net::ListenerImpl::AcceptConnection(userver::v2_8::engine::io::Socket&, userver::v2_8::server::net::PortConfig const&)::{lambda()#1}> >&&, std::integer_sequence<unsigned long, 0ul, 1ul>) at /usr/include/c++/11/tuple:1855\n25# decltype(auto) std::apply<userver::v2_8::server::net::ListenerImpl::AcceptConnection(userver::v2_8::engine::io::Socket&, userver::v2_8::server::net::PortConfig const&)::{lambda(auto:1, auto:2)#5}, std::tuple<userver::v2_8::engine::io::Socket, userver::v2_8::utils::FastScopeGuard<userver::v2_8::server::net::ListenerImpl::AcceptConnection(userver::v2_8::engine::io::Socket&, userver::v2_8::server::net::PortConfig const&)::{lambda()#1}> > >(userver::v2_8::server::net::ListenerImpl::AcceptConnection(userver::v2_8::engine::io::Socket&, userver::v2_8::server::net::PortConfig const&)::{lambda(auto:1, auto:2)#5}&&, std::tuple<userver::v2_8::engine::io::Socket, userver::v2_8::utils::FastScopeGuard<userver::v2_8::server::net::ListenerImpl::AcceptConnection(userver::v2_8::engine::io::Socket&, userver::v2_8::server::net::PortConfig const&)::{lambda()#1}> >&&) at /usr/include/c++/11/tuple:1867\n26# [start of coroutine]\n text=Direction::PerformIo: Broken pipeError while SendAll to [::ffff:127.0.0.1]:53702, fd=76 (userver::v2_8::engine::io::IoSystemError)

После появления исключения TryHandleError объем используемой памяти увеличивался: 1) 700Мб 2) 900Мб 3) 1,2Гб 4) 1.5Гб 5) 1,7 Гб и т.д.

taoga avatar Jul 02 '25 11:07 taoga

Сделал демонстратор ошибки из примера. Описание в readme

chaotic_service_exp.tar.gz

taoga avatar Jul 03 '25 07:07 taoga

Повторил данную проблему. Автоматический запуск теста воспроизводящего проблему добавил (не нужно Ctrl+C жать). Также сделал патч, который данную проблему решает (для v2.8, хотя и в последующих версиях код не менялся), но требуется проверка. test+patch.zip

SSmolyakov avatar Jul 04 '25 17:07 SSmolyakov

(на английском опишу для охвата аудитории) I reproduced the same issue in more easy way. Just insert code in any your http handler userver::engine::io::IoSystemError ex(123, "Direction::PerformIo"); ex << "Error while "; //(ex << ... << context); //ex << ", fd=" << Fd(); auto log_level = logging::Level::kInfo; LOG(log_level) << ex; Every time you invoke this code, memory will leak about size of binary. This code was taken from TryHandlerError, described above in callstack

alex-aparin avatar Jul 05 '25 07:07 alex-aparin