Growing memory usage in web_protocol.py

Open ShamariYoti opened this issue 9 months ago • 1 comments

Describe the bug

I've been seeing a memory leak on one of our services, when running tracemalloc, one of the patterns I can see is that the memory usage in web_protocol.py keeps growing when being compared against the first snapshot. See below.

I'm wondering if it's a known issue that web_protocol.py isn't cleaning up ?

*** top 10 stats ***
/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py:380: size=37.5 KiB (+2760 B), count=521 (+48), average=74 B
/usr/local/lib/python3.10/tracemalloc.py:423: size=1024 B (+1024 B), count=13 (+13), average=79 B
/usr/local/lib/python3.10/tracemalloc.py:560: size=608 B (+608 B), count=5 (+5), average=122 B
/usr/local/lib/python3.10/asyncio/base_events.py:438: size=1487 B (+504 B), count=12 (+3), average=124 B
/usr/local/lib/python3.10/tracemalloc.py:315: size=416 B (+416 B), count=8 (+8), average=52 B
/usr/local/lib/python3.10/site-packages/prometheus_client/multiprocess.py:148: size=2304 B (-336 B), count=12 (-2), average=192 B
/usr/local/lib/python3.10/tracemalloc.py:558: size=280 B (+224 B), count=5 (+4), average=56 B
/usr/local/lib/python3.10/site-packages/aiohttp/web_app.py:569: size=904 B (-168 B), count=2 (-1), average=452 B
/usr/local/lib/python3.10/abc.py:123: size=76.4 KiB (-63 B), count=936 (-1), average=84 B
/app/mtcnn_server/app/profiler.py:9: size=448 B (+32 B), count=2 (+1), average=224 B
*** top 10 stats ***
/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py:380: size=38.2 KiB (+3450 B), count=533 (+60), average=73 B
/usr/local/lib/python3.10/tracemalloc.py:423: size=1176 B (+1176 B), count=16 (+16), average=74 B
/usr/local/lib/python3.10/tracemalloc.py:560: size=656 B (+656 B), count=6 (+6), average=109 B
/usr/local/lib/python3.10/tracemalloc.py:315: size=520 B (+520 B), count=10 (+10), average=52 B
/usr/local/lib/python3.10/site-packages/prometheus_client/multiprocess.py:148: size=2304 B (-336 B), count=12 (-2), average=192 B
/usr/local/lib/python3.10/asyncio/base_events.py:438: size=1319 B (+336 B), count=11 (+2), average=120 B
/usr/local/lib/python3.10/tracemalloc.py:558: size=336 B (+280 B), count=6 (+5), average=56 B
/app/mtcnn_server/app/profiler.py:9: size=480 B (+64 B), count=2 (+1), average=240 B
/usr/local/lib/python3.10/abc.py:123: size=76.4 KiB (-63 B), count=936 (-1), average=84 B
<frozen importlib._bootstrap_external>:672: size=85.2 KiB (+0 B), count=916 (+0), average=95 B
*** top 10 stats ***
/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py:380: size=38.9 KiB (+4140 B), count=545 (+72), average=73 B
/usr/local/lib/python3.10/tracemalloc.py:423: size=1328 B (+1328 B), count=19 (+19), average=70 B
/usr/local/lib/python3.10/tracemalloc.py:560: size=704 B (+704 B), count=7 (+7), average=101 B
/usr/local/lib/python3.10/tracemalloc.py:315: size=624 B (+624 B), count=12 (+12), average=52 B
/usr/local/lib/python3.10/asyncio/base_events.py:438: size=1487 B (+504 B), count=12 (+3), average=124 B
/usr/local/lib/python3.10/site-packages/prometheus_client/multiprocess.py:148: size=2304 B (-336 B), count=12 (-2), average=192 B
/usr/local/lib/python3.10/tracemalloc.py:558: size=392 B (+336 B), count=7 (+6), average=56 B
/usr/local/lib/python3.10/site-packages/aiohttp/web_app.py:569: size=904 B (-168 B), count=2 (-1), average=452 B
/app/mtcnn_server/app/profiler.py:9: size=480 B (+64 B), count=2 (+1), average=240 B
/usr/local/lib/python3.10/abc.py:123: size=76.4 KiB (-63 B), count=936 (-1), average=84 B
*** top 10 stats ***
/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py:380: size=39.5 KiB (+4830 B), count=557 (+84), average=73 B
/usr/local/lib/python3.10/tracemalloc.py:423: size=1480 B (+1480 B), count=22 (+22), average=67 B
/usr/local/lib/python3.10/tracemalloc.py:560: size=752 B (+752 B), count=8 (+8), average=94 B
/usr/local/lib/python3.10/tracemalloc.py:315: size=728 B (+728 B), count=14 (+14), average=52 B
/usr/local/lib/python3.10/asyncio/base_events.py:438: size=1487 B (+504 B), count=12 (+3), average=124 B
/usr/local/lib/python3.10/tracemalloc.py:558: size=448 B (+392 B), count=8 (+7), average=56 B
/usr/local/lib/python3.10/site-packages/prometheus_client/multiprocess.py:148: size=2304 B (-336 B), count=12 (-2), average=192 B
/usr/local/lib/python3.10/site-packages/aiohttp/web_app.py:569: size=904 B (-168 B), count=2 (-1), average=452 B
/app/mtcnn_server/app/profiler.py:9: size=480 B (+64 B), count=2 (+1), average=240 B
/usr/local/lib/python3.10/abc.py:123: size=76.4 KiB (-63 B), count=936 (-1), average=84 B
*** top 10 stats ***
/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py:380: size=40.2 KiB (+5520 B), count=569 (+96), average=72 B
/usr/local/lib/python3.10/tracemalloc.py:423: size=1632 B (+1632 B), count=25 (+25), average=65 B
/usr/local/lib/python3.10/tracemalloc.py:315: size=832 B (+832 B), count=16 (+16), average=52 B
/usr/local/lib/python3.10/tracemalloc.py:560: size=800 B (+800 B), count=9 (+9), average=89 B
/usr/local/lib/python3.10/asyncio/base_events.py:438: size=1487 B (+504 B), count=12 (+3), average=124 B
/usr/local/lib/python3.10/tracemalloc.py:558: size=504 B (+448 B), count=9 (+8), average=56 B
/usr/local/lib/python3.10/site-packages/prometheus_client/multiprocess.py:148: size=2304 B (-336 B), count=12 (-2), average=192 B
/usr/local/lib/python3.10/site-packages/aiohttp/web_app.py:569: size=904 B (-168 B), count=2 (-1), average=452 B
/app/mtcnn_server/app/profiler.py:9: size=480 B (+64 B), count=2 (+1), average=240 B
/usr/local/lib/python3.10/abc.py:123: size=76.4 KiB (-63 B), count=936 (-1), average=84 B
*** top 10 stats ***
/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py:380: size=40.9 KiB (+6210 B), count=581 (+108), average=72 B
/usr/local/lib/python3.10/tracemalloc.py:423: size=1784 B (+1784 B), count=28 (+28), average=64 B
/usr/local/lib/python3.10/tracemalloc.py:315: size=936 B (+936 B), count=18 (+18), average=52 B
/usr/local/lib/python3.10/tracemalloc.py:560: size=848 B (+848 B), count=10 (+10), average=85 B
/usr/local/lib/python3.10/asyncio/base_events.py:438: size=1487 B (+504 B), count=12 (+3), average=124 B
/usr/local/lib/python3.10/tracemalloc.py:558: size=560 B (+504 B), count=10 (+9), average=56 B
/usr/local/lib/python3.10/site-packages/prometheus_client/multiprocess.py:148: size=2304 B (-336 B), count=12 (-2), average=192 B
/usr/local/lib/python3.10/site-packages/aiohttp/web_app.py:569: size=904 B (-168 B), count=2 (-1), average=452 B
/app/mtcnn_server/app/profiler.py:9: size=544 B (+128 B), count=2 (+1), average=272 B
/usr/local/lib/python3.10/abc.py:123: size=76.4 KiB (-63 B), count=936 (-1), average=84 B
*** top 10 stats ***
/usr/local/lib/python3.10/linecache.py:137: size=529 KiB (+529 KiB), count=5311 (+5311), average=102 B
/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py:380: size=42.2 KiB (+7590 B), count=605 (+132), average=71 B
/usr/local/lib/python3.10/tracemalloc.py:423: size=1936 B (+1936 B), count=31 (+31), average=62 B
/usr/local/lib/python3.10/sre_compile.py:804: size=3056 B (+1672 B), count=9 (+3), average=340 B
/usr/local/lib/python3.10/tracemalloc.py:535: size=1240 B (+1240 B), count=3 (+3), average=413 B
/usr/local/lib/python3.10/tracemalloc.py:315: size=1040 B (+1040 B), count=20 (+20), average=52 B
/usr/local/lib/python3.10/tracemalloc.py:447: size=896 B (+896 B), count=2 (+2), average=448 B
/usr/local/lib/python3.10/tracemalloc.py:248: size=832 B (+832 B), count=1 (+1), average=832 B
/app/mtcnn_server/app/profiler.py:38: size=832 B (+832 B), count=1 (+1), average=832 B
/usr/local/lib/python3.10/sre_parse.py:199: size=0 B (-832 B), count=0 (-1)

The traceback for that line is

  File "/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py", line 380
    messages, upgraded, tail = self._request_parser.feed_data(data)

To Reproduce

I think from a simple application like this will show the growing memory usage in web_protocol.py. I was just sending images to the /image enpdoint.

from aiohttp import web
import gc
import asyncio
import tracemalloc
from time import time

import objgraph

import tracemalloc


# list to store memory snapshots
snaps = []


def snapshot():
    snaps.append(tracemalloc.take_snapshot())


def display_stats():
    stats = snaps[0].statistics('filename')
    print("\n*** top 5 stats grouped by filename ***")
    for s in stats[:5]:
        print(f"\n{str(s)}")


def compare():
    first = snaps[0]
    for snapshot in snaps[1:]:
        stats = snapshot.compare_to(first, 'lineno')
        print("\n*** top 10 stats ***")
        for s in stats[:10]:
            print(f"\n{str(s)}")


def print_trace():
    # pick the last saved snapshot, filter noise
    snapshot = snaps[-1].filter_traces((
        tracemalloc.Filter(False, "<frozen importlib._bootstrap>"),
        tracemalloc.Filter(False, "<frozen importlib._bootstrap_external>"),
        tracemalloc.Filter(False, "<unknown>"),
    ))
    largest = snapshot.statistics("traceback")[0]

    print(f"\n*** Trace for largest memory block - ({largest.count} blocks, {largest.size/1024} Kb) ***")
    for largestt in largest.traceback.format():
        print(f"\n{str(largestt)}")

async def hanlder(request):
    print(f'read request')
    req = await request.json()
    return web.Response(text="Request has been receieved")

async def on_startup(app) -> None:
    asyncio.create_task(show_memory())

async def show_memory():
    print('start tracing memory')

    tracemalloc.start(10)
    start = tracemalloc.take_snapshot()

    snapshot_num = 1
    while True:
        for _ in range(10):
            await asyncio.sleep(2)

            gc.collect()
            snapshot()

        display_stats()
        compare()
        print_trace()

my_web_app = web.Application()
my_web_app.router.add_route('POST', '/image', hanlder)
my_web_app.on_startup.append(on_startup)
web.run_app(my_web_app)

Expected behavior

That the memory being used in web_protocol.py gets cleaned up.

Logs/tracebacks

File "/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py", line 380
    messages, upgraded, tail = self._request_parser.feed_data(data)

Python Version

$ 3.10.12

aiohttp Version

3.11.14

multidict Version

6.1.0

propcache Version

0.2.1

yarl Version

1.18.3

OS

Ubuntu 22.04.5 LTS

Related component

Server

Additional context

No response

Code of Conduct

[x] I agree to follow the aio-libs Code of Conduct

Apr 01 '25 17:04 ShamariYoti

The line in that traceback is just storing partial data received. Given that you're requesting it to read the full data into memory, it's not surprising that there's an increase. It should clear up after the requests are completed though.

Apr 01 '25 23:04 Dreamsorcerer