pylakey
Very high RAM usage in 0.24.3
Using v0.24.3, just importing aiotdlib gets htop to indicate 950M of RAM usage.
For comparison, doing the same experiment with v0.22.0, htop indicated 91408 bytes used.
This does not look like RAM that can be reclaimed, since upgrading my aiotdlib-based project on my tiny VPS gets it OOM-killed on startup systematically.
For the record, here's a profiling attempt using tracemalloc
import linecache
import tracemalloc
def display_top(snapshot, key_type='lineno', limit=20):
snapshot = snapshot.filter_traces((
tracemalloc.Filter(False, "<frozen importlib._bootstrap>"),
tracemalloc.Filter(False, "<unknown>"),
))
top_stats = snapshot.statistics(key_type)
print("Top %s lines" % limit)
for index, stat in enumerate(top_stats[:limit], 1):
frame = stat.traceback[0]
print("#%s: %s:%s: %.1f KiB"
% (index, frame, frame.lineno, stat.size / 1024))
line = linecache.getline(frame.filename, frame.lineno).strip()
if line:
print(' %s' % line)
other = top_stats[limit:]
if other:
size = sum(stat.size for stat in other)
print("%s other: %.1f KiB" % (len(other), size / 1024))
total = sum(stat.size for stat in top_stats)
print("Total allocated size: %.1f KiB" % (total / 1024))
tracemalloc.start()
import aiotdlib
snapshot = tracemalloc.take_snapshot()
display_top(snapshot)
Output:
Top 20 lines
#1: /tmp/venv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py:674:674: 135197.0 KiB
proxy = _PydanticWeakRef(v)
#2: /tmp/venv/lib/python3.11/site-packages/pydantic/_internal/_core_utils.py:568:568: 121114.0 KiB
return _validate_core_schema(schema)
#3: /tmp/venv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py:677:677: 50377.6 KiB
result[k] = proxy
#4: /tmp/venv/lib/python3.11/site-packages/pydantic/_internal/_core_utils.py:200:200: 16238.3 KiB
schema = self._schema_type_to_method[schema['type']](schema.copy(), f)
#5: <frozen importlib._bootstrap_external>:729:729: 11174.0 KiB
#6: /tmp/venv/lib/python3.11/site-packages/pydantic/plugin/_schema_validator.py:50:50: 9680.9 KiB
return SchemaValidator(schema, config)
#7: <frozen abc>:106:106: 5682.9 KiB
#8: /tmp/venv/lib/python3.11/site-packages/pydantic/_internal/_core_utils.py:336:336: 4923.1 KiB
replaced_field = v.copy()
#9: /tmp/venv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py:561:561: 4552.7 KiB
cls.__pydantic_serializer__ = SchemaSerializer(schema, core_config)
#10: /tmp/venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:2246:2246: 4117.1 KiB
def json_schema_update_func(
#11: /tmp/venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:1177:1177: 2850.3 KiB
js_annotation_functions=[get_json_schema_update_func(json_schema_updates, json_schema_extra)]
#12: <frozen abc>:123:123: 2355.0 KiB
#13: /tmp/venv/lib/python3.11/site-packages/pydantic/_internal/_core_metadata.py:87:87: 1524.6 KiB
metadata = {k: v for k, v in metadata.items() if v is not None}
#14: /tmp/venv/lib/python3.11/site-packages/pydantic/_internal/_core_utils.py:99:99: 1476.2 KiB
type_ref = f'{module_name}.{qualname}:{id(origin)}'
#15: /usr/lib/python3.11/copyreg.py:105:105: 1335.9 KiB
return cls.__new__(cls, *args)
#16: /tmp/venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:1172:1172: 1266.8 KiB
json_schema_updates = {k: v for k, v in json_schema_updates.items() if v is not None}
#17: /tmp/venv/lib/python3.11/site-packages/pydantic/_internal/_core_metadata.py:82:82: 1108.5 KiB
pydantic_js_functions=js_functions or [],
#18: /tmp/venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:544:544: 805.7 KiB
metadata = build_metadata_dict(js_functions=[partial(modify_model_json_schema, cls=cls, title=title)])
#19: <frozen abc>:107:107: 701.8 KiB
#20: /tmp/venv/lib/python3.11/site-packages/pydantic/_internal/_config.py:173:173: 673.9 KiB
core_config = core_schema.CoreConfig(
13488 other: 20259.2 KiB
Total allocated size: 397415.6 KiB
I don't know if that's useful :shrug:
Some more profiling attempts
With guppy:
Partition of a set of 4716517 objects. Total size = 482965223 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 770037 16 192286688 40 192286688 40 dict (no owner)
1 1730499 37 138439920 29 330726608 68 dict of
pydantic._internal._model_construction._PydanticWeakR
ef
2 1730499 37 96907944 20 427634552 89
pydantic._internal._model_construction._PydanticWeakRe
f
3 133093 3 8830608 2 436465160 90 list
4 70915 2 8002048 2 444467208 92 str
5 28939 1 4398728 1 448865936 93 function
6 61251 1 4004448 1 452870384 94 tuple
7 2297 0 3877336 1 456747720 95 pydantic._internal._model_construction.ModelMetaclass
8 10428 0 3608528 1 460356248 95 types.CodeType
9 9463 0 2044008 0 462400256 96 pydantic.fields.FieldInfo
with pympler:
types | # objects | total size
========================================================= | =========== | ============
dict | 778998 | 187.23 MB
pydantic._internal._model_construction._PydanticWeakRef | 1730499 | 92.42 MB
list | 133102 | 8.63 MB
str | 40681 | 5.35 MB
pydantic._internal._model_construction.ModelMetaclass | 2297 | 3.70 MB
code | 9298 | 3.14 MB
function (json_schema_update_func) | 20269 | 2.94 MB
tuple | 35927 | 2.06 MB
pydantic.fields.FieldInfo | 9463 | 1.95 MB
cell | 41459 | 1.58 MB
weakref.ReferenceType | 17967 | 1.37 MB
pydantic_core._pydantic_core.SchemaValidator | 2296 | 1.37 MB
type | 1280 | 1.35 MB
set | 4902 | 1.25 MB
collections.OrderedDict | 2296 | 1.06 MB
Pydantic v2 release notes mention "performance improvements" I wonder if such RAM usage is expected, or is because we misuse pydantic, or is because of a bug/memleak in pydantic. Anyway that sucks!
Yes, I also noticed that import time is very high now as well as RAM consumption. I think the only way is to test with other pydantic versions or wait fixes from pydantic team.
fwiw it looks like the Pydantic team is looking at further optimizations and using this project as one of their evaluation targets:
https://github.com/pydantic/pydantic/discussions/6748
I'm seeing the same issues with load time, almost 20 seconds - half from the model_rebuild() calls and half from the initial model initialization :(
After some more research, I was able to decrease the start time by 90% with the following changes:
- Add
defer_build=True,to theBaseObjectconstructor https://github.com/pylakey/aiotdlib/blob/50dcf77f83bac1fe01c51d1f65b8abfc983794d1/aiotdlib/api/types/base.py#L39-L45 - Remove the
model_rebuild()from the types template and regenerate the models https://github.com/pylakey/aiotdlib/blob/50dcf77f83bac1fe01c51d1f65b8abfc983794d1/aiotdlib_generator/templates/types_template.py.jinja2#L45-L56
I'm not at all confident that this is a permanent fix and all it really does is delay the incurred RAM/speed impact from pydantic 2, but I'm only using a small fraction of the available 2000ish Telegram APIs so this seems to be a net positive so far. I don't have a memory profiler, but I assume the speed improvement I'm seeing is coupled with less RAM usage since it's no longer building all of the models.
.model_rebuild() is necessary for self-referenced models and cross-references. I think we can't just get rid of .model_rebuild() call. It will cause issues in some non-standard usage cases.
Will defer_build solve this issue and force rebuild model in runtime? If yes, we can try
I think we can wait for the release of Pydantic 2.11 to make things significantly better.
https://github.com/pydantic/pydantic/discussions/6748#discussioncomment-12010610