Atem18
High Memory Consumption
Hi @Atem18,
opened ne issue because not definitely sure if the same root cause.
Could you provide how you are profiling and the minimal code for reproduction? My current setup is guppy3 and the following code (readme, public api only):
https://pastebin.com/pdu5Kxzm
Results after about a 20 minutes run:
first:
Partition of a set of 761710 objects. Total size = 76471286 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 167518 22 17767591 23 17767591 23 str
1 146885 19 9673200 13 27440791 36 tuple
2 31820 4 4852848 6 32293639 42 list
3 26729 4 4765545 6 37059184 48 types.CodeType
4 13259 2 4272904 6 41332088 54 dict (no owner)
5 141435 19 3985788 5 45317876 59 int
6 52951 7 3940826 5 49258702 64 bytes
7 3597 0 3588208 5 52846910 69 type
8 24096 3 3277056 4 56123966 73 function
9 27418 4 3070816 4 59194782 77 asttokens.util.Token
<1349 more rows. Type e.g. '_.more' to view.>
after 20 minutes:
Partition of a set of 761735 objects. Total size = 76488074 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 167527 22 17768268 23 17768268 23 str
1 146885 19 9673200 13 27441468 36 tuple
2 31822 4 4853136 6 32294604 42 list
3 26729 4 4780648 6 37075252 48 types.CodeType
4 13257 2 4272608 6 41347860 54 dict (no owner)
5 141438 19 3985876 5 45333736 59 int
6 52951 7 3940826 5 49274562 64 bytes
7 3597 0 3588208 5 52862770 69 type
8 24096 3 3277056 4 56139826 73 function
9 27418 4 3070816 4 59210642 77 asttokens.util.Token
<1349 more rows. Type e.g. '_.more' to view.>
So I wasn't able to reproduce the memory issue.
Dear @Kiessar, the minimal code is this one : https://github.com/Atem18/kraky#websocket
The memory seems to jump really fast over 5 minutes without any reasons.
The best would be if I could put an APM to trace the whole process.
Dear @Atem18, do you use the private API in your test? I don't use it (public only) and can't reproduce the error.
My Next step would be to setup a fresh virtual environment and see if it behaves the same.
@Kiessar Of course yes I am using both public and private API as in the example.
@Atem18 fresh environment + private api use, no rising memory consumption after 1 hour runtime. Which OS do you run on? Would you mind sharing your dependencies including versions?
The only things I changed was removing, the prints for the "results" and adding memory consumption print outs.
@Kiessar, Ubuntu with only kraky as deps.
And the memory was increasing suddently after multiples hours.
@Atem18 could you try to add like 10 other pairs to the example and check if the memory consumption happens faster then?
@Kiessar Do you have any issue with memory ? If not, I propose to close this.