String interpolation causes un-collectable memory leaks.

[terminal-cs]

Area of Cosmos - What area of Cosmos are we dealing with?

Strings/Memory

Expected Behaviour - What do you think that should happen?

Old data should be able to be collected by Heap.Collect();

Actual Behaviour - What unexpectedly happens?

Data is leaked and cannot be freed at all.

Reproduction - How did you get this error to appear?

Create a new string over and over again like so: This shouldn't happen, and it does not seem to effect string concatination. The code below does not cause leaks:

Version - Were you using the User Kit or Dev Kit? And what User Kit version or Dev Kit commit (Cosmos, IL2CPU, X#)?

Latest devkit

[quajak]

Additional detail is that the issue does not occur when the string interpolation is moved into a separate method.

[terminal-cs]

Additional detail is that the issue does not occur when the string interpolation is moved into a separate method.

No it still does, we used concatination there aswell

[GoldenretriverYT]

Even normal concatination also seems to have this problem despite your results, which might explain my OS constantly leaking memory over time, especially when doing stuff with strings.

[terminal-cs]

Even normal concatination also seems to have this problem despite your results, which might explain my OS constantly leaking memory over time, especially when doing stuff with strings.

There are several leaks, but I have tested, concatination does not leak when heap.collect() is called

[quajak]

Yeah, there seems to be an issue with string more generally. It seems to be a lot worse when using string interpolation.

[GoldenretriverYT]

Even normal concatination also seems to have this problem despite your results, which might explain my OS constantly leaking memory over time, especially when doing stuff with strings.

There are several leaks, but I have tested, concatination does not leak when heap.collect() is called

Concatination definitely does leak as you can see in the screenshot though. I also tried without foreach and even using an array of primitives, so it isn't an issue with foreach or structs either. As quajak said, this is probably an issue with strings as a whole.

[terminal-cs]

Even normal concatination also seems to have this problem despite your results, which might explain my OS constantly leaking memory over time, especially when doing stuff with strings.

There are several leaks, but I have tested, concatination does not leak when heap.collect() is called

Concatination definitely does leak as you can see in the screenshot though. I also tried without foreach and even using an array of primitives, so it isn't an issue with foreach or structs either. As quajak said, this is probably an issue with strings as a whole.

There are other factors that you have included in your code that can be leaking. I have tested and verified concatination does not leak at all

[GoldenretriverYT]

Your example does not do a lot of concatenation unlike mine, which does around 1024 concatenations per round. There are no other factors that could be leaking there considering I tested it with primitive types and without foreach too.

Just for you, I removed the method and inlined the code, still leaks. I even got rid of the array, and it still leaks.

There is basically nothing else that can leak anymore, unless Heap.Collect is causing leaks itself.

[terminal-cs]

Your example does not do a lot of concatenation unlike mine, which does around 1024 concatenations per round. There are no other factors that could be leaking there considering I tested it with primitive types and without foreach too.

Just for you, I removed the method and inlined the code, still leaks. I even got rid of the array, and it still leaks.

There is basically nothing else that can leak anymore, unless Heap.Collect is causing leaks itself.

The consol scrolling could be leaking

[GoldenretriverYT]

The foreach is gone, as I already have said. And console scrolling definitely isn't the leaking part, but I can get rid of that too, I will test that

[GoldenretriverYT]

Still leaks.

[terminal-cs]

Still leaks.

Then i don't know, There was zero memory leaks when I was testing. When did you update cosmos last

[GoldenretriverYT]

Its the latest devkit, too. Your test, as I have already said, has done a lot fewer concatenations and as Quajak said, its most likely that interpolation causes heavier memory leaks but that doesn't mean concatenation doesnt cause any at all.

[GoldenretriverYT]

Your example maybe doesnt leak any at all because you never assign a string to anything, you just pass it as a parameter which might be stored in the stack and not the heap, it might be that but I am unsure

[GoldenretriverYT]

Have done further tests:

No leaks:

Heap.Collect();
Console.CursorLeft = 0;
Console.CursorTop = 0;
Console.WriteLine("Memory: " + GCImplementation.GetUsedRAM());

No leaks:

string o = "Memory: " + GCImplementation.GetUsedRAM();

Console.CursorLeft = 0;
Console.CursorTop = 0;
Console.WriteLine("Memory: " + o);

No leaks:

string o = "Memory: ";
o += GCImplementation.GetUsedRAM();

Console.CursorLeft = 0;
Console.CursorTop = 0;
Console.WriteLine("Memory: " + o);

No leaks:

string o = "Memory: ";
o += GCImplementation.GetUsedRAM();

for(int i = 0; i < 1; i++) {
    o += i;
}

Heap.Collect();
Console.CursorLeft = 0;
Console.CursorTop = 0;
Console.WriteLine(o);

No leaks:

string o = "Memory: ";
o += GCImplementation.GetUsedRAM();

for(int i = 0; i < 1; i++) {
    o += i + "|" + i*2;
}

Heap.Collect();
Console.CursorLeft = 0;
Console.CursorTop = 0;
Console.WriteLine(o);

Leaks memory (i < 4 does not leak; i<8 doesnt; i<32 doesnt; i<64 & i<128 does):

string o = "Memory: ";
o += GCImplementation.GetUsedRAM();

for(int i = 0; i < 128; i++) {
    o += i + "|" + i*2;
}

Heap.Collect();
Console.CursorLeft = 0;
Console.CursorTop = 0;
Console.WriteLine(o);

I do not know the internals of how strings work in Cosmos, but could it maybe be that the string has an padded buffer so it doesn't always cause new allocations for small concatenations but when it gets too large it has to expand the buffer but it doesn't free the other one? The interpolation one might not be related to this then.

[quajak]

FYI using HeapSmall.GetAllocatedObjectCount() gives you a more accurate measurement of how many elements are currently allocated. The issue is that GetUsedRAM uses the number of allocated pages and a single string leaking will not trigger a new page being allocated each time.

[GoldenretriverYT]

Tried that and with i<16 it still does not increase but with i<128 it does for some reason

[GoldenretriverYT]

So I tested just adding a single character at a time with x repetitions, and this starts to happen at exactly 247 repetitions; 246 does not leak at all.

public int rep = 0;

protected override void BeforeRun() {
    Console.Clear();
    string s = Console.ReadLine();
    int i = int.Parse(s);
    rep = i;
}

protected override void Run() {
    string o = "";

    for (int i = 0; i < rep; i++) {
        o += "i";
    }

    Heap.Collect();
    Console.CursorLeft = 0;
    Console.CursorTop = 0;
    Console.WriteLine("AllocObjects: " + HeapSmall.GetAllocatedObjectCount() + "       " + o.Length);
}

[quajak]

I was able to look into it a bit yesterday and confirmed the issue. I found there is an issue with the garbage collector and objects allocated using HeapLarge. Fixing this did not resolve this issue fully. @GoldenretriverYT if you are interested in helping fix this issue, please reach out to me on discord and then we can co-ordinate the effort.

Due to the design of heap into being split into heap small and large, HeapSmall.GetAllocatedObjectCount will undercount the number of actually allocated objects since it does not know about any large objects.

[terminal-cs]

I was able to look into it a bit yesterday and confirmed the issue. I found there is an issue with the garbage collector and objects allocated using HeapLarge. Fixing this did not resolve this issue fully. @GoldenretriverYT if you are interested in helping fix this issue, please reach out to me on discord and then we can co-ordinate the effort.

Due to the design of heap into being split into heap small and large, HeapSmall.GetAllocatedObjectCount will undercount the number of actually allocated objects since it does not know about any large objects.

Why do we separate heaps anyways? Isn't is just supposed to be a continuous region of memory for any objects of any size?

[quajak]

It is explained a bit in https://cosmosos.github.io/articles/Kernel/MemoryManagement.html but could be clearer. The idea is that we want to efficiently allocate objects way smaller than a single page (eg allocating one page of 4096 bytes for an object of 16 bytes), so individual pages in the heap are either used for small objects and managed by HeapSmall or larger objects, managed by HeapLarge

[GoldenretriverYT]

I was able to look into it a bit yesterday and confirmed the issue. I found there is an issue with the garbage collector and objects allocated using HeapLarge. Fixing this did not resolve this issue fully. @GoldenretriverYT if you are interested in helping fix this issue, please reach out to me on discord and then we can co-ordinate the effort.

Due to the design of heap into being split into heap small and large, HeapSmall.GetAllocatedObjectCount will undercount the number of actually allocated objects since it does not know about any large objects.

Ah no sorry, my understanding of that stuff like GC & Memory Managment is too low to be any help at that

[ascpixi]

I was able to look into it a bit yesterday and confirmed the issue. I found there is an issue with the garbage collector and objects allocated using HeapLarge. Fixing this did not resolve this issue fully. @GoldenretriverYT if you are interested in helping fix this issue, please reach out to me on discord and then we can co-ordinate the effort.

Due to the design of heap into being split into heap small and large, HeapSmall.GetAllocatedObjectCount will undercount the number of actually allocated objects since it does not know about any large objects.

Has the cause of issue in HeapLarge been identified, or does this require additional debugging?

[AsertCreator]

note: c# compiler actually uses some builders and utilities to make string interpolation. these are not StringBuilders. did we plug those?

[terminal-cs]

note: c# compiler actually uses some builders and utilities to make string interpolation. these are not StringBuilders. did we plug those?

Almost all of the stream related helpers work, including that

[zarlo]

note: c# compiler actually uses some builders and utilities to make string interpolation. these are not StringBuilders. did we plug those?

all the builder and utilities that dont use reflection should work with out plugs

[quajak]

Ah no sorry, my understanding of that stuff like GC & Memory Managment is too low to be any help at that

I am happy to provide guidance and honestly the code for GC + Memory Management is pretty small in Cosmos.

Has the cause of issue in HeapLarge been identified, or does this require additional debugging?

The issue with HeapLarge should already be fixed in https://github.com/CosmosOS/Cosmos/pull/2604 but there is an additional issue in HeapSmall which needs to be debugged and fixed.