Adding items to large Lists sometimes breaks

[dyfer]

Environment

• SuperCollider version: 3.15-dev, 3.14, 3.13
• Operating system: macOS
• Other details (Qt version, audio driver, etc.):

Steps to reproduce

(
var allItems;
var numItems = 2000; // lower numbers, like 300, will likely not cause the error

allItems = numItems.collect({ Dictionary() }).as(List);

// ~allItems = allItems; // !!!!! uncommenting this gets rid of the error...

allItems.do({|thisDict, inc|
	var newDict;
	
	newDict = Dictionary.newFrom((\inc: inc));
	allItems.add(newDict); // add to the List
	
	allItems.do({|t|
		if(t.isKindOf(Dictionary).not) {
			"%: detected a non-dictionary entry after add: %".format(inc, t.asCompileString).error;
		};
	});
	
	allItems.remove(thisDict); // current item from the list
	allItems.do({|t|
		if(t.isKindOf(Dictionary).not) {
			"%: detected a non-dictionary entry after remove: %".format(inc, t.asCompileString).error;
		};
	});
});
)

Expected vs. actual behavior

Expected: list should be populated with Dictionaries only.

Actual: with large lists, other objects leak into the List. The code above posts the message with the object in question being rendered as a compile string:

755: detected a non-dictionary entry after add: [nil, nil, 'inc', 755]

(To clarify: I'm posting an error message, it's not an actual error being thrown).

This is an issue that #7180 works around for the test runner.

While I understand that manipulating the List while it's being iterated may be a bad practice... BUT it seems to work unless we hit some memory size. Is this a garbage collection issue?

Also, as noted in the reproducer, assigning the list to an environment variable gets rid of the error... Which I find rather mysterious.

[JordanHendersonMusic]

Does this happen on 3.14?

[dyfer]

Yes, it's an old bug, I've seen it in the test runner ever since we started running all the tests (albeit infrequently). To be sure, I double checked and the reproducer is valid in both 3.13 and 3.14.

[JordanHendersonMusic]

Thanks, I will try and figure out what's going on here when I get some time. Redoing kitchen floor this week!

[JordanHendersonMusic]

I haven't been able to find the cause, but doing allItems.copy.do({|thisDict, inc|... at the top seem to fix the issue.

[dyfer]

Sure, it makes sense. When we don't touch the List that we iterate on, the issue won't happen.

But the problem is that the original issue is only triggered when (apparently) a certain memory threshold is surpassed.

[JordanHendersonMusic]

It also doesn't break when it is an array, which is odd! I can't figure out why the list causes it to break.

[dyfer]

It also doesn't break when it is an array, which is odd! I can't figure out why the list causes it to break.

Array and List behave differently. Array may return a new object when adding new elements, so one has to always do array = array.add(item). A List will always add items to the same object, so there's no need to reassign it.

[JordanHendersonMusic]

I checked that, the list still breaks when the array isn't resized. The add and remove means the array is only increased by 1 element, so it never reaches the threshold to allocate.

I'm quite stumped by this as I don't see how this could be a GC issue if it works with array, but not list.

[dyfer]

Thanks for looking into this. I'm not sure how relevant recreating this with an Array is and I don't know how GC works exactly...

To me the mysterious part is that this issue seems to be somehow related to the total memory used (either by the contents of the List, or by the interpreter as a whole, not sure). If the Dictionaries added to the list are bigger, the issue will be triggered after a different number of iterations.