Issues with twinBASIC enumeration implementation

[WaynePhillipsEA]

When enumerating the dictionary content with For-Each, the twinBASIC hacks are causing mem corruption, leading to crashes.

For example, with just this simple enumerator I can see that a Collection enumerator vtable is replaced (as expected, OK), but also the adjacent field member which is used internally to maintain a linked list of live enumerators for the collection is also getting overwritten with something:

        For Each v In d
        Next v

The offending code in VBA-FastDictionary is this:

        With Mem.Common(0)
            #If TWINBASIC Then
                .saP.pvData = ObjPtr(DictEnum) + ptrSize
            #Else
                .saP.pvData = ObjPtr(DictEnum) + nextItemOffset
            #End If
            Mem.RPtr(0) = VarPtr(Vars.Keys(i))                               ' --> tB internal linked list gets overwritten here
            .saP.pvData = .dPtr
        End With

This causes crashes, particularly when stepping through the code, since at that point there are multiple open enumerators against the collection (since the debugger breifly enumerates it for display in the IDE), and the clobbered linked list gets manipulated/corrupted.

[WaynePhillipsEA]

BTW, if you provide me with a list of what you need twinBASIC to expose in order to remove all the hacks, I will consider them, as currently these hacks look to rely on a lot of implementation detail that could easily change in tB at any point.

[cristianbuse]

Hi @WaynePhillipsEA

It would actually be great to remove all these hacks. I did not want to bother you with this since you have more important things to do. However, I was planning to, at some point.

The VBA implementation basically uses the Collection's ability to enumerate items at speeds far superior of what can be achieved in native non-compiled PCode. However, speed is not an issue in TB so don't really need this.

Probably the best approach would be to just have a custom enumerator class that is just for TB users - I know TB can do that e.g. this.

My biggest problem is the constraint to keep all the code limited to the Dictionary.cls module, which is purely a VBA distribution problem, not a TB problem. This of course is very restrictive as I cannot declare a class inside a .cls module. It would be so much easier to just use a .twin class module instead.

I read twinbasic/lang-design/issues/6 but I have no idea if there are newer discussions than that or if anything was implemented since then. I am only using the TB Collection because it exposes the enumerator but it would be super cool if TB would have some kind of generic approach to enumerate a class internal array so it can be used in a For Each loop. Is there any documentation that I should read before I bother you any further?

[mburns08109]

Just FYI, I'd love to have this working "right" in tB...would use this in place of scripting.dictionary in a heartbeat. I don't suppose a tB-specific edition could be done? (which would also remove the .cls/.twin issue)

[cristianbuse]

@mburns08109 I could definitely do a tB-specific edition. Most of the code is VBA-specific anyway.