SystemError: Handle is not initialized. in _weakrefset.py still happens

[uvermin]

Hello, I've experienced a problem similar to the one in this closed issue. I'm using the latest version of IronPython (2.7.7), on a windows10 64 bit.

I'm calling to python from C#, in a multi-threaded environment. My python code uses a lot of isinstance calls, and some of them (about once every 10,000 times) raise either a "SystemError: Handle is not initialized." or a "ValueError: Index was out of range. Must be non-negative and less than the size of the collection.\nParameter name: index".

By catching the exception in python I was able to locate the source to the same lines in _weakrefset.py, as described here:

On "Lib\_weakrefset.py"

35 :class WeakSet(object):
....
68 :    def __contains__(self, item):
69 :        try:
**70*:           wr = ref(item)      # <- here is raised "ValueError"

Exception ("Index was out of range")

71 :        except TypeError:
72 :            return False
**73*:       return wr in self.data  # <- here is raised "SystemError"

Exception ("Handle is not initialized")

I've tried using the script given in the original issue to recreate the problem, but that didn't work. The C# code I'm using is open-source, and I don't know C# enough to understand where exactly my python is being called and in what context. Please, If someone could help me recreate the problem, I will upload a detailed script that raises the same exceptions. Thanks!

[slozier]

If you could provide an example to reproduce the issue that would be helpful.

[kunom]

@uvermin: Alternatively, it would be great if you could provide us with the underlying CLR stack trace information (use the -X:ExceptionDetail and -X:ShowClrExceptions interpreter startup flags).

[andygraham]

In my mail archives I have an IronPython-users email interchange with Andres Sommerhoff from April 2015 who suffered something exactly like this with those two exceptions arising from repeated use of isinstance() and posted in the thread you reference. The email subject is "weakref random "SystemError" and "ValueError" exception (affecting _weakrefset.py and abc.py)".

It smells like some sort of “unsynchronised acess by different threads to an object problem ” and at the time I suggested a very hacky workaround that worked for his problem and might help you till the real problem is found. It retries till a proper value is found.

  def __contains__(self, item):
       while 1:
           try:
               try:
                   wr = ref(item)
               except TypeError:
                   return False
               return wr in self.data
           except Exception:
               pass

which he refined to

    def __contains__(self, item):
        while 1:
            try:
                try:
                    wr = ref(item) 
                except TypeError:
                    return False
                return wr in self.data
            except (SystemError, ValueError): #i.e: Don't catch KeyboardInterrupt, MemoryError, etc
                try: i+=1   
                except: i=0       # i declared here. Don't want waste resource outside... (not Pythonic, I know)
                if i==10: raise   # Retry max 10 times and re-raise 

[kunom]

On my machine, I cannot reproduce any exception from the code mentioned above, not even when running one million iterations.

My changes at #1198 should at least have dealt with the "SystemError: Handle is not initialized" error, regardless of missing synchronization, that's why am interested in the CLR stack trace. If you search the source code for SafeHandle or GCHandle instances, then you end up at very few places, one of them being WeakHandle.

In addition, I wrote back then:

Please note that while my patch changes (and hopefully improves) some aspects in _weakref.cs, I still do think that _weakref.cs has several critical issues: There is absolutely no locking on the internal data structures, which makes me think that there could be various runtime issues in a multithreaded environment.