fetchall function sometime go for ever with no limitation.

[thinksource]

I use pypyodbc library, and get out of memory error for Issue #2042.

The problem is in pypyodbc function:

   def fetchall(self):
        if not self.connection:
            self.close()

        rows = []
        while True:
            row = self.fetchone()
            if row is None:   //// It seems the row always is not None.
                break
            rows.append(row)
        return rows

This function I get:

C:\Dataload\Dataload Executables\AO3Dataload\src>pypy mmtmain.py -E PYPY_GC_MAX_DELTA=4.0GB

Getting all related products info... RPython traceback: File "rpython_jit_metainterp_warmspot.c", line 1300, in ll_portal_runnerUnsi gned_Bool_pypy_interpreter File "rpython_jit_metainterp_warmstate.c", line 4795, in maybe_compile_and_run star_5 File "rpython_jit_metainterp_warmstate.c", line 10053, in execute_assembler__s tar_2_2 File "rpython_jit_metainterp_compile.c", line 5694, in DoneWithThisFrameDescrR ef_handle_fail out of memory: couldn't allocate the next arena

Even if the row after the list, it is still can be None type.

[braian87b]

Hi, I am not completely understand what it could be the problem... was python 2 or 3? you have 64bit windows and more than 4gb ram?

This is my approach to fetchall:

def create_new_connection():
    return pyodbc.connect(connection_string, timeout=10, unicode_results=True, readonly=False)

def fetchall_query(sqlSentence, parameters=(), cursor=None):
    no_transaction = cursor is None
    if cursor is None: cursor = db_connection.cursor()
    if debug: print "-- %r %r" % (sqlSentence, parameters)
    cursor.execute(sqlSentence, parameters)
    #if not cursor.description: raise DatabaseNoColumnDescriptionsError("'cursor.description' is None in 'raw_query'")
    rows = [dict(zip([column[0] for column in cursor.description], row)) for row in cursor.fetchall()]
    if no_transaction: cursor.close()
    return rows

Of course it will load entire results in a List of Dict's but will be enought and easy to work with small results set < 10k rows, or even < 50k rows if you have enought memory.

I leave it here if others need it...

[thinksource]

python 2 with 64bit windows 7 and 8gb ram. I used pypy replace python 2.7