psycopg2-ctypes icon indicating copy to clipboard operation
psycopg2-ctypes copied to clipboard

Memory leak when pulling more records than itersize

Open d33tah opened this issue 10 years ago • 0 comments

Here's how I called a query:

    cursor = connection.cursor()                                                   
    sql = """                                                                      
    SELECT  "wikispy_edit"."id",                                                   
            "wikispy_edit"."wikipedia_edit_id",                                    
            "wikispy_edit"."title",                                                
            "wikispy_edit"."ip",                                                   
            "wikispy_edit"."wiki_id",                                              
            "wikispy_rdns"."rdns",                                                 
            "wikispy_wiki"."language",                                             
            "wikispy_wiki"."domain"                                                
    FROM "wikispy_edit"                                                            
    INNER JOIN "wikispy_rdns"                                                      
        ON ( "wikispy_edit"."ip" = "wikispy_rdns"."ip" )                           
    INNER JOIN "wikispy_wiki"                                                      
        ON ( "wikispy_edit"."wiki_id" = "wikispy_wiki"."id" )                      
    WHERE                                                                          
            REVERSE("wikispy_rdns"."rdns") LIKE REVERSE(%s)                        
        AND                                                                        
            "wikispy_wiki"."name" = %s                                             
    LIMIT 2001                                                                     
    """                                                                            
    params = ['%' + rdns, wikiname]                                                
    #if sql_scans_table(sql, "wikispy_edit", params):                              
    #    raise RuntimeError("The query is too big.")                               
    # http://stackoverflow.com/a/2679222/1091116                                   
    cursor.execute(sql, params)                                                    
    description = [x[0] for x in cursor.description]                               
    for row in cursor:                                                             
        yielded = dict(zip(description, row))                                      
        yield yielded                                                              

If this gives more than 2000 rows, which is itersize defined in psycopg2ct/_impl/cursor.py, the Python process eats all the memory possible and probably goes into some infinite loop. Raising works as a workaround.

d33tah avatar Apr 17 '15 22:04 d33tah