psycopg2-ctypes
psycopg2-ctypes copied to clipboard
Memory leak when pulling more records than itersize
Here's how I called a query:
cursor = connection.cursor()
sql = """
SELECT "wikispy_edit"."id",
"wikispy_edit"."wikipedia_edit_id",
"wikispy_edit"."title",
"wikispy_edit"."ip",
"wikispy_edit"."wiki_id",
"wikispy_rdns"."rdns",
"wikispy_wiki"."language",
"wikispy_wiki"."domain"
FROM "wikispy_edit"
INNER JOIN "wikispy_rdns"
ON ( "wikispy_edit"."ip" = "wikispy_rdns"."ip" )
INNER JOIN "wikispy_wiki"
ON ( "wikispy_edit"."wiki_id" = "wikispy_wiki"."id" )
WHERE
REVERSE("wikispy_rdns"."rdns") LIKE REVERSE(%s)
AND
"wikispy_wiki"."name" = %s
LIMIT 2001
"""
params = ['%' + rdns, wikiname]
#if sql_scans_table(sql, "wikispy_edit", params):
# raise RuntimeError("The query is too big.")
# http://stackoverflow.com/a/2679222/1091116
cursor.execute(sql, params)
description = [x[0] for x in cursor.description]
for row in cursor:
yielded = dict(zip(description, row))
yield yielded
If this gives more than 2000 rows, which is itersize defined in psycopg2ct/_impl/cursor.py, the Python process eats all the memory possible and probably goes into some infinite loop. Raising works as a workaround.