jaydebeapi
jaydebeapi copied to clipboard
use jaydebeapi to execute and fetchall cost a lot of time
use python cx_Oracle to execute and fetchall cost : 4.3s use jaydebeapi execute and fetchall cost: 658s I noticed in the source code of jaydebeapi that fetchall and fetchmany is made up of fetchone,is that right and efficient?
def fetchone(self):
if not self._rs:
raise Error()
if not self._rs.next():
return None
row = []
for col in range(1, self._meta.getColumnCount() + 1):
sqltype = self._meta.getColumnType(col)
converter = self._converters.get(sqltype, _unknownSqlTypeConverter)
v = converter(self._rs, col)
row.append(v)
return tuple(row)
def fetchmany(self, size=None):
if not self._rs:
raise Error()
if size is None:
size = self.arraysize
# TODO: handle SQLException if not supported by db
self._rs.setFetchSize(size)
rows = []
row = None
for i in range(size):
row = self.fetchone()
if row is None:
break
else:
rows.append(row)
# reset fetch size
if row:
# TODO: handle SQLException if not supported by db
self._rs.setFetchSize(0)
return rows
def fetchall(self):
rows = []
while True:
row = self.fetchone()
if row is None:
break
else:
rows.append(row)
return rows
This can be mostly mitigated/optimized in the downstream application by explicitly setting the fetch size before calling fetchall, like this:
cursor._rs.setFetSize(2000)
allRecords = cursor.fetchall()
Using an oracle db, this bumped my performance up by over 100x