jaydebeapi icon indicating copy to clipboard operation
jaydebeapi copied to clipboard

use jaydebeapi to execute and fetchall cost a lot of time

Open xu136090331 opened this issue 6 years ago • 2 comments

use python cx_Oracle to execute and fetchall cost : 4.3s use jaydebeapi execute and fetchall cost: 658s I noticed in the source code of jaydebeapi that fetchall and fetchmany is made up of fetchone,is that right and efficient?

    def fetchone(self):
        if not self._rs:
            raise Error()
        if not self._rs.next():
            return None
        row = []
        for col in range(1, self._meta.getColumnCount() + 1):
            sqltype = self._meta.getColumnType(col)
            converter = self._converters.get(sqltype, _unknownSqlTypeConverter)
            v = converter(self._rs, col)
            row.append(v)
        return tuple(row)

    def fetchmany(self, size=None):
        if not self._rs:
            raise Error()
        if size is None:
            size = self.arraysize
        # TODO: handle SQLException if not supported by db
        self._rs.setFetchSize(size)
        rows = []
        row = None
        for i in range(size):
            row = self.fetchone()
            if row is None:
                break
            else:
                rows.append(row)
        # reset fetch size
        if row:
            # TODO: handle SQLException if not supported by db
            self._rs.setFetchSize(0)
        return rows

    def fetchall(self):
        rows = []
        while True:
            row = self.fetchone()
            if row is None:
                break
            else:
                rows.append(row)
        return rows

xu136090331 avatar Aug 29 '19 02:08 xu136090331

This can be mostly mitigated/optimized in the downstream application by explicitly setting the fetch size before calling fetchall, like this:

cursor._rs.setFetSize(2000)
allRecords = cursor.fetchall()

Using an oracle db, this bumped my performance up by over 100x

datastrategies6 avatar Dec 09 '21 02:12 datastrategies6