Performance issues reading many columns/rows simultaneously

[daviessm]

First off, I don't think this is the fault of rust-oracle. It seems to be either OCI or the way ODPI-C uses OCI but I'd appreciate your opinion.

I have an application which, using many CPU threads, reads all data from all tables in a database schema to generate a SHA256 checksum of each table. I would expect the CPU to be maxing out with the SHA256 functions but when profiling the application most of the time seems to be taken up with allocating memory for table data and freeing it after use.

I've made a small reproduction program at https://github.com/daviessm/rust_oracle_performance which demonstrates the issue. Although it discards all the data rather than hashing it, I would hope for much better performance. The flamegraph at https://raw.githubusercontent.com/daviessm/rust_oracle_performance/master/flamegraph.svg shows 28% of the runtime is in dpiGen__release and 31% is in dpiConn_newVar.

To run the program, use the parameters -d for database connection string (e.g. 127.0.0.1/orcl), -u for username and -p for password. On the first run it'll create a table called test1 which contains 50 columns and 2,000,000 rows of VARCHAR2 data, then reads from it using one thread for each CPU on the client. Subsequent runs will use the existing table.