holoclean Replace to_sql() with copy_expert() to improve performance while savi…

Replace to_sql() with copy_expert() to improve performance while savi…

Open fatangare opened this issue 5 years ago • 1 comments

Improving performance while saving data in Postgresql in case of large dataset.

to_sql() method is slow and takes times to save data in Postgres. It is replaced with copy_exprt() to save data in Postgres tables fast.

Oct 17 '19 11:10 fatangare

I also added one commit to parallelize compute_norm_cond_entropy_corr() method.

With single thread on hospital data, it takes 3.11 sec. With 6 threads, it takes 1.05 sec on my mac (16GB RAM)

Oct 17 '19 12:10 fatangare