evidence Optimize `run sources`

Improve the speed of running sources and writing parquet, improve logging on source runs, reduce memory footprint if possible.

[ ] test cloud SPA mode before shipping

Sep 30 '24 17:09 mcrascal

perhaps put logging behind debug flag

Sep 30 '24 17:09 archiewood

Think it would be really helpful if npm run sources logs includes under each source query 1) the total time of the query exec, and 2) and where possible depending on the connector the query exec time from db stats (possibly in debug mode only)

This would help with optimizing source queries, as well as understand where there are bottlenecks during deployments/builds

Sep 30 '24 19:09 ud3sh

Think it would be really helpful if npm run sources logs includes under each source query 1) the total time of the query exec, and 2) and where possible depending on the connector the query exec time from db stats (possibly in debug mode only)

This would help with optimizing source queries, as well as understand where there are bottlenecks during deployments/builds

Archie opened an issue for this https://github.com/evidence-dev/evidence/issues/2629 //cc @csjh

Oct 08 '24 18:10 ud3sh

Adding a note here for better error handling when query result is too large, or region is not set correctly (bigquery example)

https://evidencedev.slack.com/archives/C023LRB9Z40/p1727121470531939?thread_ts=1727101387.250899&cid=C023LRB9Z40

Oct 16 '24 16:10 hughess