databricks-sql-go icon indicating copy to clipboard operation
databricks-sql-go copied to clipboard

Added heartbeat to cloud download

Open rcypher-databricks opened this issue 1 year ago • 2 comments

While downloading cloud files a heartbeat will query operation status at regular intervals to keep the connection alive. Implemented heartbeat type which takes an instance of driver.Pinger and calls it at a regular interval. Implemented statusGetter type which implements driver.Pinger by calling GetOperationStatus() Updated NewBatchLoader() to take an argument of type driver.Pinger and updated batchLoader to instantiate a heartbeat instance using the pinger. Updated NewRows() to create an instance of statusGetter and updated the various constructor functions to take a diver.Pinger argument. Added interface fetcher.Overwatch. NewConcurrentFetcher() now takes an Overwatch argument. An instance of Overwatch will be started/stopped in conjunction with fetcher start/stop.

rcypher-databricks avatar Oct 13 '23 00:10 rcypher-databricks

Core review: can you be triple sure that we're not leaking go routines?

Feature review: why is this needed? how can the session timeout while we're downloading results?

During cloud fetch, the client downloads cloud files without the involvement of DBSQL or cluster so the server may shut down due to inactivity. Heartbeat thread would keep it alive.

yunbodeng-db avatar Nov 21 '23 23:11 yunbodeng-db

But as I said before. The operation/session are open until the client reads the entire result set or manually closes. So the server shouldn't be shutting down due to inactivity.

rcypher-databricks avatar Nov 22 '23 19:11 rcypher-databricks