graph-notebook icon indicating copy to clipboard operation
graph-notebook copied to clipboard

Allow %load_ids magic to expose load status for each bulk loader job

Open triggan opened this issue 1 year ago • 0 comments

Is your feature request related to a problem? Please describe. The %load_ids magic currently fetches a full list of all bulk load IDs that were previously submitted to a writer instance. However, to fetch the status of each load job, a user would then have to use individual calls to %load_status to provide any further details for each job. If a user is submitting multiple bulk load jobs (particularly when they are queuing multiple jobs), there's no easy way to see which jobs are queued and which ones are executing/completed.

Describe the solution you'd like Add a --details flag to %load_ids that results in a table of all bulk load jobs that have been submitted and the status for each (ex: LOAD_COMPETED, LOAD_QUEUED, LOAD_IN_PROGRESS, etc.). Potentially also provide a --limit X parameter to only return the status for the last X bulk load jobs, as some users may have many hundreds of jobs that have previously been submitted.

Describe alternatives you've considered An alternative would be to provide an example of using a notebook variables to take the output of %load_ids and pipe that into multiple cells of %load_status, but that would be cumbersome if one needed to review status for dozens of issued bulk load jobs.

Additional context N/A

triggan avatar Aug 09 '22 18:08 triggan