datajoint-python icon indicating copy to clipboard operation
datajoint-python copied to clipboard

add mechanism to check for processing progress at schema level

Open ttngu207 opened this issue 1 year ago • 6 comments

Add new method progress for the schema object, returning the populate progress for all imported/computed tables for a given schema (also accounts for reserved, error, ignore jobs)

ttngu207 avatar Aug 11 '22 16:08 ttngu207

@ttngu207 Perhaps we should consider utilizing populate's progress? You should be able to call this to determine the table populate status.

guzman-raphael avatar Aug 11 '22 20:08 guzman-raphael

@guzman-raphael , the problem with table.progress() is that it is not taking into account entries in the jobs table (e.g. reserved, error, ignore).

ttngu207 avatar Aug 11 '22 20:08 ttngu207

Right, just thinking how we can report progress consistently at the table level and the schema level.

guzman-raphael avatar Aug 11 '22 20:08 guzman-raphael

Yeah, ideally we'd fix the .progress() at the table level to account for reserve, error and ignore jobs, but that would break backward compatibility.

ttngu207 avatar Aug 11 '22 21:08 ttngu207

We can introduce an option to return as pandas too but have it default to the current behavior.

guzman-raphael avatar Aug 11 '22 21:08 guzman-raphael

You mean for the new schema.progress() to return a query expression instead of pandas dataframe?

ttngu207 avatar Aug 11 '22 22:08 ttngu207