scio icon indicating copy to clipboard operation
scio copied to clipboard

Add option for SMB joins to fall back to regular joins

Open clairemcginty opened this issue 4 months ago • 0 comments

Add an option, either pipeline option or API method param, to instruct Scio to fall back to a non-SMB join if it can't perform an SMB join (due to incompatible hash types, missing metadata, etc).

SMB does all this validation at job submission time -- after sc.run() but before the job is submitted to dataflow -- so we should be able to catch exceptions at that time.

clairemcginty avatar Feb 21 '24 14:02 clairemcginty