byzer-lang
byzer-lang copied to clipboard
[Engine][2.1.0][Enhance] Add retry mechanism for async request callback
The request to MLSQL engine can be async and once the request finished then we will callback the caller with the parameter callback
.
However, the implementation of callback for now is not robust. We can enhance the feature with retry mechanism. The rest controller should add a new parameter retries
, default is -1 and this means no retry.
// streaming.rest.RestController
def query = {
if (paramAsBoolean("async", false)) {
JobManager.asyncRun(sparkSession, jobInfo, () => {
try {
ScriptSQLExec.parse(param("sql"), context,
skipInclude = paramAsBoolean("skipInclude", false),
skipAuth = paramAsBoolean("skipAuth", true),
skipPhysicalJob = paramAsBoolean("skipPhysicalJob", false),
skipGrammarValidate = paramAsBoolean("skipGrammarValidate", true))
outputResult = getScriptResult(context, sparkSession)
// here is where we add retry mechanism.
htp.post(new Url(param("callback")),
Map("stat" -> s"""succeeded""",
"res" -> outputResult,
"jobInfo" -> JSONTool.toJsonStr(jobInfo)))