byzer-lang icon indicating copy to clipboard operation
byzer-lang copied to clipboard

[Engine][2.1.0][Enhance] Add retry mechanism for async request callback

Open allwefantasy opened this issue 3 years ago • 0 comments

The request to MLSQL engine can be async and once the request finished then we will callback the caller with the parameter callback.

However, the implementation of callback for now is not robust. We can enhance the feature with retry mechanism. The rest controller should add a new parameter retries, default is -1 and this means no retry.

// streaming.rest.RestController
def query = {
        if (paramAsBoolean("async", false)) {
          JobManager.asyncRun(sparkSession, jobInfo, () => {
            try {
              ScriptSQLExec.parse(param("sql"), context,
                skipInclude = paramAsBoolean("skipInclude", false),
                skipAuth = paramAsBoolean("skipAuth", true),
                skipPhysicalJob = paramAsBoolean("skipPhysicalJob", false),
                skipGrammarValidate = paramAsBoolean("skipGrammarValidate", true))

              outputResult = getScriptResult(context, sparkSession)
               // here is where we add retry mechanism.
              htp.post(new Url(param("callback")),
                Map("stat" -> s"""succeeded""",
                  "res" -> outputResult,
                  "jobInfo" -> JSONTool.toJsonStr(jobInfo)))

allwefantasy avatar Apr 14 '21 01:04 allwefantasy