Tracer: errors silently ignored

[martin-henz]

Some errors are silently ignored in stepper. Is validation happening before evaluation in the stepper? Examples of programs whose execution is prevented but no error is reported:

function f(x, x) {}

const x = 1; const x = 2;

[CATISNOTSODIUM]

After investigating this issue, it turns out that the CSE machine also has the same issue.

I suspect that all preprocessing errors are ignored before dispatching to different runners. This includes lexing errors, parsing errors, and other relevant compile-time errors.

// From src/runner/index.ts
export async function sourceFilesRunner(
  filesInput: FileGetter,
  entrypointFilePath: string,
  context: Context,
  options: RecursivePartial<IOptions> = {}
): Promise<{
  result: Result
  verboseErrors: boolean
}> {
  const preprocessResult = await preprocessFileImports(
    filesInput,
    entrypointFilePath,
    context,
    options
  )

  if (!preprocessResult.ok) {
    return {
      result: { status: 'error' },
      verboseErrors: preprocessResult.verboseErrors
    }
  }
// (code omitted ... The code basically dispatches to different runners: substitutor (stepper), CSE machine, and Native)

As you can see from the code snippet above, if preprocessResult.ok === false, the function will split out error before dispatching to runner. Since different runners have different ways of displaying error, I don't think it's correct to split out preprocessed error this early.

Potential solution

1. Defer preprocessed errors Preprocessed errors are passed through context (right? I'm not quite sure about this). So, instead of returning error before dispatching, we implement preprocessed error handling separately for every runner instead. Since I'm not very mature with the codebase, I'm not sure if this solution will break other parts or not.
2. Keep the behavior unchanged. Personally, I don't think it's very beneficial to display preprocessed errors in non-native runners (Stepper and CSE), since students can still investigate those errors by running the code in the standard source.