Broken Korean text in sandpack-react test error messages

[dmhuh]

Bug report

Packages affected

• [ ] sandpack-client
• [x] sandpack-react

Description of the problem

When a test case fails and the test description is in Korean, the error message displays broken characters, as below:

What were you doing when the problem occurred?

What steps can we take to reproduce the problem?

1. Visit the Sandpack Preset playground on the Sandpack documentation site.

2. Change the test description in the code to Korean. (e.g., 덧셈의 교환 법칙)

3. Modify the test case to intentionally fail. Here is an example modification.

import { add } from './add';

describe('add', () => {
  test('덧셈의 교환 법칙', () => {
    expect(add(1, 2)).toBe(add(3, 1));  // This change will cause the test to fail
  });
});

4. Run the test to observe the error message with broken characters.

Link to sandbox: -

Your Environment

Software	Name/Version
Sandpack-client version	x
Sandpack-react version	2.1.10
Browser	Chrome
Operating System	MacOS

[kangju2000]

After investigating this issue, I've observed that the same error occurs in CodeSandbox. This suggests that the problem needs to be addressed in the codesandbox-client repository.

I attempted to debug locally by logging to the console, but encountered difficulties running the project. As a result, I'm sharing the code sections that I suspect may be related to the issue:

1. Remote Module Fetching

Location: https://github.com/codesandbox/codesandbox-client/blob/master/packages/sandpack-core/src/manager.ts#L139-L156

async function fetchRemoteModule(url: string): Promise<IRemoteModuleResult> {
  try {
    const r = await fetchWithRetries(url);

    if (!r.ok) {
      throw new Error(`Fetching ESModule return error status ${r.status}`);
    }

    const content = await r.text();
    return {
      url: r.url,
      content,
    };
  } catch (err) {
    console.error(err);
    throw new ModuleNotFoundError(url, true);
  }
}

Potential solution

The fetchWithRetries function accepts a second argument for fetch's requestInit. We could add encoding-related logic there.

2. File Evaluation

Location: https://github.com/codesandbox/codesandbox-client/blob/master/packages/app/src/sandbox/eval/transpilers/babel/worker/evaluate.ts#L197-L220

export function evaluateFromPath(
  fs: any,
  BFSRequire: Function,
  path: string,
  currentPath: string,
  availablePlugins: Object,
  availablePresets: Object
) {
  const resolvedPath = patchedResolve().sync(path, {
    filename: currentPath,
    extensions: ['.cjs', '.js', '.json'],
  });

  const code = fs.readFileSync(resolvedPath).toString();

  return evaluate(
    fs,
    BFSRequire,
    code,
    resolvedPath,
    availablePlugins,
    availablePresets
  );
}

Potential solution

Encoding issues might arise during file reading. We could use the detect-character-encoding library (or implement custom logic) to identify the character encoding, then use iconv-lite to convert it to UTF-8.

It seems that identifying the correct locations for encoding handling is key to resolving this issue. Your insights on this would be greatly appreciated. @danilowoz 🙏