Failed to load extension: It isn't UTF-8 encoded

[qiweiii]

Describe the bug

after pnpm dev, I got this error from chrome:

To Reproduce

branch: https://github.com/qiweiii/markdown-sticky-notes/tree/migrate-wxt Steps to reproduce the bug using the reproduction:

1. Install dependencies: pnpm i
2. Start dev mode: pnpm dev

Expected behavior

Expected to open in chrome

Screenshots

Environment

System:
    OS: macOS 14.2.1
    CPU: (12) arm64 Apple M2 Max
    Memory: 640.95 MB / 32.00 GB
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 20.9.0 - ~/.nvm/versions/node/v20.9.0/bin/node
    Yarn: 1.22.21 - ~/.nvm/versions/node/v20.9.0/bin/yarn
    npm: 10.1.0 - ~/.nvm/versions/node/v20.9.0/bin/npm
    pnpm: 8.13.1 - ~/.nvm/versions/node/v20.9.0/bin/pnpm
    bun: 1.0.11 - ~/.bun/bin/bun
  Browsers:
    Chrome: 120.0.6099.216
    Safari: 17.2.1
  npmPackages:
    wxt: ^0.14.3 => 0.14.3 

Additional context

I am trying to migrate this old extension to wxt, the branch is wip so may have many problems, any help would be appreciated!

[aklinker1]

Did some googling, I found a few other references to this UTF8 issue.

• https://stackoverflow.com/questions/49979397/chrome-says-my-content-script-isnt-utf-8
• https://github.com/lukehorvat/github-ast-viewer/issues/4

There's probably some non-encoded UTF characters being output that makes Chrome think your file isn't valid. The file was too big to look through myself, so I switched to trying to find the code/modules causing the problem.

After a binary search of commenting things out, I found this has something to do with the @uiw/react-codemirror module. I was able to get it building by commenting it out in the editor component. Then I had to make a few other changes to actually get a working build:

Full patch to get it working

diff --git a/entrypoints/content/Editor.tsx b/entrypoints/content/Editor.tsx
index d5e7ec5..5b28859 100644
--- a/entrypoints/content/Editor.tsx
+++ b/entrypoints/content/Editor.tsx
@@ -1,5 +1,5 @@
 import { useRef } from "react";
-import CodeMirror, { ViewUpdate } from "@uiw/react-codemirror";
+// import CodeMirror, { ViewUpdate } from "@uiw/react-codemirror";
 import { markdown, markdownLanguage } from "@codemirror/lang-markdown";
 import { languages } from "@codemirror/language-data";
 import * as themes from "@uiw/codemirror-themes-all";
@@ -19,7 +19,7 @@ const Editor = (props: Props) => {
   return (
     <div>
       {/* https://github.com/uiwjs/react-codemirror */}
-      <CodeMirror
+      {/* <CodeMirror
         value={props.value}
         extensions={[
           markdown({ base: markdownLanguage, codeLanguages: languages }),
@@ -30,7 +30,7 @@ const Editor = (props: Props) => {
         autoFocus={props.autofocus}
         style={{ fontSize: props.fontSize, fontFamily: props.fontFamily }}
         // TODO: check CodeMirror component has any more stuff need to move here or not
-      />
+      /> */}
     </div>
   );
 };
diff --git a/entrypoints/content/Note.tsx b/entrypoints/content/Note.tsx
index 6e5698a..1a7e9e7 100644
--- a/entrypoints/content/Note.tsx
+++ b/entrypoints/content/Note.tsx
@@ -23,8 +23,8 @@ import { useOutsideClickRef } from "rooks";
 import nightOwl from "react-syntax-highlighter/dist/cjs/styles/prism/night-owl";
 
 import Editor from "./Editor";
-import themes from "../themes";
-import fonts from "../fonts";
+import themes from "@/utils/themes";
+import fonts from "@/utils/fonts";
 import type { Note } from "./storage";
 
 const ITEM_HEIGHT = 20;
diff --git a/entrypoints/options/Preference.tsx b/entrypoints/options/Preference.tsx
index 8a1e663..3af059f 100644
--- a/entrypoints/options/Preference.tsx
+++ b/entrypoints/options/Preference.tsx
@@ -8,8 +8,8 @@ import Select, { SelectChangeEvent } from "@mui/material/Select";
 import Input from "@mui/material/Input";
 import { styled, withStyles } from "@mui/material/styles";
 
-import themes from "../themes";
-import fonts from "../fonts";
+import themes from "@/utils/themes";
+import fonts from "@/utils/fonts";
 
 const ITEM_HEIGHT = 20;
 const ITEM_PADDING_TOP = 5;
diff --git a/entrypoints/fonts.ts b/utils/fonts.ts
similarity index 100%
rename from entrypoints/fonts.ts
rename to utils/fonts.ts
diff --git a/entrypoints/themes.ts b/utils/themes.ts
similarity index 100%
rename from entrypoints/themes.ts
rename to utils/themes.ts
diff --git a/wxt.config.ts b/wxt.config.ts
index 03dae2e..6fa6100 100644
--- a/wxt.config.ts
+++ b/wxt.config.ts
@@ -10,10 +10,10 @@ export default defineConfig({
     short_name: "markdown-sticky-notes",
     name: "Markdown Sticky Notes",
     icons: {
-      "16": "notes16.png",
-      "32": "notes32.png",
-      "64": "notes64.png",
-      "128": "notes128.png",
+      "16": "icon/notes16.png",
+      "32": "icon/notes32.png",
+      "64": "icon/notes64.png",
+      "128": "icon/notes128.png",
     },
     permissions: ["storage", "tabs"],
     web_accessible_resources: [

Obviously removing the editor isn't a solution lol. Instead, I'd recommend you load the editor inside an iframe, and just have the content script insert the iframe. That way you don't have to include @uiw/react-codemirror inside a content script. Maybe that would work-around the error with your dependency.

See WXT's createIframeUi helper: https://wxt.dev/guide/content-script-ui.html#iframe

[qiweiii]

@aklinker1 much appreciated, I will give it a try

[lionelhorn]

@aklinker1

Depending on the wxt.config.ts and vite build options (terser used as minifier or not), I also get these errors.

Would something as simple as this in a vite plugin that can be activated or not solve the issue?

https://github.com/PlasmoHQ/plasmo/blob/6c559127f70d356e3325712e70502b082579f13f/packages/parcel-optimizer-terser/src/to-utf8.ts

[aklinker1]

If one of you could try this function out, that'd be helpful. I'm a little concerned about iterating over every character of the bundle to fix a rare case that doesn't happen for everyone.

You can test this out by adding a custom vite plugin, and calling the function @lionelhorn mentioned on the bundle during the generateBundle hook.

On my phone right now, will add a full example if that's not enough to get you started

[qiweiii]

I will try this solution when I get home!

[Xdy1579883916]

我也遇到了这个问题，首先我尝试关闭vite minify, 编译后chrome能正常加载扩展。 I also encountered this problem. First, I tried to turn off Vite Minify, and after compiling, Chrome was able to load the extension normally.

于是我使用了 terser So I used Terser

vite() {
        return {
            build: {
                minify: "terser",
                terserOptions: {
                    compress: {
                        drop_console: true,
                        drop_debugger: true,
                    },
                },
            }
        }
    }

可以通过这个设置尝试是否能够解决上述问题

You can try this setting to see if it can solve the above problem

[GorvGoyl]

@Xdy1579883916 Terser is significantly slow https://github.com/privatenumber/minification-benchmarks

@qiweiii did you find a solution to this?

[qiweiii]

Yes, I used a custom vite plugin as suggested above

In my code, it's here

[aklinker1]

I'm onboard bringing this into WXT now that there's a working solution, wondering if there's a more performant way. For large extensions, I'd rather we don't iterate through megabytes of strings in JS. Does esbuild have a setting we could use instead?

[aklinker1]

Hmm, looks like esbuild already does this: https://esbuild.github.io/api/#charset

Which makes sense. This was working in dev mode, where esbuild is used by vite. This wasn't working in production builds, which uses rollup. So we actually need to configure rollup... Do they have a setting?

[aklinker1]

Hmm, doesn't seem like it. I guess processing the bundle before it's written to the file system is the way to go...

[GorvGoyl]

For large extensions, I'd rather we don't iterate through megabytes of strings in JS

That's my concern as well. Peaking at @qiweiii's extension code, it seems the culprit is rehype-katex, which is the same for us! I wonder if there's a rehype-katex specific solution (such as encoding only this file) that doesn't affect other files, or if we should ask the rehype-katex author to support UTF-8. cc: @wooorm 👀

[qiweiii]

I added rehype-katex recently for math support in markdown. Previously, the vite plugin was added for @uiw/react-codemirror, so I guess @aklinker1 wants a more general solution

[aklinker1]

Yeah, I'm looking for a more general solution.

I guess for now there's a workaround (https://github.com/wxt-dev/wxt/issues/353#issuecomment-2475525361), and this seems to happen rarely enough that I don't want to add this to all projects for performance reasons. Perhaps Vite 6 and rolldown will use the esbuild approach and default to writing files as ascii instead of utf-8. Then the problem will fix itself.

Until then, we could create a wxt module that adds this workaround's vite plugin and tell people to install it if they run into this error.

[andrewshvv]

Yes, I used a custom vite plugin as suggested above

In my code, it's here

Can confirm it is working for react-markdown, rehype-katex, remark-math setup.

[sgcullen]

EDIT 3: SOLVED - read on to the end for the solution in my case.

I'm now experiencing this issue if I try to load unpacked.

"Could not load file 'content-scripts/...for content script. It isn't UTF-8 encoded.
Could not load manifest."

Edit. I've been trying to get to the bottom of this and after much looking it seems to be related to me migrating from webext-bridge to webest-core/messaging.

Edit 2. Digging much deeper, I've tried to understand why this suddenly started happening. It looks like it ultimately stems from Dexie - the IndexDB wrapper I'm using. In my content script's, I have the following:

var an = Oe.reject,
        Ko = '',
        ii =
          'Invalid key provided. Keys must be of type string, number, Date or Array<string | number | Date>.',
        Zx = 'String expected.',
        ru = [],
        Ld = '__dbnames',
        sm = 'readonly',
        im = 'readwrite';
      function Go(a, c) {

Note the Ko value.

This seems to come from here: https://github.com/dexie/Dexie.js/blob/048334aa84f00a5bdf8f4f6720397409a3ec1495/src/globals/constants.ts#L7

So before it's minified it looks like this:

import { Dexie } from "../classes/dexie";

export const DEXIE_VERSION = '{version}'; // Replaced by build-script.
export const maxString = String.fromCharCode(65535);
export const minKey = -Infinity; // minKey can be constant. maxKey must be a prop of Dexie (_maxKey)
export const INVALID_KEY_ARGUMENT =
  "Invalid key provided. Keys must be of type string, number, Date or Array<string | number | Date>.";
export const STRING_EXPECTED = "String expected.";
export const connections: Dexie[] = [];
export const dexieStackFrameFilter = frame => !/(dexie\.js|dexie\.min\.js)/.test(frame);
export const DBNAMES_DB = '__dbnames';
export const READONLY = 'readonly';
export const READWRITE = 'readwrite';

And the particular offending line is: String.fromCharCode(65535);

Here's a discussion which I think is pertinent: https://issues.chromium.org/issues/395595611

All my Dexie interactions occur in the service worker but there are certainly some references to types now in the content scripts because of the way I've used sendMessage and the protocol typings from webext-core/messenger that were not there with webext-bridge.

Perhaps this makes more sense to @aklinker1 and others than it does to me and you will have some suggestions as to how best to proceed.

Thanks all!

The Solution in my case (still using webext-core/messaging

The issue was caused by importing sendMessage from @/entrypoints/background where my handlers are defined.

The content scripts were importing the entire background script, which included Dexie and all its dependencies and crucially the file linked to above which caused the non-UTF-8 character String.fromCharCode(65535)line to to end up in the content script bundles.

The import chain had the content scripts pulling in:

All the handler imports My SearchService registration And...all of Dexie The solution was to separate the messaging definition from the background script by creating a dedicated messaging module:

The background script could then import sendMessage and onMessage for webext-core/messaging and the content scripts could import just these from the messaging service module. This saved the content scripts from pulling in all of the unwanted stuff.

One hint that this was happening was that on removing webext-bridge and moving to webext-core/messenger, the content script sizes all went up significantly. I didn't think too much of it and just assumed it was maybe a large package but this import chain likely explains it!

For info, webext-bridge was good and I had chosen it previously after some consideration, however, it uses ports under the hood.