ReferenceError: TextEncoder is not defined

[timafield]

Not sure if this is maybe specific to my version of node (v12), or my environment (typescript: 6.13.4, etc...)

When I try to call, MyMessage.toBinary(myMsg); ../../node_modules/@protobuf-ts/runtime/src/binary-writer.ts throws ReferenceError: TextEncoder is not defined

Changing it to be an explicit import in the js fixed it for me, like: Inserting: const util_1 = require("util"); after the imports on line 6. And making line 13: this.textEncoder = textEncoder !== null && textEncoder !== void 0 ? textEncoder : new TextEncoder(); Read: this.textEncoder = textEncoder !== null && textEncoder !== void 0 ? textEncoder : new util_1.TextEncoder();

I haven't taken the time to checkout the project. But I'd be happy to make the change when I have time. I'd seek to add import { TextEncoder } from 'util'; in binary-writer.ts. It should be pretty safe, but I don't know if TextEncoder was meant to be taken from util or if there might have been a different package intended. ¯_(ツ)_/¯

I haven't gotten as far as decoding any messages, so new TextDecoder() may be an issue too if we initialize it like that somewhere.

Lmk if I'm way off here.

[timostamm]

Hi Tim,

we can't import TextEncoder from 'util' because 'util' is only available in node. Later node versions (not sure which) make TextEncoder available in the global scope, like web browsers do.

There is a workaround:

import {TextEncoder} from "util";
import {BinaryWriteOptions, BinaryWriter} from "@protobuf-ts/runtime";

const myWriteOptions: BinaryWriteOptions = {
    // set a custom writer factory that passes the node 
    // text encoder to the writer
    writerFactory: () => new BinaryWriter(new TextEncoder()),
    writeUnknownFields: true
};

// assuming you have a message type "Test" and an instance of it "t1"
let bytes = Test.toBinary(t1, myWriteOptions);

Let me know if this works for you.

Could you check if globalThis.TextEncoder is defined in your environment? (JS or TS, should not matter).
If so, new TextEncoder() could be replaced with new globalThis.TextEncoder(), and the workaround would not be necessary.

[timafield]

Ah that makes sense. Unfortunately globalThis is defined but globalThis.TextEncoder is not. Looking at the node docs, they claim that it was added at least at some point in v11.X. I was on v12.16.0, and updated to the latest v12.19.0, but I still didn't see it. (I can't really go up to 14 for instance due to my application).

Thank you for the help, though.

[alejandroclaro]

Same problem in Node 14.x

[timostamm]

Same problem in Node 14.x

This project is running continuous integration on Node 14.5, and there are many tests of the runtime library that will hit TextEncoder, and there are also conformance tests that run on multiple variations of the generated code, and we haven't seen this error yet.

I've just tested on Node v14.19.0 and cannot reproduce either.

Can you give more details about your environment? The exact Node version, code generation parameters and the module system being used?

[alejandroclaro]

Thank you for the quick answer @timostamm

my environment is:

NodeJS: v14.18.3 Typescript: v4.4.3 Jest: v26.6.1

The error occurs inside my own unit test using Jest. To be sure, I printed process.version variable and got '14.18.3'.

We have experienced the lack of TextEncoder in the pass in nodeJS and electron. To solve that, we have an util function like this:

function encodeUtf8String(input?: string): Uint8Array {
  if (isNodeJS()) {
    return new Uint8Array(input ? [] : Buffer.from(input, 'utf-8').buffer);
  } else if (isBrowser()) {
    return new TextEncoder().encode(input);
  } else {
    throw new Error('Unknown environment. Encoding is not possible.');
  }
}

to avoid having to deal with this in multiple places.

I created a custom TextEncoderLike using my function, and it's working fine.

It would be nice if something similar could be the default behavior in the BinaryWriter.

[timostamm]

I can't reproduce with node v14.18.3 either. I suspect this is related to transpilation, or how node is run.

I have another project here that also uses TextEncoder where I have never seen this issue. The sources are compiled with typescript v4.5.5 to CommonJS, then run with jest v27.5.1.

tsconfig.json

	{
	  "include": [
		"src/**/*.ts"
	  ],
	  "compilerOptions": {
		"target": "es2020",
		"lib": [
		  "ES2016",
		  "ES2020.BigInt"
		],
		"strict": true,
		"importsNotUsedAsValues": "error",
		"noImplicitAny": true,
		"strictNullChecks": true,
		"strictFunctionTypes": true,
		"strictBindCallApply": true,
		"strictPropertyInitialization": true,
		"noImplicitThis": true,
		"useUnknownInCatchVariables": true,
		"noUnusedLocals": true,
		"noImplicitReturns": true,
		"noFallthroughCasesInSwitch": true,
		"noImplicitOverride": true,

		// We need node's module resolution, so we do not have to skip lib checks
		"moduleResolution": "Node",
		"skipLibCheck": false,

		// We need correct line numbers in jest test failures
		"sourceMap": true
	  }
	}

jest.config.js

	/*
	 * For a detailed explanation regarding each configuration property and type check, visit:
	 * https://jestjs.io/docs/configuration
	 */
	/** @type {import('@jest/types').Config.InitialOptions} */
	const config = {

	  // Indicates which provider should be used to instrument code for coverage
	  coverageProvider: "v8",

	  // The root directory that Jest should scan for tests and modules within
	  rootDir: "dist/esm",

	};

	export default config;

Jest is run with:

npx tsc --project tsconfig.json --module commonjs --outDir ./dist/cjs
npx jest

How do you run Jest? Using Node's Buffer is an option, but I'm very wary of that because it needs bypassing all linters and type checks. So I'd like to understand what's going on first.

[alejandroclaro]

Let me roll back my TextEncoderLike and perform some test to understand what could be the cause that TextEncoder is not available.

I'm executing Jest directly by invoking 'jest' with a jest.config.json a little more elaborated than yours. Also, the tsconfig.json is different.

[ballcoach12]

FWIW, I got the same error. I solved the problem by upgrading to the latest Jest and ts-jest versions.