Revert ts5.9 changes for Uint8Array

[paulmillr]

⚙ Compilation target

es2022

⚙ Library

es2022, dom

Missing / Incorrect Definition

1. User sees weird error. "Uint16Array is not assignable to Uint16Array<ArrayBuffer>". First thoughts: WTF? What does this even mean? "But I don't use SharedArrayBuffer in my project!!"

TS2345: Argument of type 'Uint16Array' is not assignable to parameter of type 'Uint16Array<ArrayBuffer>'.
  Type 'ArrayBufferLike' is not assignable to type 'ArrayBuffer'.
    Type 'SharedArrayBuffer' is not assignable to type 'ArrayBuffer'.
      Types of property '[Symbol.toStringTag]' are incompatible.
        Type '"SharedArrayBuffer"' is not assignable to type '"ArrayBuffer"'.

2. User heads to blog post and sees "explicitly writing out Uint8Array<ArrayBuffer> rather than a plain Uint8Array"
3. User applies the changes.
4. The changes make library unusable in TS <=5.6.
5. User reverts to older TS to ensure his library doesn't require "latest ts".
6. There is zero other documentation about resolving the problem, so it's unclear how to solve it. It's also not obvious that () => new Uint8Array, () => Uint8Array.from([]) and () => Uint8Array.of() now all return subtyped uint8array, which fails in TS <=5.6

---

Previous issue text

ts5.7 change which made uint8arrays generic was okay.

ts5.9 introduced absolutely horrific change which breaks Uint8Array-heavy projects.

It seems like some changes made types too strict. It's unclear at which phase - perhaps during inference (i'm using infer). I'm now constantly getting those errors and it's really unhelpful - it's unclear what to do. Blog post recommendations (https://www.typescriptlang.org/docs/handbook/release-notes/typescript-5-9.html#libdts-changes) make arrays subtype-specific. It seems like this is a wrong way to fix things. Users should not be forced to always consume ArrayBuffer-based type arrays. It's perfectly fine to use either ArrayBuffer or SharedArrayBuffer based arrays, the changes are minor. The defaults should be able to consume both types!

// type Poly = Uint16Array
TS2345: Argument of type 'Poly' is not assignable to parameter of type 'Uint16Array<ArrayBuffer>'.
  Type 'ArrayBufferLike' is not assignable to type 'ArrayBuffer'.
    Type 'SharedArrayBuffer' is not assignable to type 'ArrayBuffer'.
      Types of property '[Symbol.toStringTag]' are incompatible.
        Type '"SharedArrayBuffer"' is not assignable to type '"ArrayBuffer"'.

Sample Code

I've spent a lot of time but was not able to produce a standalone example which reproduces the issue. There are a bunch of types at play and it's unclear which exact type causes the issue.

The code which produces errors is in commit cb76058 of https://github.com/paulmillr/noble-post-quantum

git clone https://github.com/paulmillr/noble-post-quantum
cd noble-post-quantum
git checkout cb76058
npm install
npm run build

[MartinJohns]

This is intentional and documented. They mention a way to fix it, which you didn't use.

https://www.typescriptlang.org/docs/handbook/release-notes/typescript-5-9.html#libdts-changes

[paulmillr]

@MartinJohns Nope. As I mentioned above:

The "fix" mentioned in blog posts makes arrays subtype-specific. The goal is for my code to consume any typed array - both ArrayBuffer and SharedArrayBuffer based. I am not using any "specific" methods. It seems like type inference screws up somewhere and makes stuff strict.

Changing this line from type Poly = Int32Array to type Poly = Int32Array<any> fixes things but it seems to be wrong. Shouldn't Int32Array<any> be the default?

https://github.com/paulmillr/noble-post-quantum/blob/cb76058c3328cb5ec09c007824ecd8de3f345dc4/src/ml-dsa.ts#L77

[Renegade334]

The problem, as you've identified, is where in-types and out-types collide. The array buffer view constructors almost always return a T<ArrayBuffer> by design; if that type is inferred and is then carried forward to type, say, a function parameter, then the change in variance can lead to issues.

These cases now require explicit annotations in certain places, to avoid inferring a narrowed type from a newly constructed array buffer view where narrowing is not desirable. In this instance, two changes are all that's needed for the project to build:

--- a/src/ml-dsa.ts
+++ b/src/ml-dsa.ts
@@ -75,7 +75,7 @@ export const PARAMS: Record<string, DSAParam> = {
 
 // NOTE: there is a lot cases where negative numbers used (with smod instead of mod).
 type Poly = Int32Array;
-const newPoly = (n: number) => new Int32Array(n);
+const newPoly = (n: number): Int32Array => new Int32Array(n);
 
 const { mod, smod, NTT, bitsCoder } = genCrystals({
   N,
diff --git a/src/ml-kem.ts b/src/ml-kem.ts
index fbd9a6a..18c8994 100644
--- a/src/ml-kem.ts
+++ b/src/ml-kem.ts
@@ -46,7 +46,7 @@ const { mod, nttZetas, NTT, bitsCoder } = genCrystals({
   Q,
   F,
   ROOT_OF_UNITY,
-  newPoly: (n: number) => new Uint16Array(n),
+  newPoly: (n: number): Uint16Array => new Uint16Array(n),
   brvBits: 7,
   isKyber: true,
 });

ts5.9 introduced absolutely horrific change which breaks Uint8Array-heavy projects.

This is only semi-accurate; these changes already existed from TS 5.7, but only for es2024 targets, which is where the changes to ArrayBuffer and SharedArrayBuffer made those interfaces diverge (#59417). Reverting the change which made this apply to all targets in TS 5.9 would only kick this can down the road until users start adopting es2024, and reverting the original change altogether isn't an option for the reasons detailed there.

[paulmillr]

The fix is reasonable, simple, and makes sense. Thanks.

However, it's completely non-obvious. If it's non-obvious to me (a lib author), then to my users it could be even worse. They would just slip on Uint8Array and call it a day. Or they would slip on Uint8Array<ArrayBuffer> and get more type errors in some cases.

Is there any way to make this kind of fix more documented? The blog post really does a bad job in this regard. Something like "Writing fn = () => new Uint8Array() would always return specific Uint8Array<ArrayBuffer> since ts5.9" would work.

[paulmillr]

Applied this fix in other libs -- just for the record, those are the diffs of changes that I thought are correct vs what's correct as per comment above

• https://github.com/paulmillr/noble-curves/commit/878621b4034405ccb45075220c4840a170770ea1
• https://github.com/paulmillr/noble-hashes/commit/46bf893a18a62837bccf471cf3841b775de27830
• https://github.com/paulmillr/noble-secp256k1/commit/c00b81b17de67b65fec0d705c25e1ec0549cbc05

[paulmillr]

Ok it gets worse. Generic typed arrays aren't supported before ts 5.7. So a project with ts 5.0 can't compile fixes recommended by ts 5.9.

Ts 5.0 stuff doesn't work in 5.9. Ts 5.9 stuff (subtype) doesn't work in 5.0.

The docs must definitely be updated to accomodate for this, listing proper fixes.

[jakebailey]

TS 5.0 is well out of support, no? (See also, https://github.com/DefinitelyTyped/DefinitelyTyped?tab=readme-ov-file#support-window, which is as conservative as we get)

[paulmillr]

Instead of ts 5.0 I should have said "any ts <= 5.6". Which msft seems to support.

[pnappa]

Yeah, I hit a similar cascade of changes due to the Uint8Array update.

For the record, I'm not for reverting the changes per se, I'm just wishing to highlight the additional work this required for developers (and as a consequence, politely request that the ramifications of these effects on downstream users is considered more thoroughly).

Our problem was a result of various Uint8Array and Buffer usages in NodeJS libraries & our code - some old libraries expect/return Buffer, and some expect/return Uint8Array.

I essentially had to rip out all Buffer usages, and insert a shim to convert them to/from Uint8Array, where expected. It's an unfortunate load of work required, which required a bunch of package updates, tsconfig updates, and removal of dependencies to resolve this. It's also an annoying performance downside (by way of having to go through constructors to get the right type) just to satisfy the types without casting.

This all came about because a developer was using VSCode which automatically uses the latest TS version, whilst the codebase was still a couple minor versions behind. Not wanting to resist the future, I had to update the codebase to 5.9, causing all these changes (which would have had to happen sooner or later, given this Uint8Array change).

Oh well! Life in the JS ecosystem 😄

[Nerivec]

Quick note, this also affects Buffer and other related types that use the same mechanism. It makes a mess with narrowing, and node built-ins (& external deps...). Some of the related issues were marked "Needs investigation" while others "Working as intended"... https://github.com/microsoft/TypeScript/issues/62168 https://github.com/microsoft/TypeScript/issues/61793 Seems the whole thing is rather "uncertain" at the moment...

[RyanCavanaugh]

Effectively ~all of these errors people are encountering represent problems that really could have induced actual runtime errors. For better or worse we do generally treat "This was not actually reliably working in prior versions" lib situations as bugs. The docs can always do better and I'd be happy to see a useful update to the appropriate location, but there's not a lot of universally-applicable guidance we can give in these situations since the exact manifestation of type errors can vary quite a lot depending on what's going on.

[paulmillr]

So this fails in ts5.9. How exactly are library authors supposed to write this code to ensure it's working in both ts5.9 and ts5.5?! @RyanCavanaugh @Renegade334

function create(): Uint8Array {
  return new Uint8Array(1);
}
function create2(): Uint8Array<ArrayBuffer> {
  return new Uint8Array(1);
}

fetch('url', { body: create() }); // fails in ts5.9
fetch('url', { body: create2() }); // fails in ts5.5

[MartinJohns]

https://www.typescriptlang.org/docs/handbook/declaration-files/publishing.html#version-selection-with-typesversions

[Renegade334]

How exactly are library authors supposed to write this code to ensure it's working in both ts5.9 and ts5.5?!

Removing the explicitly typed return, and just allowing it to be inferred, is the easiest – although this is only an option for internal consumption, not if the type ends up getting publicly exported from your library.

Some places have been using a derived alias instead, eg.

type NonSharedUint8Array = ReturnType<typeof Uint8Array.from>;

and this does survive declaration emit, so should work just fine with downstream consumers.

(@types/node uses the typesVersions system, but it can be rather cumbersome.)

[paulmillr]

@MartinJohns to be clear, do you suggest to compile additionally with ts5.5?

Increasing pkg size 2x? (Esm + cjs d.ts.map). And how exactly is that proposal supposed to work with pkg.json export maps, which map /a.js to esm/a.js, cjs/a.js, and their corresponding types?

[paulmillr]

@Renegade334 i’m talking about public library APIs, yes. The lib tsconfig also uses isolatedDeclatations, which forces everything to be explicitly typed.

Is the suggestion to add return type NonSharedUint8Array to all library methods which return generic U8A?

[Renegade334]

I suspect that it should do the trick, aye. (Providing that's definitely what you're returning, but if you're constructing the Uint8Array yourself, that's almost certainly the case.)

Again, one of those hacks that should probably be mentioned somewhere...

[guest271314]

Yeah, this is indeed a cascade of TypeScript errors. Yes, it's a bug. It can't be intentional that TypeScript deviates from W3C File API and WHATWG Web IDL.

There's no way this should be throwing am error

async function sendMessage(message: Uint8Array): Promise<void> {
  await new Blob([
    new Uint8Array(new Uint32Array([message.length]).buffer),
    message, // <= Uint8Array
  ])

Type 'Uint8Array<ArrayBufferLike>' is not assignable to type 'BlobPart'

What? Sure it is per the controlling specifications

• https://w3c.github.io/FileAPI/#constructorParams
• https://webidl.spec.whatwg.org/#BufferSource

[darksabrefr]

IMO, if these changes really make sense, the problematic thing that causes frictions is the default type of the generic. Defining Uint8Array as interface Uint8Array<TArrayBuffer extends ArrayBufferLike = ArrayBuffer> instead of interface Uint8Array<TArrayBuffer extends ArrayBufferLike = ArrayBufferLike> would have cause far less pain to everyone. SharedArrayBuffer are not used as often than ArrayBuffer and it's pretty unattended that new Uint8Array() doesn't return the Uint8Array type (it returns Uint8Array<ArrayBuffer> where currently the type Uint8Array means Uint8Array<ArrayBufferLike>). The same remark goes to other typed arrays and node's Buffer.

[guest271314]

I don't know about the same code supporting different versions of TypeScript source.

Over here https://github.com/microsoft/TypeScript/issues/62546 this was a sound answer with solution https://github.com/microsoft/TypeScript/issues/62546#issuecomment-3374526284 to the BlobPart inquiry I had. Since I don't write TypeScript every day I doubt I would have written that same code in isolation, without a lot of testing to see what works.

[Renegade334]

IMO, if these changes really make sense, the problematic thing that causes frictions is the default type of the generic. Defining Uint8Array as interface Uint8Array<TArrayBuffer extends ArrayBufferLike = ArrayBuffer> instead of interface Uint8Array<TArrayBuffer extends ArrayBufferLike = ArrayBufferLike> would have cause far less pain to everyone. SharedArrayBuffer are not used as often than ArrayBuffer and it's pretty unattended that new Uint8Array() doesn't return the Uint8Array type (it returns Uint8Array<ArrayBuffer> where currently the type Uint8Array means Uint8Array<ArrayBufferLike>). The same remark goes to other typed arrays and node's Buffer.

This just flips the polarity of the assignability problem, except now with even more side-effects, since eg. not all Uint8Arrays would be assignable to Uint8Array.

Any approach leads to some cases of "my code worked before and now doesn't". The one chosen keeps all generics assignable to the default, keeps the shape of the .buffer property consistent with TS 5.6 in the default case, and is the most straightforward in terms of the changes needed to work around assignability issues, even if the fixes aren't immediately obvious from the errors generated. There's going to be no appetite for a second massive breaking change just to exchange one set of errors for another.

[paulmillr]

@Renegade334 the current new state of ts worsens DevEx a lot. A solution needs to be found in order to bring it back to a reasonable state.

Right now developers would look at all of this and decide that “as unknown as any” is fine because errors are barely debuggable.

[RyanCavanaugh]

A solution needs to be found in order to bring it back to a reasonable state.

We looked for a solution and didn't find any that were acceptable, and can't keep looking forever. Open to ideas though!

[guest271314]

Right now developers would look at all of this and decide that “as unknown as any” is fine because errors are barely debuggable.

We looked for a solution and didn't find any that were acceptable, and can't keep looking forever. Open to ideas though!

Doesn't matter once it gets to and is executed by JavaScript runtimes. TypeScript is to me a convenience syntax for clarity. When there is no clarity without ambiguity in the syntax, especially relevant to trying to support something other that TOT releases, I suggest to leave comments linking to these issues, which are the primary source for the current art. That's basically what the syntax is doing anyway.

[guest271314]

Another option, for developers, is to just not use TypedArray's in TypeScript at all. Just use ArrayBuffer and DataView.

That's what I did for an AssemblyScript version of an algorithm I've ported to multiple programming languages, and different JavaScript engines and runtimes.

When the tooling takes more time to debug than the source code itself, there's a serious problem with the tooling.

[RyanCavanaugh]

@guest271314 I think you've weighed in on this enough now, thanks for the feedback

[webmaster128]

To make users' lives easier after this change, I think it would be great if TypeScript offered a way to narrow down Uint8Array<ArrayBufferLike> to e.g. Uint8Array<ArrayBuffer>. Use case is: I have a function, consume Uint8Array but internally I can only process Uint8Array<ArrayBuffer> (e.g. importKey from subtle).

See this example

export function myfunc(source: Uint8Array<ArrayBuffer> | Uint8Array) {
  if (source.buffer instanceof ArrayBuffer) {
    // source is still Uint8Array<ArrayBufferLike> | Uint8Array<ArrayBuffer>, not Uint8Array<ArrayBuffer>
    // can we get a Uint8Array<ArrayBuffer> please 🐱 
  }

  throw new Error("Shared buffers are not yet supported"); // or copy or whatever fallback
}

or SO question. Maybe there is already something I don't know?

[paulmillr]

@RyanCavanaugh what was the reason to not default Uint8Array to Uint8Array<ArrayBuffer>? The inheritance changes overall may be not ideal, but this default alias would have prevented all kinds of craziness in the issue.

[bradthomasbrown]

Joining in with paulmillr, the way this works right now is extremely confusing and unintuitive and very simple and straightforward usage led me right here instead of having working, error-free code.

function foo(bar:()=>Uint8Array) {
    let baz = new Uint8Array();
    baz = bar();
    // Type 'Uint8Array<ArrayBufferLike>' is not assignable to type 'Uint8Array<ArrayBuffer>'.
}

If these are supposed to be generics, why would you not either: A: have the generics default to a standard type B: have the generics not default to any standard type and have the developer be explicit rather than C: have the generics default to two conflicting types (current behavior)

[snarbles2]

The default generic argument is the wider option (<ArrayBufferLike>). It would be weird if (arg: Uint8Array) => {} couldn't accept all Uint8Arrays. Probably don't want to change that, so strike option (A).

Option B would have been fine by me, but IIRC a breaking change has already been made once to fix a mistyping. I doubt there's appetite to break everyone again.

Honestly, it seems like the biggest problem a lot of people are running into is that the inferred type for variables and arguments is too narrow. Making the Uint8Array constructor return a Uint8Array<ArrayBufferLike> would not be wrong, just less precise, and would alleviate that issue. This wouldn't preclude people from being more precise when they need to be.

---

what was the reason to not default Uint8Array to Uint8Array<ArrayBuffer>?

If the default generic argument were ArrayBuffer, then when you write a function that accepts Uint8Array, it wouldn't be able to accept all Uint8Arrays. It would move the same confusion to a different spot and exchange some false positives for false negatives.