[Feature request] Deno Support

[omar2205]

Name of the feature Support running in the Deno runtime.

Additional context I tried all the tricks, esm.sh, unpkg, esm.run, and npm:@xenova/transformers, but nothing worked.

Using unpkg import I get

This browser lacks typed array (Uint8Array) support which is required 
by `buffer` v5.x. Use `buffer` v4.x if you require old browser support.

Using esm.sh

Uncaught SyntaxError: The requested module '/v114/[email protected]/deno/pngjs.mjs' 
does not provide an export named 'PNG'

[xenova]

This seems to be an problem with onnxruntime itself: https://github.com/microsoft/onnxruntime/issues/10913

The person who opened the issue did seem to find a workaround, so, maybe that will help too?

@josephrocca is quite active in the community, and since he opened up that issue, he might have some more insight.

[omar2205]

So @nestarz created a Deno onnx_runtime here https://github.com/nestarz/onnx_runtime

[xenova]

So @nestarz created a Deno onnx_runtime here https://github.com/nestarz/onnx_runtime

That's good to know - thanks! Have you tried overriding the dependency in your project?

[omar2205]

Not sure how. For example, this is my script file

  import * as t from 'npm:@xenova/transformers'
  
  const pipeline = t.default.pipeline
  
  let pipe = await pipeline('sentiment-analysis')
  
  let out = await pipe('I love transformers!')

This gets a new error, error: Uncaught Error: Not implemented: Worker from onnxruntime-web/1.14.0/dist/ort-web.node.js

[xenova]

Maybe this might help: https://www.reddit.com/r/Deno/comments/x9q9vp/how_do_you_patch_a_module_like_patchpackage_in/ ? I haven't used Deno before, so, I probably won't be able to provide much more help (other than googling or "chatgpt-ing" 🤣 )

[xenova]

Have you been able to get it working in Deno yet? If not, feel free to try with the latest version 2.0.0-alpha.0. This may fix it.

[omar2205]

Hey, I just tried it, using npm: and esm.sh and both got an error. esm.sh:

Uncaught Error: Dynamic require of "../bin/napi-v3/linux/x64/onnxruntime_binding.node" is not supported

npm:

Uncaught Error:                                                                 
Something went wrong installing the "sharp" module                              
                                                                                
Cannot find module '../build/Release/sharp-linux-x64.node'
Require stack:

[takoyaro]

it looks like sharp doesn't support deno at the moment

[xenova]

@omar2205 Have you tried the steps outlined here: https://github.com/lovell/sharp/issues/2583#issuecomment-1310628079 ?

This is an apparent workaround until https://github.com/denoland/deno/issues/16164 is made available.

[omar2205]

Thanks for the ping @xenova, I followed that and it worked, however, I'm met with Invalid URL: error.

import { pipeline } from 'npm:@xenova/transformers';
/* warning
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
*/

let pipe = await pipeline('sentiment-analysis');

/* Error
No model specified. Using default model: "Xenova/distilbert-base-uncased-finetuned-sst-2-english".
Uncaught TypeError: Invalid URL: 'Xenova/distilbert-base-uncased-finetuned-sst-2-english/tokenizer.json'
*/

[xenova]

Okay great! That error is probably deno specific, because the check to see whether something is a valid URL is surrounded by a try-catch block (and works in other environments).

https://github.com/xenova/transformers.js/blob/0b403ce8e4cf4ab447f8f3b208920aa4e77dc499/src/utils/hub.js#L171-L179

While I look into it, you can get around it temporarily by disabling local models:

import { pipeline, env } from 'npm:@xenova/transformers';

env.allowLocalModels=false;

let pipe = await pipeline('sentiment-analysis');

[omar2205]

I tried to do that before, but it fails:

No model specified. Using default model: "Xenova/distilbert-base-uncased-finetuned-sst-2-english".
Uncaught TypeError: Invalid URL: 'Xenova/distilbert-base-uncased-finetuned-sst-2-english/tokenizer.json'

[xenova]

Hmm, it must be failing elsewhere then. Are you able to send the full stack trace?

[omar2205]

Here you go

No model specified. Using default model: "Xenova/distilbert-base-uncased-finetuned-sst-2-english".
Uncaught TypeError: Invalid URL: 'Xenova/distilbert-base-uncased-finetuned-sst-2-english/tokenizer.json'
    at getSerialization (ext:deno_url/00_url.js:88:11)
    at new URL (ext:deno_url/00_url.js:383:27)
    at new Request (ext:deno_fetch/23_request.js:297:25)
    at Cache.[[[matchAll]]] (ext:deno_cache/01_cache.js:230:11)
    at Cache.match (ext:deno_cache/01_cache.js:173:36)
    at getModelFile (file:///home/ubuntu/.cache/deno/npm/registry.npmjs.org/@xenova/transformers/2.0.0-alpha.2/src/utils/hub.js:347:32)
    at eventLoopTick (ext:core/01_core.js:166:11)
    at async getModelJSON (file:///home/ubuntu/.cache/deno/npm/registry.npmjs.org/@xenova/transformers/2.0.0-alpha.2/src/utils/hub.js:456:18)
    at async Promise.all (index 0)
    at async loadTokenizer (file:///home/ubuntu/.cache/deno/npm/registry.npmjs.org/@xenova/transformers/2.0.0-alpha.2/src/tokenizers.js:49:16)

[xenova]

Thanks! It looks like deno has it's own caching API, which only accepts HTTP URLs. I'll see what I can do.

[xenova]

In the meantime, can you try disabling the cache system? You can do this by setting: env.useBrowserCache=false;

[omar2205]

That fixed it. But we got another error 😅

> let out = await pipe('I love transformers!');

An error occurred during model execution: "TypeError: Tensor.data must be a typed array (9) for int64 tensors, but got typed array (-1).".
Inputs given to model: {
  input_ids: Tensor {
    dims: [ 1, 6 ],
    type: "int64",
    data: BigInt64Array(6) [ 101n, 1045n, 2293n, 19081n, 999n, 102n ],
    size: 6
  },
  attention_mask: Tensor {
    dims: [ 1, 6 ],
    type: "int64",
    data: BigInt64Array(6) [ 1n, 1n, 1n, 1n, 1n, 1n ],
    size: 6
  }
}
Uncaught TypeError: Tensor.data must be a typed array (9) for int64 tensors, but got typed array (-1).
    at file:///home/ubuntu/.cache/deno/npm/registry.npmjs.org/onnxruntime-node/1.14.0/dist/backend.js:45:108
    at Array.processTicksAndRejections (ext:deno_node/_next_tick.ts:23:21)
    at eventLoopTick (ext:core/01_core.js:178:29)

[xenova]

BigInt64Array is a typed array, so, I'm not too sure what the error message means... 👀 This is most likely a limitation of onnxruntime-web/onnxruntime-node :/

[omar2205]

Yeah looks like it. Is it possible to use this https://github.com/nestarz/onnx_runtime?

[xenova]

If you'd like, you can fork this repo and replace the onnxruntime import with that?

[omar2205]

After second thought, I don't know how that would even work. It's written for Deno.

[xenova]

They seem to provide an import syntax in their README:

import * as ort from "https://deno.land/x/onnx_runtime/mod.ts";

You can edit the ONNX imports in https://github.com/xenova/transformers.js/blob/main/src/backends/onnx.js

[josephrocca]

Maybe I'm missing something, but: Wouldn't it make more sense to use the browser version of onnxruntime for Deno? Deno is aimed at being strongly web-compatible, so running transformers.js/onnxruntime with node.js emulation (i.e. importing with npm: rather than e.g. jsdelivr) seems like a very round-about way that would result in more problems? Again, might be missing something as I have only skimmed this issue.

[omar2205]

Trying using jsdelivr gives Uncaught DOMException error

[xenova]

I set up a codespace and got it working:

import { pipeline, env } from 'https://cdn.jsdelivr.net/npm/@xenova/transformers';
// import {pipeline, env} from '@xenova/transformers';

env.useBrowserCache=false;
env.allowLocalModels=false;

let pipe = await pipeline('text-classification')

let output = await pipe('I love Transformers.js')

console.log(output)

console output:

$ cat main.js | deno run --allow-all --unstable -
No model specified. Using default model: "Xenova/distilbert-base-uncased-finetuned-sst-2-english".
[ { label: "POSITIVE", score: 0.99961256980896 } ]

---

The only drawback with this approach is that you might experience a performance hit since it's running with the WASM backend (and not native CPU). I'll look into why the tensor types are giving errors when running the native version.

[omar2205]

I got Uncaught DOMException when I opened this issue and tried it yesterday But it looks like it worked fine with you, and in this replit https://replit.com/@omar22051/IncompleteMenacingPattern#index.ts

When I tried it in a deno deploy, I got hit with the memory limit.

[xenova]

I got Uncaught DOMException when I opened this issue and tried it yesterday But it looks like it worked fine with you, and in this replit https://replit.com/@omar22051/IncompleteMenacingPattern#index.ts

I don't quite know where Transformers.js would be throwing a DOMException 👀 Is it maybe an issue with your environment (since it works in that replit)?

When I tried it in a deno deploy, I got hit with the memory limit.

Do you know what the memory limit is for deno deploy? Some sources suggest either 256MB or 512MB. The model you're testing with isn't that large.

[omar2205]

I believe it's 512 MB. I managed to get a QA bot going, deploy to Deno Deploy here, which was the demo I was going for.

I don't know if you want to close this issue or wait for onnx to support Deno.

[xenova]

Cool! I'll try look into the previous issue and see if there's a workaround since it might take a while for them to support Deno properly. I think it will also be good to create a Deno tutorial/example project, so, I'll close the issue after I make that.

[birkskyum]

Related to

• https://github.com/denoland/deno/issues/20071