NPM install failing with 403 response

[JulioAlbinatiCortez]

Hi folks,

I'm trying to install [email protected] over Node v12 and getting the following error: Error: 403 status code downloading tarball https://tokenizers-releases.s3.amazonaws.com/node/0.8.2/index-v0.8.2-node-v72-win32-ia32-unknown.tar.gz

Any ideas on how to overcome this issue?

[Narsil]

ia32 what's your platform/CPU ? We seem to prebuild only for x64.

An short term fix would be for you to build directly from sources.

git checkout https://github.com/huggingface/tokenizers
cd tokenizers/bindings/node
npm run compile

[JulioAlbinatiCortez]

@Narsil I'm running in Windows 11 x64. Any extra step I would need here?

[Narsil]

It seems you are on i386 not x64 so you need to build from source I guess as said earlier.

Unfortunately I don't really know enough about npm to understand why it's showing a different error or if we should update our builds somehow.

[JulioAlbinatiCortez]

Hey @Narsil, I've updated my Node version to ensure it's x64. However, I'm still getting a similar error (note that it is now trying to fetch the x64 version):

Error: 403 status code downloading tarball https://tokenizers-releases.s3.amazonaws.com/node/0.12.0/index-v0.12.0-node-v93-win32-x64-unknown.tar.gz

[JulioAlbinatiCortez]

@Narsil when I'm building from source, I get this error instead:

[email protected] compile neon build --release

neon ERR! spawn cargo ENOENT

Error: spawn cargo ENOENT at Process.ChildProcess.handle.onexit (node:internal/child_process:283:19) at onErrorNT (node:internal/child_process:478:16) at processTicksAndRejections (node:internal/process/task_queues:83:21)

[Narsil]

Do you have rust installed ?

https://www.rust-lang.org/tools/install

[JulioAlbinatiCortez]

@Narsil I hadn't it installed. After installing Rust, I'm getting a different error when compiling neon-sys v0.3.3:

error: failed to run custom build command for neon-sys v0.3.3

Caused by: process didn't exit successfully: C:\Users\jualbina\source\repos\tokenizers\bindings\node\native\target\release\build\neon-sys-476d271a48ad654e\build-script-build (exit code: 1) --- stdout 'Skipping node-gyp installation as part of npm install.' cargo:node_arch=x64 cargo:node_root_dir=C:\Users\jualbina\.node-gyp\16.14.2 cargo:node_lib_file=C:\Users\jualbina\.node-gyp\16.14.2\<(target_arch)\node.lib

build-release node-gyp build

TARGET = Some("x86_64-pc-windows-msvc") HOST = Some("x86_64-pc-windows-msvc") AR_x86_64-pc-windows-msvc = None AR_x86_64_pc_windows_msvc = None HOST_AR = None AR = None running: "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX64\x64\lib.exe" "-out:C:\Users\jualbina\source\repos\tokenizers\bindings\node\native\target\x86_64-pc-windows-msvc\release\build\neon-sys-e95ac94588217cd9\out\libneon.a" "-nologo" "C:\Users\jualbina\source\repos\tokenizers\bindings\node\native\target\x86_64-pc-windows-msvc\release\build\neon-sys-e95ac94588217cd9\out\native\build\Release\obj\neon\neon.obj" LINK : fatal error LNK1181: cannot open input file 'C:\Users\jualbina\source\repos\tokenizers\bindings\node\native\target\x86_64-pc-windows-msvc\release\build\neon-sys-e95ac94588217cd9\out\native\build\Release\obj\neon\neon.obj' exit code: 1181

--- stderr gyp info it worked if it ends with ok gyp info using [email protected] gyp info using [email protected] | win32 | x64 gyp info spawn C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\15.0\Bin\MSBuild.exe gyp info spawn args [ gyp info spawn args 'build/binding.sln', gyp info spawn args '/clp:Verbosity=minimal', gyp info spawn args '/nologo', gyp info spawn args '/p:Configuration=Release;Platform=x64' gyp info spawn args ] gyp ERR! UNCAUGHT EXCEPTION gyp ERR! stack Error: spawn C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\15.0\Bin\MSBuild.exe ENOENT gyp ERR! stack at Process.ChildProcess._handle.onexit (node:internal/child_process:283:19) gyp ERR! stack at onErrorNT (node:internal/child_process:478:16) gyp ERR! stack at processTicksAndRejections (node:internal/process/task_queues:83:21) gyp ERR! System Windows_NT 10.0.22000 gyp ERR! command "C:\Program Files\nodejs\node.exe" "C:\Users\jualbina\source\repos\tokenizers\bindings\node\native\target\x86_64-pc-windows-msvc\release\build\neon-sys-e95ac94588217cd9\out\native\node_modules\node-gyp\bin\node-gyp.js" "build" gyp ERR! cwd C:\Users\jualbina\source\repos\tokenizers\bindings\node\native\target\x86_64-pc-windows-msvc\release\build\neon-sys-e95ac94588217cd9\out\native gyp ERR! node -v v16.14.2 gyp ERR! node-gyp -v v3.6.2 gyp ERR! This is a bug in node-gyp. gyp ERR! Try to update node-gyp and file an Issue if it does not help: gyp ERR! https://github.com/nodejs/node-gyp/issues

error occurred: Command "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX64\x64\lib.exe" "-out:C:\Users\jualbina\source\repos\tokenizers\bindings\node\native\target\x86_64-pc-windows-msvc\release\build\neon-sys-e95ac94588217cd9\out\libneon.a" "-nologo" "C:\Users\jualbina\source\repos\tokenizers\bindings\node\native\target\x86_64-pc-windows-msvc\release\build\neon-sys-e95ac94588217cd9\out\native\build\Release\obj\neon\neon.obj" with args "lib.exe" did not execute successfully (status code exit code: 1181).

warning: build failed, waiting for other jobs to finish... error: build failed neon ERR! cargo build failed

Error: cargo build failed at Target. (C:\Users\jualbina\source\repos\tokenizers\bindings\node\node_modules\neon-cli\lib\target.js:98:27) at Generator.next () at fulfilled (C:\Users\jualbina\source\repos\tokenizers\bindings\node\node_modules\neon-cli\lib\target.js:24:58) at processTicksAndRejections (node:internal/process/task_queues:96:5)

It seems like another dependency issue. Is there a documented process for building the Node bindings from scratch?

[Narsil]

Yes, but there are issues, for instance we don't support node 16+ (we need to rewrite the bindings).

Can you try with Node 12/14 ? I know this is bothering, and node-gyp errors are pretty hard to understand/debug

[praeclarum]

Hi, I'm new to node and this was tripping me up. I just want to note that @Narsil's solution works. When I dialed my node back to 14 the download worked.

I think it would be helpful to mark this package as incompatible with node 15+ as, as you said, those gyp errors are terrible.

[jrglg]

@Narsil I'm having issues here as well... This should be updated.

What can I do after compiling for x64? I'm getting the same error when trying to do npm install of this https://github.com/hashicorp/learn-cdktf-assets-stacks-lambda/blob/rln-update-cdktf-0_13/cdktf/package.json

[Narsil]

I don't really know why cdktf relies on this package.

Have you tried using node 12/14 ?

[jrglg]

Somehow it's working now.

I did add to the path lot of stuff but for now it's fine. Thank you

El jue., 3 nov. 2022 16:25, Nicolas Patry @.***> escribió:

I don't really know why cdktf relies on this package.

Have you tried using node 12/14 ?

— Reply to this email directly, view it on GitHub https://github.com/huggingface/tokenizers/issues/984#issuecomment-1302279774, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMTE3JDBUQMQKU5WOXIS733WGPKONANCNFSM5TZXKKNQ . You are receiving this because you commented.Message ID: @.***>

[itsthomthomob]

Sort of disappointed the tokenizer package requires node v14 or 12. I'm building a vector database and using Next as a front end, in node version 18.12.1. I receive the same npm install tokenizer 403 error.

I installed Rust and switched over to node v14.0.0 with nvm, but too many of my packages (Next, Material UI, etc.) require newer node versions

[Narsil]

PRs are welcome. But rewriting the neon bindings is just too big an endeavor for us at the moment.

unstable_wasm feature and example could be an escape hatch. (It's unstable for a reason please read carefully the caveats)

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.