30350n
Mouser and LCSC crawling is broken
I've tried Digikey, Mouser, and LCSC, but none of them work.
Digikey brings me to what looks like a Lynx interactive for their login page. I can type in username and password, but I'm not sure why I need to if it's using an API private key. Also Lynx won't let me select Login (assuming it's a button) anyway. So there's no way to continue.
Mouser does return a list of parts with my API key, but when I select the part I want, it says warning: failed to get page, retrying in 3.0s via 'www2.mouser.com' with new session and user agent. It tries another eu. url too, but no luck. Then it returns warning: failed to finalize part specifications from 'https://www.mouser.com/ProductDetail/Microchip-Technology/2N4001?qs=7D1LtPJG0i0bXRytgh1wrg%3D%3D' (blocked) and exits saying it failed to import.
LCSC doesn't return anything and says failed to get webpage.
This tool looks like it's a lot simpler than others, which is nice, but I can't get it to work for some reason.
Also, I tried it on WIndows with Git Shell, and the tool doesn't work because the shell doesn't recognize selection with Spacebar and Enter. Tried it on linux, and the script in the bash shell works.
Digikey brings me to what looks like a Lynx interactive for their login page. I can type in username and password, but I'm not sure why I need to if it's using an API private key. Also Lynx won't let me select Login (assuming it's a button) anyway. So there's no way to continue.
Not sure what you mean with "Lynx interactive" exactly (google also didn't come up with any useful results). Are you just talking about the OAuth browser popup? If so, I'm honestly also not sure why that's required in addition to the API key, but I think that's just how it is. (See the digikey-api python library and the DigiKey API docs).
Mouser does return a list of parts with my API key, but when I select the part I want, it says warning: failed to get page, retrying in 3.0s via 'www2.mouser.com' with new session and user agent. It tries another eu. url too, but no luck. Then it returns warning: failed to finalize part specifications from 'https://www.mouser.com/ProductDetail/Microchip-Technology/2N4001?qs=7D1LtPJG0i0bXRytgh1wrg%3D%3D' (blocked) and exits saying it failed to import.
Can confirm, this is happening for me too. The problem here is that Mousers API sucks ass, which I've been circumventing by just crawling the page directly. Seems like they recently "hardened" their crawling protection again. That's great.
LCSC doesn't return anything and says failed to get webpage.
Similar thing, LCSC doesn't really offer an API directly, but there's API endpoints which URLs got recently changed (see #47 and #48).
Also, I tried it on WIndows with Git Shell, and the tool doesn't work because the shell doesn't recognize selection with Spacebar and Enter. Tried it on linux, and the script in the bash shell works.
I'm using/developing this from Windows, mainly using the VSCode terminal and Cmder. Standard cmd should also work I'm pretty sure.
Same problem here, and I looked for documentation in the readme about how to use this source repo instead of the one that pip installs, however I didn't find any documentation on how to do so.
Note: I'm a python n00b, with no real desire to learn python since I already develop in Rust/Java/Scala/Groovy/C/C++.. anyway, in case others run into this, what I had to do was:
pip uninstall inventree_part_import
git clone <this repo>
cd <cloned_repo>
git submodule init
git submodule update
pip install .
Likely there's other ways that might work, but it might be helpful if the readme has some documentation about how to do this.
After the above I'm happy to report I was able to import a component from LCSC into a fresh install of inventree in my MSys2 UCRT64 environment on Windows 10.
Hydra@DESKTOP-H1JUK44 ~/.../projects/spracing-manufacturing/inventreepartimport
$ pip uninstall inventree_part_import
Found existing installation: inventree-part-import 1.6
Uninstalling inventree-part-import-1.6:
Would remove:
c:/msys64/ucrt64/bin/inventree-part-import.exe
c:/msys64/ucrt64/lib/python3.10/site-packages/inventree_part_import-1.6.dist-info/*
c:/msys64/ucrt64/lib/python3.10/site-packages/inventree_part_import/*
Proceed (Y/n)? y
Successfully uninstalled inventree-part-import-1.6
Hydra@DESKTOP-H1JUK44 ~/.../projects/spracing-manufacturing/inventreepartimport
$ cd inventree_part_import
Hydra@DESKTOP-H1JUK44 ~/.../spracing-manufacturing/inventreepartimport/inventree_part_import (master)
$ pip install .
Processing d:/users/hydra/documents/dev/projects/spracing-manufacturing/inventreepartimport/inventree_part_import
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: beautifulsoup4 in c:/msys64/ucrt64/lib/python3.10/site-packages (from inventree-part-import==1.7rc0) (4.12.3)
Requirement already satisfied: click in c:/msys64/ucrt64/lib/python3.10/site-packages (from inventree-part-import==1.7rc0) (8.1.7)
Requirement already satisfied: cutie in c:/msys64/ucrt64/lib/python3.10/site-packages (from inventree-part-import==1.7rc0) (0.3.2)
Requirement already satisfied: digikey-api>=1.0 in c:/msys64/ucrt64/lib/python3.10/site-packages (from inventree-part-import==1.7rc0) (1.0.0)
Requirement already satisfied: fake-useragent in c:/msys64/ucrt64/lib/python3.10/site-packages (from inventree-part-import==1.7rc0) (1.5.1)
Requirement already satisfied: inventree>=0.13.2 in c:/msys64/ucrt64/lib/python3.10/site-packages (from inventree-part-import==1.7rc0) (0.14.0)
Requirement already satisfied: isocodes in c:/msys64/ucrt64/lib/python3.10/site-packages (from inventree-part-import==1.7rc0) (2024.2.2)
Requirement already satisfied: mouser>=0.1.5 in c:/msys64/ucrt64/lib/python3.10/site-packages (from inventree-part-import==1.7rc0) (0.1.5)
Requirement already satisfied: platformdirs>=3.2.0 in c:/msys64/ucrt64/lib/python3.10/site-packages (from inventree-part-import==1.7rc0) (4.2.2)
Requirement already satisfied: pyyaml in c:/msys64/ucrt64/lib/python3.10/site-packages (from inventree-part-import==1.7rc0) (6.0.1)
Requirement already satisfied: requests in c:/msys64/ucrt64/lib/python3.10/site-packages (from inventree-part-import==1.7rc0) (2.32.3)
Requirement already satisfied: tablib[ods,xls,xlsx] in c:/msys64/ucrt64/lib/python3.10/site-packages (from inventree-part-import==1.7rc0) (3.6.1)
Requirement already satisfied: thefuzz in c:/msys64/ucrt64/lib/python3.10/site-packages (from inventree-part-import==1.7rc0) (0.22.1)
Requirement already satisfied: retrying>=1.3.3 in c:/msys64/ucrt64/lib/python3.10/site-packages (from digikey-api>=1.0->inventree-part-import==1.7rc0) (1.3.4)
Requirement already satisfied: inflection>=0.3.1 in c:/msys64/ucrt64/lib/python3.10/site-packages (from digikey-api>=1.0->inventree-part-import==1.7rc0) (0.5.1)
Requirement already satisfied: certauth>=1.3.0 in c:/msys64/ucrt64/lib/python3.10/site-packages (from digikey-api>=1.0->inventree-part-import==1.7rc0) (1.3.0)
Requirement already satisfied: urllib3>=1.25.3 in c:/msys64/ucrt64/lib/python3.10/site-packages (from digikey-api>=1.0->inventree-part-import==1.7rc0) (2.2.2)
Requirement already satisfied: colorama in c:/msys64/ucrt64/lib/python3.10/site-packages (from click->inventree-part-import==1.7rc0) (0.4.6)
Requirement already satisfied: charset-normalizer<4,>=2 in c:/msys64/ucrt64/lib/python3.10/site-packages (from requests->inventree-part-import==1.7rc0) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in c:/msys64/ucrt64/lib/python3.10/site-packages (from requests->inventree-part-import==1.7rc0) (3.7)
Requirement already satisfied: certifi>=2017.4.17 in c:/msys64/ucrt64/lib/python3.10/site-packages (from requests->inventree-part-import==1.7rc0) (2024.6.2)
Requirement already satisfied: soupsieve>1.2 in c:/msys64/ucrt64/lib/python3.10/site-packages (from beautifulsoup4->inventree-part-import==1.7rc0) (2.5)
Requirement already satisfied: readchar!=3.0.5 in c:/msys64/ucrt64/lib/python3.10/site-packages (from cutie->inventree-part-import==1.7rc0) (4.1.0)
Requirement already satisfied: odfpy in c:/msys64/ucrt64/lib/python3.10/site-packages (from tablib[ods,xls,xlsx]->inventree-part-import==1.7rc0) (1.4.1)
Requirement already satisfied: openpyxl>=2.6.0 in c:/msys64/ucrt64/lib/python3.10/site-packages (from tablib[ods,xls,xlsx]->inventree-part-import==1.7rc0) (3.1.4)
Requirement already satisfied: xlrd in c:/msys64/ucrt64/lib/python3.10/site-packages (from tablib[ods,xls,xlsx]->inventree-part-import==1.7rc0) (2.0.1)
Requirement already satisfied: xlwt in c:/msys64/ucrt64/lib/python3.10/site-packages (from tablib[ods,xls,xlsx]->inventree-part-import==1.7rc0) (1.3.0)
Requirement already satisfied: rapidfuzz<4.0.0,>=3.0.0 in c:/msys64/ucrt64/lib/python3.10/site-packages (from thefuzz->inventree-part-import==1.7rc0) (3.9.3)
Requirement already satisfied: pyopenssl in c:/msys64/ucrt64/lib/python3.10/site-packages (from certauth>=1.3.0->digikey-api>=1.0->inventree-part-import==1.7rc0) (23.1.1)
Requirement already satisfied: tldextract in c:/msys64/ucrt64/lib/python3.10/site-packages (from certauth>=1.3.0->digikey-api>=1.0->inventree-part-import==1.7rc0) (5.1.2)
Requirement already satisfied: et-xmlfile in c:/msys64/ucrt64/lib/python3.10/site-packages (from openpyxl>=2.6.0->tablib[ods,xls,xlsx]->inventree-part-import==1.7rc0) (1.1.0)
Requirement already satisfied: six>=1.7.0 in c:/msys64/ucrt64/lib/python3.10/site-packages (from retrying>=1.3.3->digikey-api>=1.0->inventree-part-import==1.7rc0) (1.16.0)
Requirement already satisfied: defusedxml in c:/msys64/ucrt64/lib/python3.10/site-packages (from odfpy->tablib[ods,xls,xlsx]->inventree-part-import==1.7rc0) (0.7.1)
Requirement already satisfied: cryptography<41,>=38.0.0 in c:/msys64/ucrt64/lib/python3.10/site-packages (from pyopenssl->certauth>=1.3.0->digikey-api>=1.0->inventree-part-import==1.7rc0) (40.0.2)
Requirement already satisfied: requests-file>=1.4 in c:/msys64/ucrt64/lib/python3.10/site-packages (from tldextract->certauth>=1.3.0->digikey-api>=1.0->inventree-part-import==1.7rc0) (2.1.0)
Requirement already satisfied: filelock>=3.0.8 in c:/msys64/ucrt64/lib/python3.10/site-packages (from tldextract->certauth>=1.3.0->digikey-api>=1.0->inventree-part-import==1.7rc0) (3.15.3)
Requirement already satisfied: cffi>=1.12 in c:/msys64/ucrt64/lib/python3.10/site-packages (from cryptography<41,>=38.0.0->pyopenssl->certauth>=1.3.0->digikey-api>=1.0->inventree-part-import==1.7rc0) (1.15.1)
Requirement already satisfied: pycparser in c:/msys64/ucrt64/lib/python3.10/site-packages (from cffi>=1.12->cryptography<41,>=38.0.0->pyopenssl->certauth>=1.3.0->digikey-api>=1.0->inventree-part-import==1.7rc0) (2.21)
Building wheels for collected packages: inventree-part-import
Building wheel for inventree-part-import (pyproject.toml) ... done
Created wheel for inventree-part-import: filename=inventree_part_import-1.7rc0-py3-none-any.whl size=45187 sha256=8523223d3fd2386d1e2a19c5304e4a22b2836581f91e925dec3b6d4b9ff63c96
Stored in directory: C:/msys64/tmp/pip-ephem-wheel-cache-47gt92pi/wheels/27/d0/d1/a09c80d0402cf4b65af52e3f19a0cd8216397db12ac4c4775d
Successfully built inventree-part-import
Installing collected packages: inventree-part-import
Successfully installed inventree-part-import-1.7rc0
Hydra@DESKTOP-H1JUK44 ~/.../spracing-manufacturing/inventreepartimport/inventree_part_import (master)
$ inventree-part-import.exe -c ../inventree_part_import_config/ C474855
set configuration directory to '../inventree_part_import_config/'
warning: deprecated parameter 'max_results' in config.yaml (renamed to 'interactive_part_matches')
warning: deprecated parameter 'max_results' in config.yaml (renamed to 'interactive_part_matches')
(only 1 of 5 available supplier modules are configured)
successfully loaded 17 pre creation hooks!
warning: category 'Electronic Components' on host is not defined in categories.yaml
successfully setup categories!
searching for C474855 ...
successfully added LCSC part C474855 (http://192.168.18.5/supplier-part/1/)
successfully imported all parts!
For future reference: You can simply install the newest git version via pipx install git+https://github.com/30350n/inventree_part_import (you might have to uninstall any old installed versions first).
Also, yes LCSC import was fixed in #48. I think I wanted to wait until Mouser is fixed too before creating a new release, but that might take some more time ...
For future reference: You can simply install the newest git version via
pipx install git+https://github.com/30350n/inventree_part_import(you might have to uninstall any old installed versions first).
pipx doesn't work for me on Msys2 UCRT64.
Trying to install this using pipx results in distutils.errors.DistutilsPlatformError: --plat-name must be one of ('win32', 'win-amd64', 'win-arm32', 'win-arm64')
If you add SETUPTOOLS_USE_DISTUTILS=stdlib as per this: https://github.com/pyproj4/pyproj/issues/1009#issuecomment-1225740999 then things progress until one of the dependencies tries to build the 'cryptography' package, as follows:
error: subprocess-exited-with-error
package init file 'src/cryptography/hazmat/bindings/_rust/__init__.py' not found (or not a regular file)
package init file 'src/cryptography/hazmat/bindings/_rust/openssl/__init__.py' not found (or not a regular file)
error: failed to run custom build command for `openssl-sys v0.9.102`
error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path src/rust/Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib --` failed with code 101
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (cryptography)
- log file contains:
Running `C:\msys64\tmp\pip-install-v6qjl3ql\cryptography_ae7f14d153354833a5a9b384b4af64a4\src\rust\target\release\build\pyo3-build-config-d0d0260f7f6779ba\build-script-build`
Running `C:\msys64\tmp\pip-install-v6qjl3ql\cryptography_ae7f14d153354833a5a9b384b4af64a4\src\rust\target\release\build\openssl-sys-aa5e8fa26a6155d3\build-script-main`
Running `D:\Users\Hydra\.rustup\toolchains\nightly-2024-03-11-x86_64-pc-windows-msvc\bin\rustc.exe --crate-name pyo3_build_config --edition=2021 'D:\Users\Hydra\.cargo\registry\src\index.crates.io-6f17d22bba15001f\pyo3-build-config-0.20.3\src\lib.rs' --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off -C overflow-checks=on --cfg 'feature="abi3"' --cfg 'feature="default"' --cfg 'feature="extension-module"' --cfg 'feature="resolve-config"' -C metadata=89639489921b0bcd -C extra-filename=-89639489921b0bcd --out-dir 'C:\msys64\tmp\pip-install-v6qjl3ql\cryptography_ae7f14d153354833a5a9b384b4af64a4\src\rust\target\release\deps' -C strip=debuginfo -L 'dependency=C:\msys64\tmp\pip-install-v6qjl3ql\cryptography_ae7f14d153354833a5a9b384b4af64a4\src\rust\target\release\deps' --extern 'once_cell=C:\msys64\tmp\pip-install-v6qjl3ql\cryptography_ae7f14d153354833a5a9b384b4af64a4\src\rust\target\release\deps\libonce_cell-375fe2f463e976bb.rmeta' --extern 'target_lexicon=C:\msys64\tmp\pip-install-v6qjl3ql\cryptography_ae7f14d153354833a5a9b384b4af64a4\src\rust\target\release\deps\libtarget_lexicon-d01c0dce12373169.rmeta' --cap-lints allow`
error: failed to run custom build command for `openssl-sys v0.9.102`
Caused by:
process didn't exit successfully: `C:\msys64\tmp\pip-install-v6qjl3ql\cryptography_ae7f14d153354833a5a9b384b4af64a4\src\rust\target\release\build\openssl-sys-aa5e8fa26a6155d3\build-script-main` (exit code: 101)
--- stdout
cargo:rerun-if-env-changed=X86_64_PC_WINDOWS_MSVC_OPENSSL_LIB_DIR
X86_64_PC_WINDOWS_MSVC_OPENSSL_LIB_DIR unset
cargo:rerun-if-env-changed=OPENSSL_LIB_DIR
OPENSSL_LIB_DIR unset
cargo:rerun-if-env-changed=X86_64_PC_WINDOWS_MSVC_OPENSSL_INCLUDE_DIR
X86_64_PC_WINDOWS_MSVC_OPENSSL_INCLUDE_DIR unset
cargo:rerun-if-env-changed=OPENSSL_INCLUDE_DIR
OPENSSL_INCLUDE_DIR unset
cargo:rerun-if-env-changed=X86_64_PC_WINDOWS_MSVC_OPENSSL_DIR
X86_64_PC_WINDOWS_MSVC_OPENSSL_DIR unset
cargo:rerun-if-env-changed=OPENSSL_DIR
OPENSSL_DIR unset
note: vcpkg did not find openssl: Could not find Vcpkg tree: No vcpkg installation found. Set the VCPKG_ROOT environment variable or run 'vcpkg integrate install'
--- stderr
thread 'main' panicked at D:\Users\Hydra\.cargo\registry\src\index.crates.io-6f17d22bba15001f\openssl-sys-0.9.102\build\find_normal.rs:190:5:
Could not find directory of OpenSSL installation, and this `-sys` crate cannot
proceed without this knowledge. If OpenSSL is installed and this crate had
trouble finding it, you can set the `OPENSSL_DIR` environment variable for the
compilation process.
Make sure you also have the development packages of openssl installed.
For example, `libssl-dev` on Ubuntu or `openssl-devel` on Fedora.
If you're in a situation where you think the directory *should* be found
automatically, please open a bug at https://github.com/sfackler/rust-openssl
and include information about your system as well as this message.
$HOST = x86_64-pc-windows-msvc
$TARGET = x86_64-pc-windows-msvc
openssl-sys = 0.9.102
It looks like you're compiling for MSVC but we couldn't detect an OpenSSL
installation. If there isn't one installed then you can try the rust-openssl
README for more information about how to download precompiled binaries of
OpenSSL:
https://github.com/sfackler/rust-openssl#windows
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path src/rust/Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib --` failed with code 101
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for cryptography
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (cryptography)
And yes, I had all the OpenSSL packages and -devel packages installed and pkgconfig shows them correctly..
No known solution, my google-fu doesn't turn up anything useful.
pip install cryptography works, but pipx install cryptography does not
Thus my workaround is to use pip not pipx which isn't ideal as the goal was to install a standalone app...
This should be resolved with v1.7. I'm using the parameters provided by the Mouser API now (so if they ever decide to fix it 🙄 , that part should work out of the box). More importantly though, if crawling fails now, the part will still be imported (you might just have to define all the parameters by hand).