Scraping prices of components via TME API doesn't work

[NightSkySK]

I've tried serval components creation via TME API and none of them contain supplier prices once uploaded to Inventree.

as example YAML cached result:

Category: MLCC SMD capacitors
CategoryId: 113537
Certificates:
- ROH
CustomerSymbol: ''
Description: "Capacitor: ceramic; 100nF; 50V; X7R; \xB110%; 0805"
Guarantee: null
MinAmount: 100
Multiples: 100
OfferId: null
OriginalSymbol: 0805B104K500CT
Packing:
- Amount: 4000
  Id: RL1
  IdUnit: ST
Photo: http://ce8dc832c.cloudimg.io/v7/_cdn_/DA/F2/B0/00/1/733101_1.jpg?width=640&height=480&wat=1&wat_url=_tme-wrk_%2Ftme_new_render3d.png&wat_scale=100p&ci_sign=90695e0671e587e169217987a4726ffe310da05a
Producer: WALSIN
ProductInformationPage: http://www.tme.eu/en/details/0805b104k500ct/mlcc-smd-capacitors/walsin/
ProductStatusList: []
SuppliedAmount: 0
Symbol: 0805B104K500CT
Thumbnail: //ce8dc832c.cloudimg.io/v7/_cdn_/DA/F2/B0/00/1/733101_1.jpg?width=100&height=75&q=75&ci_sign=3d2894ffeaff097857f2163591e8f4d9bae011bd
Unit: pcs
Weight: 0.034
WeightUnit: g
category: MLCC SMD capacitors
parameters:
  Capacitance: "0.1\xB5F"
  Case - inch: 0805
  Case - mm: '2012'
  Dielectric: X7R
  Manufacturer: WALSIN
  Operating temperature: "-55...125\xB0C"
  Operating voltage: 50V
  Tolerance: "\xB110%"
  Type of capacitor: ceramic
search_timestamp: 1708505132
subcategory: null

All other API which I've tested like LCSC, Mouser, and Digi-Key have prices in cached yaml:

category: Capacitors
currency: USD
detailed_description: "10 \xB5F \xB120% 6.3V Ceramic Capacitor X5R 0603 (1608 Metric)"
digi_key_part_number: 1276-1119-1-ND
manufacturer: Samsung Electro-Mechanics
manufacturer_part_number: CL10A106MQ8NNNC
parameters:
  Applications: General Purpose
  Capacitance: "10 \xB5F"
  ECCN: EAR99
  Failure Rate: '-'
  Features: '-'
  Height - Seated (Max): '-'
  Lead Spacing: '-'
  Lead Style: '-'
  Mounting Type: Surface Mount, MLCC
  Operating Temperature: "-55\xB0C ~ 85\xB0C"
  Package / Case: 0603 (1608 Metric)
  Packaging: Cut Tape (CT)
  Ratings: '-'
  Size / Dimension: 0.063" L x 0.031" W (1.60mm x 0.80mm)
  Temperature Coefficient: X5R
  Thickness (Max): 0.035" (0.90mm)
  Tolerance: "\xB120%"
  Voltage - Rated: 6.3V
pricing:
  1: 0.11
  10: 0.079
  50: 0.0452
  100: 0.035
  500: 0.02638
  1000: 0.02072
primary_datasheet: https://mm.digikey.com/Volume0/opasdata/d220001/medias/docus/609/CL10A106MQ8NNNC_Spec.pdf
primary_photo: https://mm.digikey.com/Volume0/opasdata/d220001/medias/images/2537/Ceramic-Capacitor-CL-Series.jpg
product_description: CAP CER 10UF 6.3V X5R 0603
product_url: https://www.digikey.com/en/products/detail/samsung-electro-mechanics/CL10A106MQ8NNNC/3886777
search_timestamp: 1708598201
subcategory: Ceramic Capacitors

[eeintech]

Hello @NightSkySK

I am not very familiar with TME's API, is it supposed to return prices by default, or does it need some kind of parameter in the call to add them to the response? I would appreciate if you'd find more info regarding this and we could try to add it to a future release 😃

[NightSkySK]

I investigated a bit kintree\search\tme_api.py so fetching product info from TME API is running in three or four steps:

• /Products/GetProducts
• /Products/GetParameters
• /Products/GetPrices
• /Products/GetProductsFiles

First two are working without any problem, only /Products/GetPrices causing error [INFO] Warning: API data download failed (HTTP Error)

From my debug investigation. error is result of kintree\common\tools.py line 99 url_data = urllib.request.urlopen(url)

            url_data = urllib.request.urlopen(url)
            data = url_data.read()
            data_json = json.loads(data.decode('utf-8'))
            return data_json
    except (socket.timeout, requests.exceptions.ConnectTimeout, requests.exceptions.ReadTimeout):
        cprint(f'[INFO]\tWarning: {filetype} download socket timed out ({timeout}s)', silent=silent)
    except (urllib.error.HTTPError, requests.exceptions.ConnectionError):
        cprint(f'[INFO]\tWarning: {filetype} download failed (HTTP Error)', silent=silent)

and rise requests.exceptions.ConnectionError

I've checked the request body against TME API Manual and all seems to be correct. I've tried, also to replace /Products/GetPrices with Products/GetPricesAndStocks endpoint which are very similar but also the same error appears in the same moment.

I hope @T0jan will find some time to have a look on this case as I couldn't find root cause of this error.

[T0jan]

Sadly not a quick fix here. I took a look on it and the API reports an generic 400 error back without any explanation why, so there needs to be done some try and error here to figure out what the actual error is. Note: just dropping the currency parameter for the request does not work, it only changes the error to a 406.

[NightSkySK]

Intrestinly in TME Developers Sandbox I'm able to receive the respond 200 to the request:

documentation says that GrossPrice is optional but here seems to be passed in request, maybe this is the difference?

[T0jan]

@NightSkySK yeah I was able to get the same result in the sandbox too. adding the GrossPrice parameter to the request in kintree didn't change the outcome for me. My current idea is, that the server for whatever reason does not understand additional parameters like Currency. If I have some time I will investigate this further, but if you have the motivation feel free to share the results as well.

[NightSkySK]

Is it possible to print request generated urllib.request.Request(url, data, headers) to cURL or something similar?

[T0jan]

kinda, but I personally think its more productive to just run the command in an interactive python console and check the individual urllib objects right there.

[T0jan]

Finally found some time to get back to this and figured it out. The TME API for some reason needs the parameters to be in alphabetical order and they weren't for only the pricing request. I restructured the code a bit to account for this and allow for easier future expansion if needed.

[T0jan]

Was fixed in https://github.com/sparkmicro/Ki-nTree/releases/tag/1.1.4, closing here