Too many requests error

[brent80]

Anyone else getting this error?

or

Just me?

[eddelbuettel]

I do too, since yesterday. Similar for another Yahoo! Finance accessor, though the Python package yfinance still works.

The cookie/crumb treatments seems to have changed at their end.

[eddelbuettel]

This issue in the other (R language) repo referenced above was fixed in this commit setting additional ssl cipher headers. Maybe that's an option here too?

[mictrik]

Yes, I am getting "Unkown Error - Too Many Requests" in place of data. In the beginning it was intermittent. I could refresh and every 3rd or 4th time I would get data. Now just the error.

Yahoo API Change? It would effect many people I would think if that is thecase

Ubuntu 24.04.2 LTS Intel® Core™ i7-4770K × 8 32Gb Gnome 46 X11

[cinatic]

^^ what a timing...

@eddelbuettel thanks for the details.

it looks like this is nothing which can be solved with libsoup. I need to check if I can replace it with the curl-impersonate somehow, shouldn't be so complicated

[eddelbuettel]

@cinatic Pleasure, always glad to help. I am not that familiar with Python details but I have been told that the switch from the 'request' library to 'curl_cffi' was motivated by the need to adjust the request headers -- but that is really just second-hand knowledge I am passing along. And yep, 'curl-impersonate' may help in a similar fashion.

[cinatic]

@eddelbuettel yeah good point, actually I oversaw that they mentioned that setting the desired cipher set was sufficient for them. maybe there is a way to do this also with libsoup I saw something but it's not part of the request headers

[eddelbuettel]

Yes that seems to be the (surprisingly !!) narrow solution for the quantmod package I use in R.

[mrpeabody]

@cinatic I've prepared a patch with the curl-impersonate, seems working fine -- please see attached. This assumes curl-impersonate is installed (in my case, the AUR package is used).

curl-fix.patch.txt

[bruce-awareit]

@cinatic I've prepared a patch with the curl-impersonate, seems working fine -- please see attached. This assumes curl-impersonate is installed (in my case, the AUR package is used).

curl-fix.patch.txt

@mrpeabody Thanks. The patch works for me on gnome 48 at Arch Linux, too.

[mictrik]

Ubuntu 24.04 LTS with Gnome 46 and cURL 8.5. Just so I understand should i install cURL impersonate before running patch? i don't use cURL but cur-impersonate and it does not seem to be a part of the cURL pkg. ios there a version or method of cURL-impersonate i should preferentially install?

[eddelbuettel]

There are binaries in the releases section of the curl-impersonate repo.

But I am scratching my a little whether calling an external process every few seconds is really the best way about it.

[mictrik]

Thank you for the link. What other approach are you thinking about? These changes may become more frequent as time go's on. if they don't see a certain fingerprint . i agree that introducing another process is never ideal. I wonder if using a supported API from another provider is like Alpha Adv or FInnhub etc.. would be a way forward.

[mrpeabody]

Nothing lasts forever 🙂 Let's cross that bridge when we get there, but as of now, I didn't want to leave people hanging -- the patch could be a good stopgap solution (thus I did not create a pull request) until a better fix is available.

[mictrik]

Absolutely agree, I was just curious what he had in mind. Your work is fantastic by the way, I am learning slowly by looking at it.

[eddelbuettel]

Apparently one of the apps that was also broken (R package quantmod and its getQuote() function) now works again which I can confirm. No luck with Stocks Extension though.

[cement-head]

@eddelbuettel I don't understand how to install curl-impersonate; can someone provide detailed instructions?

[eddelbuettel]

@cement-head This should work https://github.com/lwthiker/curl-impersonate/releases

[mrpeabody]

@cement-head if your distro doesn't have it, this might be the easiest way:

wget https://github.com/lwthiker/curl-impersonate/releases/download/v0.6.1/curl-impersonate-v0.6.1.x86_64-linux-gnu.tar.gz
mkdir curl && tar xzf curl-impersonate-v0.6.1.x86_64-linux-gnu.tar.gz -C curl
sudo cp -r ./curl/* /usr/bin/

then just do clean up:

rm -rf ./curl*

Test that it worked:

$ curl_chrome131
curl: (2) no URL specified
curl: try 'curl --help' for more information

[blue4system]

FYI this flag: GLib.SpawnFlags.DO_NOT_REAP_CHILD creates a ton of defunct processes in my system, so in helpers/fetchImpersonate.js I ahd to change the code as follows:

//const [success, pid, stdinFd, stdoutFd, stderrFd] = GLib.spawn_async_with_pipes(null, args, null, GLib.SpawnFlags.DO_NOT_REAP_CHILD, null )

const [success, pid, stdinFd, stdoutFd, stderrFd] = GLib.spawn_async_with_pipes(null, args, null, GLib.SpawnFlags.SEARCH_PATH , null )

[eddelbuettel]

NB fetchImpersonate.js is from the patch suggested above using calls to curlimpersonate.

Anybody have any luck trying other patches now that backend seems to have changed?

[cinatic]

so I looked into soup and there is actually no way to have impact on the cipher stuff or at least not an obvious way.

So i have also created a version using curl-impersonate, but I am relying on Gio.Subprocess, which seems a bit leaner than the version provided by @bruce-awareit thanks, by the way! Using Subprocess was just the first thing that came to mind, so I’m not sure if it’s actually better, but it works well on my laptop. I’ll most likely create a PR over the weekend.

I really don't like this approach either but as this whole thing won’t work without it anyway I think there is no other way

[cement-head]

@mrpeabody Okay, curl_impersonate works up to curl_chrome116, but not curl_chrome131

[mrpeabody]

@mrpeabody Okay, curl_impersonate works up to curl_chrome116, but not curl_chrome131

@cement-head cool, modify the fetchImpersonate.js to use whatever version that works for you, and that should be it!

[LDB-Crunch]

@mrpeabody Okay, curl_impersonate works up to curl_chrome116, but not curl_chrome131

@cement-head cool, modify the fetchImpersonate.js to use whatever version that works for you, and that should be it!

This was the missing piece of the puzzle for me, thanks!

[cement-head]

@mrpeabody Could you explain how to install/apply the curl-fix.txt patch?

[mrpeabody]

@cement-head steps include:

• clone this repro: git clone https://github.com/cinatic/stocks-extension.git

• navigate to the repo: cd stocks-extension

• download and modify the patch file to have your chrome version (as discussed above), for example: wget https://github.com/user-attachments/files/20137029/curl-fix.patch.txt

• apply the patch: git apply ./curl-fix.patch.txt

• if you had an extension installed, remove the old one: rm -rf ~/.local/share/gnome-shell/extensions/[email protected]

• copy the [email protected] directory to ~/.local/share/gnome-shell/extensions directory: cp -r [email protected] ~/.local/share/gnome-shell/extensions

• log out and log in back into your GNOME session

[cement-head]

This is what I get when I try to start the extension:

SyntaxError: import declarations may only appear at top level of a module

Stack trace:
  _init@resource:///org/gnome/Shell/Extensions/js/extensionPrefsDialog.js:23:33
  ExtensionPrefsDialog@resource:///org/gnome/Shell/Extensions/js/extensionPrefsDialog.js:10:4
  OpenExtensionPrefsAsync/<@resource:///org/gnome/Shell/Extensions/js/extensionsService.js:129:33
  asyncCallback@resource:///org/gnome/gjs/modules/core/overrides/Gio.js:115:22
  run@resource:///org/gnome/Shell/Extensions/js/dbusService.js:186:20
  main@resource:///org/gnome/Shell/Extensions/js/main.js:22:13
  run@resource:///org/gnome/gjs/modules/script/package.js:206:19
  start@resource:///org/gnome/gjs/modules/script/package.js:190:8
  @/usr/share/gnome-shell/org.gnome.Shell.Extensions:1:17

[mrpeabody]

@cement-head I think you're using an ancient version of GNOME (44? older?) -- you might have to upgrade. I don't have an old version like this so I won't be able to adjust the patch to some older release of the stocks extension

[howard-mahe]

Thanks a lot for your work. I'm running Stock Extensions v24 for GNOME Shell 42 on Ubuntu 22.04.

I managed to tweak the patch to update the main_GS_42 branch (corresponding to the v24.2-GS42 tag for GNOME Shell 42). @cement-head If it's helpful, here are the changes I made: https://github.com/cinatic/stocks-extension/compare/main_GS_42...howard-mahe:stocks-extension:main_GS_42

There aren’t many differences from curl-fix.patch.txt. As far as I remember, I mainly updated the imports in fetchImpersonate.js to match the previous JS version. I also changed the signature of the fetch function from export const to var, and set the impersonate argument to "chrome116". Also, I had to run a make install to build the gschemas.compiled (but it has nothing to do with the patch).

[cement-head]

@mrpeabody Yep, Gnome 42.9 (Pop_OS! 22.04 LTS). I'll try @howard-mahe 's fix.