429 Too Many Requests

[yasinaslan16]

We get the same error from two different servers. Error 429 Too Many Requests returns when pulling data..

No problem on link: https://query1.finance.yahoo.com/v1/test/getcrumb

Fatal error: Uncaught GuzzleHttp\Exception\ClientException: Client error: GET https://query2.finance.yahoo.com/v1/test/getcrumb resulted in a 429 Too Many Requests response: Too Many Requests in /home/mytrnuwm/public_html/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113 Stack trace: #0 /home/mytrnuwm/public_html/vendor/guzzlehttp/guzzle/src/Middleware.php(65): GuzzleHttp\Exception\RequestException::create(Object(GuzzleHttp\Psr7\Request), Object(GuzzleHttp\Psr7\Response)) #1 /home/mytrnuwm/public_html/vendor/guzzlehttp/promises/src/Promise.php(204): GuzzleHttp\Middleware::GuzzleHttp{closure}(Object(GuzzleHttp\Psr7\Response)) #2 /home/mytrnuwm/public_html/vendor/guzzlehttp/promises/src/Promise.php(153): GuzzleHttp\Promise\Promise::callHandler(1, Object(GuzzleHttp\Psr7\Response), NULL) #3 /home/mytrnuwm/public_html/vendor/guzzlehttp/promises/src/TaskQueue.php(48): GuzzleHttp\Promise\Promise::GuzzleHttp\Promise{closure}() #4 /home/mytrnuwm/public_html/vendor/guzzlehttp/promises/src/Promise.php(248): Gu in /home/mytrnuwm/public_html/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php on line 113

[brightq]

We have exactly same issue.

[scheb]

Seem to be rate-limiting or even blocking scripted user-agents (such as curl or the Guzzle client) more restrictively now. Not sure why the tests on Github are working fine, maybe it's a special treatment for certain networks 🤷. Randomizing the user agent seems to do the job for now -> Release 4.8.0

[topclaudy]

The ‘Error 429: Too Many Requests’ has returned. It looks like Yahoo is actively ramping up their efforts to block automated traffic.

[scheb]

Yes, I've also recognized that. It seems they're also using TLS fingerprinting now, to distingish real browsers from API clients. The best way around this issue that I've found so far is to replace the CURL library in your application with curl-impersonate, which is capable to fully impersonate the appearance of common web browsers. Something that cannot be done with the normal CURL library.

I've configured it in my stock-panel application (see https://github.com/scheb/stock-panel/commit/0a86fd30a3d3b52ee563894905ee00eecea73292), so that it emulates a Chrome 116 browser, which seems to do the job for now.

[netcarlos]

I ran some tests using curl-impersonate and managed to eliminate error code 429. The 'crumb' was successfully fetched, but a subsequent call yields this response:

{"finance":{"result":null,"error":{"code":"Unauthorized","description":"User is unable to access this feature - https://bit.ly/yahoo-finance-api-feedback"}}}

[scheb]

@netcarlos Seems unrelated, I haven't come across this issue yet. If I have to guess, you might get geo-blocked.

[ovidiurosu90]

I can confirm that curl-impersonate solves this problem. Thank you @scheb for the solution.

This is how I did it (web-app on Ubuntu):

### Install curl-impersonate to avoid '429 Too Many Requests' responses
mkdir ~/curl-impersonate/
cd ~/curl-impersonate/
wget https://github.com/lwthiker/curl-impersonate/releases/download/v0.6.1/libcurl-impersonate-v0.6.1.x86_64-linux-gnu.tar.gz
tar -xf libcurl-impersonate-v0.6.1.x86_64-linux-gnu.tar.gz

sudo su
cd /usr/local/lib/
ln -s [USER_HOME]/curl-impersonate/libcurl-impersonate-chrome.so .
ls -la /usr/local/lib/libcurl-impersonate-chrome.so

### Set the user agent before you call the API
vim FinanceAPI.php

class FinanceAPI
 {
+    private const USER_AGENT_CHROME_116 = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36';
+
     public function __construct()
     {
+        // Deal with '429 Too Many Requests' errors, use curl_impersonate
+        UserAgent::setUserAgents([self::USER_AGENT_CHROME_116]);


### Calling the API via a cronjob
sudo su
crontab -e
* * * * * su - www-data -s /bin/bash -c "export LOG_CHANNEL=stdout; export LD_PRELOAD=/usr/local/lib/libcurl-impersonate-chrome.so; export CURL_IMPERSONATE=chrome116; php artisan app:finance-api-cron

### Calling the API in a request from the browser
vim public/.htaccess
+SetEnv LD_PRELOAD /usr/local/lib/libcurl-impersonate-chrome.so
+SetEnv CURL_IMPERSONATE chrome116

[FabienVINCENT]

If anyone has this problem on symfony, I solved it like this because it didn't work with curl-impersonate

#services.yaml

    # Yahoo Finance API Client
    Scheb\YahooFinanceApi\ApiClient:
        factory: ['Scheb\YahooFinanceApi\ApiClientFactory', 'createApiClient']
        arguments:
            $clientOptions:
                headers:
                    User-Agent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36'
                    Accept-Language: 'en-US,en;q=0.9'
                    Accept-Encoding: 'gzip, deflate, br'
                    Connection: 'keep-alive'
            $retries: 3
            $retryDelay: 1000

    public function __construct(
        private ApiClient $yahooFinanceClient,
    ) {}