get_ticker_info(): getcrumb(): "Too Many Requests"

[7jrxt42BxFZo4iAnN4CX]

My toml: yahoo_finance_api = { path = "path/to/yahoo_finance_api"} reqwest = { version = "0.12.19", features = ["socks", "gzip", "brotli", "deflate", "json", "rustls-tls"] } tokio = { version = "1.45.1", features = ["rt-multi-thread", "sync", "fs", "io-util", "io-std"] }

cargo test --package my_project --lib -- test_api::my_test_fn --exact --show-output
cookie: Some("A3=d=AQABBF08R2gCELvQV44LoruVOdGxOcL6McMFEgEBAQGNSGhRaJ6PzyMA_eMCAA&S=AQAAApS3Gb3xTd4mXTIt6BdI-AM; Expires=Wed, 10 Jun 2026 01:56:13 GMT; Max-Age=31557600; Domain=.yahoo.com; Path=/; SameSite=None; Secure; HttpOnly")
get_crumb status: 429 Too Many Requests, body: "Too Many Requests"
get_crumb status: 429 Too Many Requests, body: "Too Many Requests"

However, running the test_get_crumb() test inside the project works and returns crumb. Updated dependencies inside yahoo_finance_api, still works.

cargo test --package yahoo_finance_api --lib -- async_impl::tests::test_get_crumb --exact --show-output
cookie: Some("A3=d=AQABBAw8R2gCEPBw4O-0VQbRI4zeNvLH-8wFEgEBAQGNSGhRaJ6PzyMA_eMCAA&S=AQAAAkTc4Axcb8CPm24BjX6mjco; Expires=Wed, 10 Jun 2026 01:54:52 GMT; Max-Age=31557600; Domain=.yahoo.com; Path=/; SameSite=None; Secure; HttpOnly")
get_crumb status: 200 OK, body: "qyHTJ5cFcOw"

Solution: (add default-features = false to reqwest): reqwest = { version = "0.12.19", default-features = false, features = ["socks", "gzip", "brotli", "deflate", "json", "rustls-tls", "cookies", "http2", "charset", "system-proxy"] }

For those who have encountered this

[me60732]

thankyou, for a while there I thought I'd have to build my own Yahoo client, i've already built one in node, I don't want to do it again

[tirithen]

I'm having problems even after the reqwest setup. Too Many Requests.

[7jrxt42BxFZo4iAnN4CX]

I'm having problems even after the reqwest setup. Too Many Requests.

Try run test_get_crumb() test

[tirithen]

Thanks @7jrxt42BxFZo4iAnN4CX

Running the tests produces variable failures on different runs, possibly due to to frequent calls when running all tests. The test that fails most often is the test_get_crumb() test. Maybe that end point is the one deciding of the client is trusted or not.

I could see similar failures on both xemwebe:master and 7jrxt42BxFZo4iAnN4CX:crumb_error.

Every now and then the test_get_crumb() does succeed so probably the patterns are correct, but some audit system at yahoo kicks in on most calls?

❯ cargo test
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.07s
     Running unittests src/lib.rs (target/debug/deps/yahoo_finance_api-a12cb2d51772b8d0)

running 25 tests
test async_impl::tests::test_get_crumb ... FAILED
test async_impl::tests::test_get_cookie ... ok
test async_impl::tests::test_get_quote_range ... ok
test async_impl::tests::test_get_single_quote ... ok
test async_impl::tests::test_api_responses_missing_fields - should panic ... ok
test quotes::tests::test_deserialize_f64_special ... ok
test quotes::tests::test_deserialize_period_info ... ok
test quotes::tests::test_deserialize_trading_periods_complex ... ok
test quotes::tests::test_deserialize_trading_periods_complex_regular_only ... ok
test quotes::tests::test_deserialize_trading_periods_simple ... ok
test async_impl::tests::test_mutual_fund_range ... ok
test async_impl::tests::test_get_metadata ... ok
test async_impl::tests::test_large_volume ... ok
test async_impl::tests::test_mutual_fund_latest ... ok
test async_impl::tests::test_mutual_fund_history ... ok
test async_impl::tests::test_get_quote_history ... ok
test async_impl::tests::test_get_quote_history_interval ... ok
test async_impl::tests::test_get_quote_period_interval ... ok
test async_impl::tests::test_mutual_fund_latest_with_null_first_trade_date ... ok
test async_impl::tests::test_search_ticker ... ok
test async_impl::tests::test_mutual_fund_capital_gains ... ok
test async_impl::tests::test_wrong_request_get_quote_history_interval - should panic ... ok
test async_impl::tests::test_strange_api_responses ... ok
test async_impl::tests::test_get_ticker_info ... FAILED
test async_impl::tests::test_neg_time_stamp ... ok

failures:

---- async_impl::tests::test_get_crumb stdout ----

thread 'async_impl::tests::test_get_crumb' panicked at src/async_impl.rs:508:48:
called `Result::unwrap()` on an `Err` value: ConnectionFailed(reqwest::Error { kind: Request, url: "https://fc.yahoo.com/", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 101, kind: NetworkUnreachable, message: "Network is unreachable" })) })
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- async_impl::tests::test_get_ticker_info stdout ----

thread 'async_impl::tests::test_get_ticker_info' panicked at src/async_impl.rs:492:36:
called `Result::unwrap()` on an `Err` value: ConnectionFailed(reqwest::Error { kind: Request, url: "https://query2.finance.yahoo.com/v10/finance/quoteSummary/AAPL?modules=financialData,quoteType,defaultKeyStatistics,assetProfile,summaryDetail&corsDomain=finance.yahoo.com&formatted=false&symbol=AAPL&crumb=ejwI.xK.NeW", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 101, kind: NetworkUnreachable, message: "Network is unreachable" })) })


failures:
    async_impl::tests::test_get_crumb
    async_impl::tests::test_get_ticker_info

test result: FAILED. 23 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.85s

error: test failed, to rerun pass `--lib`

[tirithen]

After some more testing it seems like 7jrxt42BxFZo4iAnN4CX:crumb_error succeeds each time I run only cargo test test_get_crumb, and the same single test mostly fails on xemwebe:master (even if it succeeds some times).

So likely the branch has some improvements.

However when I use the 7jrxt42BxFZo4iAnN4CX:crumb_error fork in a test application I still get a too many requests error TooManyRequests("GET https://query1.finance.yahoo.com/v1/test/getcrumb in get_crumb") even when only calling get_ticker_info() once so there seem to still be something missing, or those end points remain more sensitive.

let mut provider = YahooConnector::new()?;
let data = provider.get_ticker_info("TSLA").await?;
dbg!(&data);

[tirithen]

Nevermind, it seems like there is a conflict somehow with the ollama-rs crate, possibly it is competing with TLS setup or similar.

In a clean project get_ticker_info works until I add ollama-rs = "0.3.1" as a dependency. Then I get too many requests. Removing the dependency will have the yahoo_finance_api call working again.

Solution is to add ollama-rs with rustls, then it works again:

ollama-rs = { version = "0.3.1", default-features = false, features = ["rustls"] }

So it seems that when combining with crates also using reqwest it is important to remember to try default-features = false on those to prevent the default-tls feature to be used.

[me60732]

Why is this such a complicated issue, for the first request made with a new instance of yahoo connector it has to make 3 quick requests, one to get the crumb, another to get the cookie, then another to actually look up the code, this is done in very quick succession, which yahoo finance api v10 picks up on a simple sleep(60) at the end of each crumb, and and get cookie, solves this problem, which yes slow, only has to be done for a single instance, second the cookie value needs to be checked and retrieved first not the crumb, because the value in the cookie determines the value of the crumb!

[7jrxt42BxFZo4iAnN4CX]

@tirithen The only way to find the cause is to create an empty project. ollama-rs uses reqwest.

I think the problem appeared with the commit b6eb4314b6c38e719d01588df417d1d54b3fa68e. So another solution is to try to remove these changes (by abandoning rustls).

And even better, add the ability to select rustls with a flag. There is no time for this yet, maybe someone will do it.