gadicc
quoteCombine shouldn't batch in groups of more than 100 quotes
Bug Report
Describe the bug
See the logs
Mar 24 16:59:55 server docker[2453223]: stotra-backend | fetch https://consent.yahoo.com/v2/collectConsent?sessionId=3_cc-session_4dc902e0-5e3d-4b36-99d5-c311a4f7d997
Mar 24 16:59:55 server docker[2453223]: stotra-backend | fetch https://consent.yahoo.com/v2/collectConsent?sessionId=3_cc-session_4dc902e0-5e3d-4b36-99d5-c311a4f7d997
Mar 24 16:59:55 server docker[2453223]: stotra-backend | fetch https://guce.yahoo.com/copyConsent?sessionId=3_cc-session_4dc902e0-5e3d-4b36-99d5-c311a4f7d997&lang=fr-FR
Mar 24 16:59:55 server docker[2453223]: stotra-backend | Fetching crumb and cookies from https://finance.yahoo.com/quote/AAPL?guccounter=1...
Mar 24 16:59:55 server docker[2453223]: stotra-backend | We expected a redirect to guce.yahoo.com, but got https://finance.yahoo.com/quote/AAPL/
Mar 24 16:59:55 server docker[2453223]: stotra-backend | We'll try to continue anyway - you can safely ignore this if the request succeeds
Mar 24 16:59:55 server docker[2453223]: stotra-backend | Success. Cookie expires on Infinity
Mar 24 16:59:55 server docker[2453223]: stotra-backend | fetch https://query1.finance.yahoo.com/v1/test/getcrumb
Mar 24 16:59:55 server docker[2453223]: stotra-backend | New crumb: Ta3BCD/W2iw
Mar 24 16:59:56 server docker[2453223]: stotra-backend | Error: Connection is deallocated and cannot be used anymore.
Mar 24 16:59:56 server docker[2453223]: stotra-backend | at Object.<anonymous> (/usr/src/app/node_modules/yahoo-finance2/dist/cjs/src/lib/yahooFinanceFetch.js:108:27)
Mar 24 16:59:56 server docker[2453223]: stotra-backend | at Generator.next (<anonymous>)
Mar 24 16:59:56 server docker[2453223]: stotra-backend | at fulfilled (/usr/src/app/node_modules/yahoo-finance2/dist/cjs/src/lib/yahooFinanceFetch.js:5:58)
Mar 24 16:59:56 server docker[2453223]: stotra-backend | at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
Minimal Reproduction
Perform many requests simultaneously
Environment
Browser or Node: node
Node version (if applicable): 23.10.0
Npm version:
Browser verion (if applicable):
Library version (e.g. 1.10.1): 2.13.3
Additional Context
Hey @Mubelotix, thanks for the report.
Granted this should be clearer, but the error above is actually coming from Yahoo's servers, not the library.
Probably a transient error that will fix itself by tomorrow, however, even if you found a way to reproduce it consistently after that, there's not much we can do it about it.
It's still surprising since we have our own concurrency protection in the library.
When you say "Perform many requests simultaneously", is it happening in a single instance or in many instances simultaneously?
Regardless, suggest waiting 24 hrs and see if the issue resolves itself.
Thank you for your answer
the error above is actually coming from Yahoo's servers
This is what I suspected, but when I use yahoo-finance2 as a standalone program rather than a library, it works, so I guess it's not about my IP getting banned
When you say "Perform many requests simultaneously", is it happening in a single instance or in many instances simultaneously?
There should be only one instance, trying to load a 609 stocks all at the same time
Regardless, suggest waiting 24 hrs and see if the issue resolves itself.
Unfortunately it has already been 10 days, and it does not seem to resolve
This code reproduces the error
const yahooFinance = require('yahoo-finance2').default;
var symbols = ["SUI","SAFRF","R","TTWO","APLE","EI4.F","AM.PA","VOLV-B.ST","GEHC","NUSI","GTX","CZFS","SHIB-USD","FAS","BIT","ERIC-B.ST","UBI.PA","NA","JCI","GCARSOA1.MX","AVGO","AMZN","VOLCAR-B.ST","^FCHI","XEYX.DE","3CP.SG","CAT.JO","VXX","ARKX","BLK","DPZ","TRUMP27872-USD","^VIX","SOL-EUR","ZBH","PNLF.JK","SYN12147-USD","QUAL","KVYO","ALHE.TA","BRK-B","FA","ETH-USD","GS","MBLY","0P0001Q7EY.L","BTBT","BSM","GRAL","LHX","TEL","AI","JTO-USD","0P0001PZVE.SS","T","SONY","VTI","OR","CLF","PFE","NYC","BAC","VNPKF","GD","ACX.MC","RACE.MI","MRG.F","CRM","SAB.VN","AIL.F","X","APP","QCOM","AUCHAZ.XC","CACI","AMGN","ENS-USD","LMT","PWR","CRON","MLN","JPY=X","BIL","RIBBU","XIACY","SAF.PA","PLTR","INTC","XIAOMI80.BK","PEPU-USD","BAYN.DE","CEG","TSLA","BSN.DU","XAUT-USD","AMRN","RKLB","^HSI","CJMB","MSFT","SPG","ACXP","RBLX","SRE","HO.PA","MSTX","GIL","GILT","NBIX","CLOV","PNNT","RL","YOU","TNK","2552.T","DG.PA","BA.L","VRTX","DOGE-USD","WINA","JWN","AF.PA","WAY","VOO","BLWYF","0P00000I7O.F","8035.T","2628.HK","TREE","AAPU","F","AM","NEE","LINK-USD","BTC-USD","EWLD.PA","IBIT","GOLD","HBAR-USD","RHM.DE","EZJ.L","CURYY","EAT","0P0001NHLD","RST.L","S","0857.HK","WBA","BNB-USD","7844.T","TOA.BK","DOBUR.IS","PNG.V","ART.MC","HMAX.TO","UI","NOC","VWS.CO","CPRX","PNL.AS","CIGI","XRP-EUR","MCD","PINE","NCT-USD","GST16352-USD","IVV","BONK-USD","NUKZ","437070.KS","0P0000UD0U.SA","CW","NRJ.PA","LRCX","HHH","KINS","URW.PA","NEAR-USD","FGR.PA","NQ=F","^NDX","PEP","INSP","IPH.XA","QUIK","AVAX-USD","USMV","CHRW","SAFRY","GLE.PA","NOVO-B.CO","FR.PA","BTC","WBD","CMG","MDLZ","AMD","PEGA","CRYP.AX","^STI","MOL.BD","KODK","BCBP","AMAT","STORJ-USD","PATA.BA","NET","FM.TO","TMUS","ARM","VT","LSE","H61N.F","HAL","NVD.DE","EMIRATESNBD.AE","163280.KQ","AAVE-USD","SPXC-USD","EURNOK=X","AAPGV","BMO.TO","0P0001M1XT.SA","TXN","PEPE24478-USD","INIKF","SPY","MC.PA","NEO","FP.VI","NASD11.SA","TWST","FNGU","OKLO","ZAR=X","MIMI","JPM","MTUM","ALDX","SEDG","OVV","AZREF","PEPECOIN-USD","ELIS.PA","QBTS","LEVI","MUFG","TAC","BAYRY","COP","FRCB","TCEHY","ML.PA","4573.TWO","SBRY.L","IRDM11.SA","SUI20947-USD","FROG.PA","REAX","NTES","MRK","ABBV","UBSFY","FIL-USD","ADA-USD","RSI","MAT","INDF.JK","AU.BK","DODO-USD","CWEN-A","AIR.PA","3YB0.F","WFC","MATIC-USD","BN.PA","DTE.DE","PTEN","MSI","WULF","SOL-USD","BNP.PA","DOT-EUR","CNI.AX","GLD","DJT","PAYX","NVDA","XRP-USD","SONYN.MX","AMAL","000270.KS","TFC","RYCEY","URNM","4612.T","BA","QCLN","VEEE","EDF","035900.KQ","VOLT","DOT-USD","OR.PA","TUPRS.IS","RGTI","BVN","CLM","IBM","MAVI.IS","0P00015EIW.F","NKE","^DJI","MRNA","DIS","NEXN","OPK","DBK.DE","SU.PA","M","MLAZR.PA","BRD.RO","TJX","NLR","600808.SS","CON","ZGN","AVO","GNK","TOEAF","TTE.PA","SSSLH.FGI","459580.KS","MNST","^GSPC","^TNX","0P0001E4NS.TW","LSEG.L","VGA.BE","PANW","MA","05Y.F","APPS","D","FLR-USD","CSF0.MU","NNOX","BRISA.IS","LSCC","PL","JNJ","META","NESN.SW","SPMO","SPXL","CEIEF","SHOP","AMR","OPENAIERC-USD","GGAL","KO","ARQQ","6752.T","QTUM","BFIT.AS","TSM","6952.T","FSS","FISI","TWLO","0AJ.F","PYPL","LVMHF","MKSI","0NWX.IL","GIGA30063-USD","O39.SI","EE","XNO-USD","GILD","CWT","LW","RTX","T.TO","DOW","PLL","^MXX","CTX-USD","RIOT","ASML","DTL.AX","BLDP","GSL","EURUSD=X","MS","MIR","032830.KS","ACA.PA","BCG.L","APD","2357.TW","PLUX.VI","NUE","3294.TWO","POKEMON-USD","0P00000F04.ST","SNY","ALO.PA","NRG","NMRA","MBG.DE","VOW3.DE","TTE","AUPH","AAL","TKNSA.IS","MAA","2611.HK","MLM","SNOW","U11.SI","BMW.DE","BZAI","SOH1.MU","V","OA41.BE","PORNROCKET-USD","GRMN","CRNC","GOOG","DAL","GEV","FDX","AMEM.DE","JSPR","PI","AAPL","CRWD","PAH3.DE","SM","SRU-UN.TO","PRU","MTTR","GFRDL.XC","TMC","QRVO","MANU","VFF","URTH","BNTX","IP","XOF=X","RDFN","BTC-EUR","FNMA","GRAB","^KLSE","ZURN.SW","ORCL","CMA","ORA.PA","KLAC","CS.PA","1810.HK","BKNG","LUCK","MERC","DAR","IHAAS.IS","MSTR","FI","TRX-USD","LUMN","ADS.DE","SOWG","FR","IONQ","LAES","SYBT","MSCI","VGT","MMM","TAO22974-USD","BBAI","NFLX","DEXE-USD","NNDM","MDT","SGO.PA","GENI.L","TOT.TO","ETHE","GOOGL","688026.SS","CIEN","LUIGI-USD","AI.PA","DOP=X","DRUG","DEC.PA","MCL-USD","SNN","MITK","ASST","JMAT.L","MINT","DKS","WMG","NCH2.DE","WTMAU","YM=F","MRVL","DSY.PA","HMC","STNE","COIN","FRA","GNG.AX","SAP","AMOI.L","XOM","005930.KS","FDJ.PA","UCO","LIKPX","USD","HVII","MDB","HUBS","^IXIC","DB","WEN","RCL","TM","SAN.PA","RNO.PA","STM","TDOC","BCC","ES=F","0P0000704P.TO","NNE","WHD","DGICA","KOF","JBL","WMT","ZIM","0P00019LWZ.F","GE","SF","ATMU","OFN-USD","TNSR-USD","0RMP.L","KRYS","LTRN","DG","MU","AR","AXON","^GDAXI","MILIF","CVX","COMP","UBS","ALK","JSAIY","KMI","SPOT","VEQT.TO","UBER","AIR","NVO","ROLEXRINGS.NS","CHIAX","BRIOF","TRB-USD","BKHA","FTNT","2222.SR","ALV.DE","CROX","FARM","HOOD","RMS.PA","^RUT","RACE","7QZ.F","ASELS.IS","5202.KL","EACW.MI","REGN","TESL","DCI","0P0001KQTS.F","PLG","OPEN","PENGU34466-USD","AGLD-USD","SA","OXY","LFGY","SBEV","NTDOY","SE","ADBE","BVI.PA","COST","LKNCY"];
async function getStockData(symbol) {
try {
const result = await yahooFinance.quoteCombine(symbol, {
fields: [
"regularMarketPrice",
"regularMarketChangePercent",
"longName",
"regularMarketPreviousClose",
"quoteType",
"averageDailyVolume10Day"
],
});
console.log(result);
} catch (error) {
console.error('Error fetching stock data:', error);
}
}
for (let symbol of symbols) {
getStockData(symbol);
}
I was able to work around this by grouping quotes in groups of 100 myself. This should be done by the library already, so I'm leaving this issue open.
Hey @Mubelotix, well spotted and thanks for all your valued investigation and input here 🙏
I'm currently working on the next major release - v3 - and hope to address this as part of that release.
Thanks for your patience, everyone.
This has been addressed in the above commit. It will be in the next v3 release, and there will be an automated message here once that release has been published.
After it's been published, you'll be able to see the new docs relating to this feature at https://jsr.io/@gadicc/yahoo-finance2/doc/other/quoteCombine. In the meantime, you could look at the jsdocs source in quoteCombine.ts.
Although v3 is not yet "official", nor marked with the @latest tag, many are using it succesfully. For more info on upgrading, see UPGRADING.md. Unfortunately, I don't have the capacity to backport fixes or code back to v2.
Thanks again to @Mubelotix for your research into this, and patience.
:tada: This issue has been resolved in version 3.8.0 :tada:
The release is available on:
Your semantic-release bot :package::rocket:
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}