Other modules: tracking progress

[gadicc]

Thanks to the awesome @roblav96 for all this beautiful work back over at https://github.com/pilwon/node-yahoo-finance/issues/43 (in 2017 :sweat_smile:) Have added status [X]'s on each point to track progress, followed by any additional comments below.

Hey man, love what you're doing here. I've been reverse engineering Yahoo's API recently and thought I'd provide some of my findings.

[n/a] Crum Manager

sstrickx/yahoofinance-api @sstrickx wrote a crum manager in Java.

As reported by @smeijer in https://github.com/pilwon/node-yahoo-finance/issues/73 (thanks again!) and confirmed by me (and @MichaelDeBoey?), this is no longer required.

[X] Recommended Symbols

GET https://query1.finance.yahoo.com/v6/finance/recommendationsbysymbol/PLUG

Thanks, @pudgereyem, who added on 2021-02-09 in #28.

[ ] Daily Gainers

GET https://query2.finance.yahoo.com/v1/finance/screener/predefined/saved?formatted=false&lang=en-US&region=US&scrIds=day_gainers&count=5&corsDomain=finance.yahoo.com

[X] Search Autocomplete

GET https://autoc.finance.yahoo.com/autoc?query=PLUG&region=1&lang=en

[ ] Symbol Conversations

GET https://finance.yahoo.com/_finance_doubledown/api/resource/canvass.getMessageList;count=20;oauthConsumerKey=finance.oauth.client.canvass.prod.consumerKey;oauthConsumerSecret=finance.oauth.client.canvass.prod.consumerSecret;query=namespace = "yahoo_finance" and (tag = "PLUG");region=US;sortBy=createdAt;userActivity=true

[ ] Symbol News

GET http://feeds.finance.yahoo.com/rss/2.0/headline?s=PLUG&region=US&lang=en-US Returns in RSS format, couldn't figure out how to get JSON.

[X] Multiple Symbol Quote

GET https://query2.finance.yahoo.com/v7/finance/quote?symbols=NVDA,PLUG,AMD

[ ] Multiple Symbol Historical Charts

GET https://query1.finance.yahoo.com/v7/finance/spark?symbols=PLUG,AMD&range=1d&formated=true Only returns close data though =/

[X] Trending Symbols

GET https://query1.finance.yahoo.com/v1/finance/trending/US?lang=en-US&region=US&count=5&corsDomain=finance.yahoo.com

Thanks, @PythonCreator27, who added this on 2021-02-18 in #66.

[ ] Symbol Options

GET https://query1.finance.yahoo.com/v7/finance/options/[symbol]

Found by @PythonCreator27, more details below in https://github.com/gadicc/node-yahoo-finance2/issues/8#issuecomment-802463467.

Live Streamer

import * as Promise from 'bluebird'
import * as Nstream from 'stream'
import * as Nhttp from 'http'
import * as Nnet from 'net'
import * as Axios from 'axios'

const YH_IDK_KEY = {
	a00: 'Ask',
	a50: 'AskSize',
	b00: 'Bid',
	b60: 'BidSize',
	c10: 'Change',
	c63: 'ChangeRealtime',
	c64: 'DisputedChangeRealtimeAfterHours',
	c85: 'ChangeRealtimeAfterHours',
	c86: 'PercentChangeRealtimeAfterHours',
	g53: 'DayLow',
	h53: 'DayHigh',
	j10: 'MarketCap',
	l84: 'PriceRealtime',
	l86: 'PriceRealtimeAfterHours',
	p20: 'PercentChange',
	p43: 'PercentChangeRealtime',
	p44: 'PercentChangeRealtimeAfterHours',
	t53: 'DisputedTimestampForCommodities',
	t54: 'DisputedTimestampForStocks',
	v53: 'Volume',
	v00: 'Volume2',
	l10: 'LastSalePrice',
	t10: 'LastSaleTime',
	l90: 'EcnQuoteLastValue',
}
const YH_KEY_IDK = {
	Ask: 'a00',
	AskSize: 'a50',
	Bid: 'b00',
	BidSize: 'b60',
	Change: 'c10',
	ChangeRealtime: 'c63',
	DisputedChangeRealtimeAfterHours: 'c64',
	ChangeRealtimeAfterHours: 'c85',
	PercentChangeRealtimeAfterHours: 'p44',
	DayLow: 'g53',
	DayHigh: 'h53',
	MarketCap: 'j10',
	PriceRealtime: 'l84',
	PriceRealtimeAfterHours: 'l86',
	PercentChange: 'p20',
	PercentChangeRealtime: 'p43',
	DisputedTimestampForCommodities: 't53',
	DisputedTimestampForStocks: 't54',
	Volume: 'v53',
	Volume2: 'v00',
	LastSalePrice: 'l10',
	LastSaleTime: 't10',
	EcnQuoteLastValue: 'l90',
}

let total = 0
let stream = null as Nhttp.IncomingMessage
function startYahooStream() {
	return Promise.resolve().then(function() {
		let keys = [
			YH_KEY_IDK.Ask,
			YH_KEY_IDK.AskSize,
			YH_KEY_IDK.Bid,
			YH_KEY_IDK.BidSize,
			YH_KEY_IDK.Volume,
			YH_KEY_IDK.Volume2,
			YH_KEY_IDK.PriceRealtime,
			YH_KEY_IDK.PriceRealtimeAfterHours,
			YH_KEY_IDK.LastSalePrice,
			YH_KEY_IDK.EcnQuoteLastValue,
			YH_KEY_IDK.LastSaleTime,
			YH_KEY_IDK.DisputedTimestampForStocks,
		]
		return Axios.default.get('https://streamerapi.finance.yahoo.com/streamer/1.0', {
			params: {
				s: 'NVDA,PLUG,AMD',
				k: keys.join(','),
				r: 0,
				callback: 'parent.yfs_u1f',
				mktmcb: 'parent.yfs_mktmcb',
				gencallback: 'parent.yfs_gencb',
				mu: 1,
				lang: 'en-US',
				region: 'US',
				localize: 0,
			},
			responseType: 'stream',
		}).then<any>(function({ data }) {
			return new Promise<string>(function(resolve, reject) {
				stream = data as Nhttp.IncomingMessage

				stream.on('data', function(chunk) {
					try {
						let input = chunk.toString()
						let jsons = [] as Array<{ [key: string]: any }>

						let split = input.split('yfs_u1f(')
						split.shift()
						total = total + split.length
						split.forEach(function(v) {
							let i = v.indexOf(');}')
							let parsed = v.substring(0, i).replace(/(['"])?([a-zA-Z0-9_]+)(['"])?:/g, '"$2": ')
							if (!parsed) return;
							jsons.push(JSON.parse(parsed))
						})

						let sendi = {}
						jsons.forEach(function(s) {
							let symbol = Object.keys(s)[0]
							if (!sendi[symbol]) sendi[symbol] = {};
							let v = s[symbol]
							Object.keys(v).forEach(function(k) {
								let n = parseFloat(v[k])
								if (n == 0) return;
								let key = YH_IDK_KEY[k] as string
								sendi[symbol][key] = n
							})
						})
						console.log('sendi >', JSON.stringify(sendi, null, 4))

					} catch (error) {
						console.error('quotes.startYahooStream > catch > error', error)
					}
				})

				stream.on('end', resolve)
				stream.on('error', reject)
			})
		})

	}).catch(function(error) {
		console.error('quotes.startYahooStream > error', error)
		return Promise.resolve()

	}).finally(function() {
		console.warn('quotes.stream > FINALLY')
		if (stream) {
			stream.removeAllListeners()
			stream.destroy()
			stream = null
		}
		return startYahooStream()
	})
}
startYahooStream()

console.log('sendi >', JSON.stringify(sendi, null, 4))

sendi >
{
    NVDA: {
        LastSalePrice: 149.6,
        Volume2: 92,
        Ask: 149.5,
        AskSize: 1800,
        Bid: 148.2,
        BidSize: 100,
        PriceRealtime: 149.6
    }
}

Hope you find this helpful!

[gadicc]

Added quote module in https://github.com/gadicc/node-yahoo-finance2/compare/e0f8d35f857826d394c6e3e811fc054e53540533...1cbc17a1741392cd29e82f2a9763bbf28007840c, released in v1.6.0.

[pudgereyem]

Created a PR that adds a new module for recommendationsBySymbol here; https://github.com/gadicc/node-yahoo-finance2/pull/28

[pudgereyem]

New module recommendationsBySymbol was added in with https://github.com/gadicc/node-yahoo-finance2/pull/28.

@gadicc, would you mind updating the Issue to reflect this since I can't do it myself? 🙏

[gadicc]

Thanks, @pudgereyem! Have updated the checklist. Thanks again for all your awesome work on this.

[advaiyalad]

@gadicc Found another endpoint: https://query1.finance.yahoo.com/v7/finance/options/[symbol]. Not sure if this was found already.

Query Params

Param Name	Param Type	Required
formatted	Boolean	false
crumb	String	false
lang	String	false
region	String	false

corsDomain omitted as it does nothing.

Another finding: query1, query2, and iquery are the same. We can fetch from query1.finance.yahoo.com OR query2.finance.yahoo.com OR iquery.finance.yahoo.com for the same data. Probably want to unify what we use in our code, or allow users to choose from each domain.

Hope this helps!

[gadicc]

Hey, that's awesome, thanks @PythonCreator27! Have added it to the list in the first post.

And for noticing about the alternative host names, we should indeed accommodate that.

[roblav96]

@gadicc Appreciate the creds friend =]

Take a peek at https://github.com/roblav96/robinhood.tools it has some additional sources of financial data n more.

[roblav96]

More specifically:

• https://github.com/roblav96/robinhood.tools/blob/master/src/common/webull.ts
• https://github.com/roblav96/robinhood.tools/blob/master/src/server/adapters/webull.ts

For example:

curl 'https://quoteapi.webull.com/api/quote/tickerRealTimes/full?tickerIds=925334567,925353501,913255891'

[gadicc]

Hey @roblav96, great to see you're still involved with all this stuff, and to see you on this new repo!

Thanks for sharing the above. I recall on the original repo too people were looking for additional sources, and I guess we should add this to the README or an FAQ or something, so thank you!

What is your rough breakdown of consumption from Yahoo vs Webull now? Just out of curiosity.

[roblav96]

@gadicc Webull provides the best scrapable market data, but they make you work for it. lol Start by exploring the network traffic of https://app.webull.com/ using your web browser's network debug tools.

[advaiyalad]

Another note: recommendationsBySymbol is using an older version of the endpoint. The newer version is: https://query1.finance.yahoo.com/v7/finance/recommendationsbysymbol/PLUG. Yahoo's website does seem to use the older version, though. Changing the endpoint, though, would be a definite breaking change. @gadicc Do you think that it is worth it, or should we just create a brand new module for the newer version of the endpoint? The newer endpoint seems to just return a Quote[] instead of its own response format.

[gadicc]

Well found, @PythonCreator27. I would just leave it for now, from what you've said (and thanks for checking this all out!), there doesn't seem to be a compelling reason to upgrade. If something happens to the v6 one, maybe we could switch to v7 but return the results in the same format if that's the only difference (and our validation checks will let us know if we missed anything).

It is something to think about though for all modules. Maybe in future we should name the modules with Yahoo's API version, e.g. recommendationsBySymbolV7(). That would make it easier to cover these kinds of cases without needing to publish a new major version of the library just for one end point. Is a bit verbose though.

In short, I suggest to leave for now and see what other cases come up in the future and then make a call. Thanks for bringing attention to this.

[gadicc]

@gadicc Webull provides the best scrapable market data, but they make you work for it. lol Start by exploring the network traffic of https://app.webull.com/ using your web browser's network debug tools.

Thanks, @roblav96! I've made a start of referencing this (a long with some of the older suggestsions from node-yahoo-finance (v1) issues) on the Wiki: https://github.com/gadicc/node-yahoo-finance2/wiki/Alternative-Sources

I invite anyone to help improve this page with anything useful.

[advaiyalad]

Another finding (endpoint): https://query2.finance.yahoo.com/ws/insights/v2/finance/insights?lang=en-US&region=US&symbol=AMD&getAllResearchReports=true&reportsCount=2

Insights - seems interesting. I was looking for answers to #138, and found this along the way... Will send a PR soon to add this.