yfinance 0.1.*: Unable to load ticker information

[untucked]

• Info about your system: yfinance version: '0.1.90'
  • operating system: -windows
• Simple code that reproduces your problem

ticker = 'AAPL'
print('ticker: %s'%(ticker))
yf_info = yf.Ticker(ticker)
bs = yf_info.balance_sheet
print(bs)

• The error message Returns empty array: when I print bs Empty DataFrame Columns: [Open, High, Low, Close, Adj Close, Volume] Index: []

[BrsrkrBrown]

Having similar issue, when trying to print regularMarketPrice and volume. printing yf.Ticker(ticker).info returns: {'regularMarketPrice': None, 'preMarketPrice': None, 'logo_url': '', trailingPegRatio':None} and print(yf.Ticker('AAPL').info['volume']) returns KeyError for 'volume'

[Travis-Barton]

Same problem for me on all stocks it seems. I get {'regularMarketPrice': None, 'preMarketPrice': None, 'logo_url': ''} for any info

[mrtomcollins]

Same problem for me

[valankar]

Same here. Problem seems to have started today, perhaps some issue with Yahoo or API.

[galashour]

same here

[ValueRaider]

Looks like Yahoo has encoded the data stores embedded in HTML that yfinance relied upon. E.g.

Before (via my session cache):

root.App || (root.App = {});
root.App.now = 1671025861087;
root.App.main = {"context":{"dispatcher":{"stores":{"PageStore":{"currentPageName":"quote", ...

After

> root.App || (root.App = {});
> root.App.now = 1671275144853;
> root.App.main = {"context":{"dispatcher":{"stores":"U2FsdGVkX18NwmzwZgyLozYcI3uwl2EXN\u002FWBnsSt9vGtBRZGAIiUIqV9mS1LKBRe42L1I1VjxC1qsKQ9g8KP4qeGBkrL28E0AG14...

Anyone able to decode this?

[Rogach]

They are encrypting the store data (it seems with AES) using crypto-js library.

Deobfuscated code is doing something like this:

var key = PBKDF2.create({ keySize: 8 }).compute(t._cs, JSON.parse(t._cr))
var plaintext = AES.decrypt("U2FsdGVk...", key)
var decoded = JSON.parse(decodeURIComponent(plaintext)

I'll continue deobfuscation, it seems I will be able to provide some working code sample soon.

[ValueRaider]

Thanks @Rogach.

Any thoughts why they would do this? HTTPS already provides secure transport. If they wanted to stop scraping then why not just remove that data instead of encrypting?

[Rogach]

Who knows. Maybe it's an attempt to make scraping a bit harder. Maybe it's just a configuration change.

If they wanted to stop scraping then why not just remove that data instead of encrypting?

They can't, because they still want to show the data to the site visitors.

[Rogach]

Here's the fully working example. Runs in NodeJS, has a dependency on crypto-js library.

var fs = require("fs");
var CryptoJS = require("crypto-js");

// replace the values with fresh values from root.App.main
var _cs = '7758ce68c359';
var _cr = '{"words":[1750977242,1882992877,448740864,329742483],"sigBytes":16}';
// place the root.App.main.context.dispatcher.stores into encrypted-stores.txt file
var stores = fs.readFileSync("encrypted-stores.txt", "utf8");

var key = CryptoJS.algo.PBKDF2.create({ keySize: 8 }).compute(_cs, JSON.parse(_cr)).toString();
var plaintext = CryptoJS.AES.decrypt(stores, key);
var decoded = JSON.parse(decodeURIComponent(escape(CryptoJS.enc.Latin1.stringify(plaintext))));

console.log(decoded);

I have to run now, don't have the time to convert this to python, sorry.

[ValueRaider]

I tried to port but give up. None of the examples I found use a "words" array as the salt, most assume the salt is embedded in the encrypted text. Example

[onesoned]

https://github.com/ranaroussi/yfinance/issues/1251#issue-1501456520

For some tickers (like SRAD or HIVE) the info-scraping is working...

[ValueRaider]

For any crypto experts that are unsure where to put decryption code, see data.py get_json_data_stores(), you can see where root.App.main element is accessed.

Edit: Please keep comments related to implementing a fix, we know the issue exists.

[paulmcq]

Since browser and curl work, is it maybe wanting a different UserAgent???

On Sat, Dec 17, 2022 at 6:24 PM Jeremy @.***> wrote:

I tried using requests to see what was returned with the scraping url, eg. requests.get('https://finance.yahoo.com/quote/SHEL.L'), but am getting a 404 when I run that. Accessing that through browser and curl works just fine though.

— Reply to this email directly, view it on GitHub https://github.com/ranaroussi/yfinance/issues/1246#issuecomment-1356539780, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD3AQGLBCCSE3RMWU2K4PJLWNZKTXANCNFSM6AAAAAATBUYB34 . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[Rogach]

Here's the sample decoder ported to Python:

from pathlib import Path
import hashlib
import json
from base64 import b64decode
from Crypto.Cipher import AES
from Crypto.Util.Padding import unpad

stores = Path("encrypted-stores.txt").read_text()
_cs = '7758ce68c359'
_cr = '{"words":[1750977242,1882992877,448740864,329742483],"sigBytes":16}'

_cr = b"".join(int.to_bytes(i, length=4, byteorder="big", signed=True) for i in json.loads(_cr)["words"])
password = hashlib.pbkdf2_hmac("sha1", _cs.encode("utf8"), _cr, 1, dklen=32).hex()

stores = b64decode(stores)
assert stores[0:8] == b"Salted__"
salt = stores[8:16]
stores = stores[16:]

def EVPKDF(
    password,
    salt,
    keySize=32,
    ivSize=16,
    iterations=1,
    hashAlgorithm="md5",
) -> tuple:
    """OpenSSL EVP Key Derivation Function
    Args:
        password (Union[str, bytes, bytearray]): Password to generate key from.
        salt (Union[bytes, bytearray]): Salt to use.
        keySize (int, optional): Output key length in bytes. Defaults to 32.
        ivSize (int, optional): Output Initialization Vector (IV) length in bytes. Defaults to 16.
        iterations (int, optional): Number of iterations to perform. Defaults to 1.
        hashAlgorithm (str, optional): Hash algorithm to use for the KDF. Defaults to 'md5'.
    Returns:
        key, iv: Derived key and Initialization Vector (IV) bytes.

    Taken from: https://gist.github.com/rafiibrahim8/0cd0f8c46896cafef6486cb1a50a16d3
    OpenSSL original code: https://github.com/openssl/openssl/blob/master/crypto/evp/evp_key.c#L78
    """

    assert iterations > 0, "Iterations can not be less than 1."

    if isinstance(password, str):
        password = password.encode("utf-8")

    final_length = keySize + ivSize
    key_iv = b""
    block = None

    while len(key_iv) < final_length:
        hasher = hashlib.new(hashAlgorithm)
        if block:
            hasher.update(block)
        hasher.update(password)
        hasher.update(salt)
        block = hasher.digest()
        for _ in range(1, iterations):
            block = hashlib.new(hashAlgorithm, block).digest()
        key_iv += block

    key, iv = key_iv[:keySize], key_iv[keySize:final_length]
    return key, iv

key, iv = EVPKDF(password, salt, keySize=32, ivSize=16, iterations=1, hashAlgorithm="md5")

cipher = AES.new(key, AES.MODE_CBC, iv=iv)
plaintext = cipher.decrypt(stores)
plaintext = unpad(plaintext, 16, style="pkcs7")
decoded = json.loads(plaintext)

print(decoded.keys())

[Korach250]

Thanks. Continuing #1248 here. 'floatShares' seems to be the issue.

1. It is curious that SOME data still gets through. With these three tickers, TISI (Float 27,595,853), PASG (Float 27,502,482), SQL (Float 4,578,990) my program runs perfectly (as it has run - systematically - for the last year).

2. However, with any other tickers' data, while it starts fine: Data Downloaded

My program follows: flt_shares = {} flt_shares[f'{symbol}'] = ticker.info.get('floatShares') df['Float'] = (flt_shares[f'{symbol}'])

Then when I try to put it in a dataframe with some formatting, it highlights an issue with the Float data: df.loc[:, 'Float_'] = df['Float'].map('{:,.0f}'.format) TypeError: unsupported format string passed to NoneType.format

[Rogach]

@ValueRaider Seems I have a solution in PR #1253

[Korach250]

Big thank you. fredrik-corneliusson Super newbie. I am doing it wrong because I go this.

ERROR: Could not find a version that satisfies the requirement yfinance-tz-cache (from versions: none) ERROR: No matching distribution found for yfinance-tz-cache

%pip install yfinance-tz-cache import yfinance as yf

Plus adding _ instead of - for import yfinance-tz-cache as yf

Also should add it is ONLY the shareFloat data I can't access. Open/Low/High/Close & Volume work perfectly using the original yfinance. As does TISI (Float 27,595,853), PASG (Float 27,502,482), SQL (Float 4,578,990). Why do these three tickers function perfectly and not the others?

[ValueRaider]

@Korach250 Not everything on GitHub is also available on pypi.org

[fredrik-corneliusson]

Big thank you. fredrik-corneliusson Super newbie. I am doing it wrong because I go this.

Thanks, but you should really thank @Rogach, who did the investigation and coming up with a working Python solution. My branch was referred by mistake, you should probably use the branch from @Rogach PR https://github.com/ranaroussi/yfinance/pull/1253 Or wait for it to be merged into yfinance.

[ValueRaider]

New release out that fixes this - 0.1.92

[WimmerErich]

Everything seems to work fine with 0.1.92.

Only thing which i recognized which was not working for me, was that i was not able to get the number of shares.

e.g.

msft.shares

[Korach250]

First time I have ever seen the open source collaboration process in action. Powerful. Thanks all. Especially @Rogach

[ValueRaider]

Only thing which i recognized which was not working for me, was that i was not able to get the number of shares.

Good spot. I can fix this.

[Korach250]

Still stuck. Same 'floatShares'' issue. Even though: Successfully installed cryptography-38.0.4 yfinance-0.1.92 Successfully installed yfinance-0.1.93 ValueRaider what do you think?

[ValueRaider]

New release 0.1.93, fixes Ticker.shares

[saviornt]

Just installed 0.1.93 and it looks like I'm having the same issue as 0.1.90 with Ticker.info

[ValueRaider]

@saviornt Can you double-check, then post sample code? Is working for me.

[saviornt]

@ValueRaider Sure! Here is the terminal output from uninstalling 0.1.90 and reinstalling via pip install yfinance: Successfully installed cffi-1.15.1 cryptography-38.0.4 pycparser-2.21 yfinance-0.1.93

my code:

ticker = "AAPL"
yf_ticker = yf.Ticker(ticker)
info = yf_ticker.info  # Returns a dictionary of values for the security
df = pd.DataFrame.from_dict(info, orient="index")
df = df.rename(columns={0: ticker})

the info dictionary returns:

{'regularMarketPrice': None, 'preMarketPrice': None, 'logo_url': '', 'trailingPegRatio': 2.6609}

and the dataframe returns:

column = 'AAPL'
index = 'regularMarketPrice', 'preMarketPrice', 'logo_url', 'trailingPegRatio'

the DF values are of course the dictionary values.

[ValueRaider]

What does this return? yf.__version__

If that returns 0.1.93, then confirm that PIP installed correct files - is "cryptography" in site-packages/yfinance/utils.py? Use print(yf) to show where PIP installed code.