pandas-datareader
pandas-datareader copied to clipboard
pydata
Response format from Yahoo seems to have changed I keep getting this error.
File "/Users/yeison/miniforge3/envs/tf-metal-0.6.0/lib/python3.10/site-packages/pandas_datareader/yahoo/daily.py", line 153, in _read_one_data data = j["context"]["dispatcher"]["stores"]["HistoricalPriceStore"] TypeError: string indices must be integers
As work around you may try to use solution from https://pypi.org/project/yfinance/ title pandas_datareader override
They started to encrypt the stores, looks like they encrypted with AES or something like that and now stores values comes as a string, not sure how to solve that
Is there some way that I can implement a temporary solution using yfinance as an override and pull request it ?
Is there some way that I can implement a temporary solution using yfinance as an override and pull request it ?
>>> from pandas_datareader import data as pdr
>>> import yfinance as yf
>>> yf.pdr_override()
>>> y_symbols = ['SCHAND.NS', 'TATAPOWER.NS', 'ITC.NS']
>>> from datetime import datetime
>>> startdate = datetime(2022,12,1)
>>> enddate = datetime(2022,12,15)
>>> data = pdr.get_data_yahoo(y_symbols, start=startdate, end=enddate)
Is there some way that I can implement a temporary solution using yfinance as an override and pull request it ?
>>> from pandas_datareader import data as pdr >>> import yfinance as yf >>> yf.pdr_override() >>> y_symbols = ['SCHAND.NS', 'TATAPOWER.NS', 'ITC.NS'] >>> from datetime import datetime >>> startdate = datetime(2022,12,1) >>> enddate = datetime(2022,12,15) >>> data = pdr.get_data_yahoo(y_symbols, start=startdate, end=enddate)
Oh yeah I have used this but I meant fixing pandas-datareader as I have already submitted the code for my dissertation so I can't change my code directly, the only way I can change my code is by changing the library hehe. sorry for confusion
They started to encrypt the stores, looks like they encrypted with AES or something like that and now stores values comes as a string, not sure how to solve that
What are the stores? I will try anything to fix this as I have just submitted my interim dissertation and without this library, my code has no data :/ due to the nature of academia I dont think asking if the marker can override with yfinance will work
What are the stores? I will try anything to fix this as I have just submitted my interim dissertation and without this library, my code has no data :/ due to the nature of academia I dont think asking if the marker can override with yfinance will work
Then you are the best candidate to fix this library. j["context"]["dispatcher"]["stores"] response contains some encrypted data instead of plain object, so you may take a look at how this data is handled in yfinance, and migrate the code. Good luck.
What are the stores? I will try anything to fix this as I have just submitted my interim dissertation and without this library, my code has no data :/ due to the nature of academia I dont think asking if the marker can override with yfinance will work
Then you are the best candidate to fix this library. j["context"]["dispatcher"]["stores"] response contains some encrypted data instead of plain object, so you may take a look at how this data is handled in yfinance, and migrate the code. Good luck.
With a stroke of magical programmer luck I have managed to migrate some code from yfinance where now the data is decrypted and I have tested on a small number of stocks I am able to get their data.
I changed code in: pandas_datareader/yahoo/daily.py Just adding a decryption function and making sure data is
handled correctly like before .
It took me all day as I have never worked on such a professional code base so far. But that leads to another issue now, How do I go about merging my solution or I guess I have to create a pull request now? I am not sure how this works!
pandas-datareader was checkedout locally to my IDE and I have made all the changes there, what do you suggest me to do? I am sure there are still things needed to be added to the codebase like requirements etc? idk this is my first time.
Thanks for any help.
File "/Users/yeison/miniforge3/envs/tf-metal-0.6.0/lib/python3.10/site-packages/pandas_datareader/yahoo/daily.py", line 153, in _read_one_data data = j["context"]["dispatcher"]["stores"]["HistoricalPriceStore"] TypeError: string indices must be integers
I have found a fix, as stated in above comment. Do you know how about I will go creating a pull request? I just tried on my IDE and got permission denied? Any suggestions?
@raphi6 have you tried branching from the downloaded repository and then pushing your changes? It should say there's no upstream branch, by setting one you'll create a pull request.
Ahh yes branching, I always forget. So now I will checkout pandas-datareader, create a branch and add my changes, create a pull request. And that should work?
Yes I think so, it may not be the best way of doing it but I do not currently know of a better one. By setting an upstream branch and pushing your branch to it, it should create the PR.
Yes I think so, it may not be the best way of doing it but I do not currently know of a better one. By setting an upstream branch and pushing your branch to it, it should create the PR.
I just tried creating a new branch and commiting, pushing, then creating a PR and get the above, What did you mean by setting an upstream branch?
You may be trying to push directly to master, which only the repo owners will have permission for (although it is advised never to push directly to master in any case). To not take up replies in the thread, I would recommend looking into the process of opening a PR on GitHub for another repo using online resources:
Apologies I can't be any more help, quite new to Git in terms of contributing to 3rd party repos.
I've opened a pull request with working code, be sure to check it out if it works for you guys.
Hi, @raphi6 , I can confirm it works perfectly fine. Thanks a lot for the quick fix! Let's see if they approve the pull request.
For reference, I cloned your pull request and installed it using the following sequence of commands (in Ubuntu 20.04):
git clone https://github.com/raphi6/pandas-datareader.git
git checkout 'Yahoo!_Issue#952'
conda uninstall pandas-datareader
conda install pycryptodome pycryptodomex
python setup.py install --record installed_files.txt
The --record argument in the install command is to get a list of installed files, so that it is easy to uninstall in the future (following this SO thread). The pycrypto* files are dependencies I has to install to make it work.
Hi, @raphi6 , I can confirm it works perfectly fine. Thanks a lot for the quick fix! Let's see if they approve the pull request.
For reference, I cloned your pull request and installed it using the following sequence of commands (in Ubuntu 20.04):
git clone https://github.com/raphi6/pandas-datareader.git git checkout 'Yahoo!_Issue#952' conda uninstall pandas-datareader conda install pycryptodome pycryptodomex python setup.py install --record installed_files.txtThe
--recordargument in the install command is to get a list of installed files, so that it is easy to uninstall in the future (following this SO thread). Thepycrypto*files are dependencies I has to install to make it work.
Can this be installed or implemented on Google Colabs?
The following seems to work well on Google Colab.
%%shell
git clone https://github.com/raphi6/pandas-datareader.git
cd pandas-datareader
git checkout 'Yahoo!_Issue#952'
pip install pycryptodome pycryptodomex
python setup.py install --record installed_files.txt
The following seems to work well on Google Colab.
%%shell git clone https://github.com/raphi6/pandas-datareader.git cd pandas-datareader git checkout 'Yahoo!_Issue#952' pip install pycryptodome pycryptodomex python setup.py install --record installed_files.txt
/usr/local/lib/python3.8/dist-packages/pandas_datareader/base.py:272: SymbolWarning: Failed to read symbol:
RemoteDataError: No data fetched using 'YahooDailyReader'
I get this which started to be an issue a year or so ago and have been using this since then pip install git+https://github.com/pydata/pandas-datareader.git
Is it possible to put them together?
Hello - I've got a few complex scripts running in jupiter notebook pulling data from yahoo finance - what are the exact commands which need to be entered in order to regain access so that I dont get the "string indices must be integers" error ? much appreciated, thx for the help
The following seems to work well on Google Colab.
%%shell git clone https://github.com/raphi6/pandas-datareader.git cd pandas-datareader git checkout 'Yahoo!_Issue#952' pip install pycryptodome pycryptodomex python setup.py install --record installed_files.txt
/usr/local/lib/python3.8/dist-packages/pandas_datareader/base.py:272: SymbolWarning: Failed to read symbol:RemoteDataError: No data fetched using 'YahooDailyReader'I get this which started to be an issue a year or so ago and have been using this since thenpip install git+https://github.com/pydata/pandas-datareader.gitIs it possible to put them together?
@pyproper Yes, it is. You can use this:
pip install git+https://github.com/raphi6/pandas-datareader.git@ea66d6b981554f9d0262038aef2106dda7138316
Notice I am using the commit hash here instead of a branch name, because it is Yahoo!_Issue#952 and there is an issue with hash characters when using pip this way.
@pyproper Yes, it is. You can use this:
pip install git+https://github.com/raphi6/pandas-datareader.git@ea66d6b981554f9d0262038aef2106dda7138316Notice I am using the commit hash here instead of a branch name, because it is
Yahoo!_Issue#952and there is an issue with hash characters when using pip this way.
Perfect! Thank You
@joanlofe
@pyproper Yes, it is. You can use this:
pip install git+https://github.com/raphi6/pandas-datareader.git@ea66d6b981554f9d0262038aef2106dda7138316Notice I am using the commit hash here instead of a branch name, because it is
Yahoo!_Issue#952and there is an issue with hash characters when using pip this way.
I had to use pip3 to install, but now have the following error:
Traceback (most recent call last):
File "USP.py", line 6, in
Thoughts?
I've opened a pull request with working code, be sure to check it out if it works for you guys.
This fix by @raphi6 needs to be made a priority. Yahoo API is bricked through PDR.
I've opened a pull request with working code, be sure to check it out if it works for you guys.
This fix by @raphi6 needs to be made a priority. Yahoo API is bricked through PDR.
Thank you! if anyone could get into contact with someone that can accept the PR, that would be great!
@KryptoEmman Have you tried
pip install packagingand then repeat again?
@joanlofe I ran that command with pip3 not pip - running on MacOS - which ran successfully, but now getting the following different error:
Traceback (most recent call last):
File "USP.py", line 6, in
Clearly missing the required Crypto module using "pip3 install Crypto", but still getting the same error, so I'm guessing this must be from another package. Tried "pip3 install hashlib" which errors out with the exit code 1 and a message stating "unknown OS, please update setup.py" which leads me to think that some of these modules are only available for MacOS.
pip3 list outputs the following:
aes 1.2.0 certifi 2022.12.7 charset-normalizer 2.1.1 crypto 1.4.1 et-xmlfile 1.1.0 forex-python 1.8 idna 3.4 lxml 4.9.1 Naked 0.1.32 numpy 1.23.4 openpyxl 3.0.10 packaging 22.0 pandas 1.5.1 pandas-datareader 0+untagged.811.gea66d6b pip 22.3.1 python-dateutil 2.8.2 pytz 2022.6 PyYAML 6.0 requests 2.28.1 setuptools 65.5.0 shellescape 3.8.1 simplejson 3.17.6 six 1.16.0 urllib3 1.26.12
Thoughts? Thanks.
@joanlofe
pip3 install pycryptodome pycryptodomex
Tried the command above but get the same result when I try to use pandas-reader with Yahoo: "ModuleNotFoundError: No module named 'Crypto'"