jugaad-py
Historical, Live Data related API's not working
Issue description
The Historical, Live Stock and F&O Api's not working post 11 AM Yesterday. Response is coming empty.
Stock Quote
# Download stock data to pandas dataframe
from jugaad_data.nse import stock_df
df = stock_df(symbol="SBIN", from_date=date(2020,1,1),
to_date=date(2020,1,30), series="EQ")
df
Error
---------------------------------------------------------------------------
JSONDecodeError Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/requests/models.py in json(self, **kwargs)
973 try:
--> 974 return complexjson.loads(self.text, **kwargs)
975 except JSONDecodeError as e:
13 frames
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
JSONDecodeError Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/requests/models.py in json(self, **kwargs)
976 # Catch JSON-related errors and raise as requests.JSONDecodeError
977 # This aliases json.JSONDecodeError and simplejson.JSONDecodeError
--> 978 raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
979
980 @property
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Live Data, market Quote
from jugaad_data.nse import NSELive
n = NSELive()
status = n.market_status()
# Print the raw response text
print("Raw response:", status.text) # Check if status has a .text attribute
Error:
--------------------------------------------------------------------------
JSONDecodeError Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/requests/models.py in json(self, **kwargs)
973 try:
--> 974 return complexjson.loads(self.text, **kwargs)
975 except JSONDecodeError as e:
7 frames
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
JSONDecodeError Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/requests/models.py in json(self, **kwargs)
976 # Catch JSON-related errors and raise as requests.JSONDecodeError
977 # This aliases json.JSONDecodeError and simplejson.JSONDecodeError
--> 978 raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
979
980 @property
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Code
from jugaad_data.nse import NSELive
from pandas import json_normalize
n = NSELive()
quotes = n.stock_quote_fno("TCS")
quotes
ERROR:
---------------------------------------------------------------------------
JSONDecodeError Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/requests/models.py](https://localhost:8080/#) in json(self, **kwargs)
973 try:
--> 974 return complexjson.loads(self.text, **kwargs)
975 except JSONDecodeError as e:
7 frames
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
JSONDecodeError Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/requests/models.py](https://localhost:8080/#) in json(self, **kwargs)
976 # Catch JSON-related errors and raise as requests.JSONDecodeError
977 # This aliases json.JSONDecodeError and simplejson.JSONDecodeError
--> 978 raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
979
980 @property
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
I am also facing the same issue. Request for guidance from experienced persons in this
Seems like data is coming encoded format from api's from live.py resulting in error.
Encoded Api response:
Code
from datetime import datetime
from requests import Session
class NSELive:
time_out = 5
base_url = "https://www.nseindia.com/api"
page_url = "https://www.nseindia.com/get-quotes/equity?symbol=LT"
_routes = {
"market_status": "/marketStatus",
# Add other routes as necessary
}
def __init__(self):
self.s = Session()
h = {
"Host": "www.nseindia.com",
"Referer": "https://www.nseindia.com/get-quotes/equity?symbol=SBIN",
"X-Requested-With": "XMLHttpRequest",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36",
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8",
"Cache-Control": "no-cache",
"Connection": "keep-alive",
}
self.s.headers.update(h)
self.s.get(self.page_url) # Initial request to set cookies
def get(self, route, payload={}):
url = self.base_url + self._routes[route]
r = self.s.get(url, params=payload)
print("URL:", url)
print("Response Status Code:", r.status_code)
print("Raw Response:", r.text)
try:
return r.json()
except Exception as e:
print("Failed to decode JSON:", e)
return None
def market_status(self):
return self.get("market_status", {})
# Test the NSELive class
if __name__ == "__main__":
n = NSELive()
market_status = n.market_status()
print("Market Status:", market_status)
Output:
URL: https://www.nseindia.com/api/marketStatus
Response Status Code: 200
�ikh��O-l�Q��{�ڡΧ)��R��Հ�| �ీX>��W߶}R��g���(��M�S{�I�R������ބk�@�?h�e�C������1�7]��ѷ��Z��QZ��0��V_�&�2A`.�����r��OZ��,(�t��x�7��f�u��/_=>·4�A���B8�p!<���'�i`I���zAJ�����$E��i$����X�r�ve������9һTO:y� N�d@��V�b�z��˺��)O��4����e�<K�۳�
� +;�����t�iH��h���hƑ��뻧��
Failed to decode JSON: Expecting value: line 1 column 2 (char 1)
Market Status: None
Normal Request using requests showing output:
Code
import requests
headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:80.0) Gecko/20100101 Firefox/80.0'}
# url = 'https://www.nseindia.com/api/option-chain-indices?symbol=NIFTY'
url = 'https://www.nseindia.com/api/marketStatus'
with requests.session() as s:
data = s.get(url, headers=headers).json()
print(type(data))
print(data)
Output:
<class 'dict'>
{'marketState': [{'market': 'Capital Market', 'marketStatus': 'Open', 'tradeDate': '15-Oct-2024 11:04', 'index': 'NIFTY 50', 'last': 25068.55, 'variation': -59.400000000001455, 'percentChange': -0.24, 'marketStatusMessage': 'Normal Market is Open'}, {'market': 'Currency', 'marketStatus': 'Open', 'tradeDate': '15-Oct-2024', 'index': '', 'last': '', 'variation': '', 'percentChange': '', 'marketStatusMessage': 'Market is Open'}, {'market': 'Commodity', 'marketStatus': 'Open', 'tradeDate': '15-Oct-2024', 'index': '', 'last': '', 'variation': '', 'percentChange': '', 'marketStatusMessage': 'Market is Open'}, {'market': 'Debt', 'marketStatus': 'Open', 'tradeDate': '15-Oct-2024', 'index': '', 'last': '', 'variation': '', 'percentChange': '', 'marketStatusMessage': 'Market is Open'}, {'market': 'currencyfuture', 'marketStatus': 'Open', 'tradeDate': '15-Oct-2024', 'index': '', 'last': '84.1100', 'variation': '', 'percentChange': '', 'marketStatusMessage': 'Market is Open', 'expiryDate': '29-Oct-2024', 'underlying': 'USDINR', 'updated_time': '15-Oct-2024 11:02', 'tradeDateFormatted': '15-Oct-2024 11:02', 'slickclass': 'slick-item'}], 'marketcap': {'timeStamp': '14-Oct-2024', 'marketCapinTRDollars': 5.47, 'marketCapinLACCRRupees': 459.9931470995962, 'marketCapinCRRupees': 45999314.71, 'marketCapinCRRupeesFormatted': '45,999,314.71', 'marketCapinLACCRRupeesFormatted': '459.99', 'underlying': 'Market Cap'}, 'indicativenifty50': {'dateTime': '14-Oct-2024 15:30', 'indicativeTime': None, 'indexName': 'NIFTY 50', 'indexLast': None, 'indexPercChange': None, 'indexTimeVal': None, 'closingValue': 25127.95, 'finalClosingValue': 25127.95, 'change': 163.71, 'perChange': 0.66, 'status': 'CLOSE'}}
Just try replacing the header dictionary in jugaad_data/nse/history.py with the header you have mentioned. It works for me.
Earlier till Yesterday, with existing headers, data was coming in original format for all cloud providers wherever the api is running. Now data is coming in encoded format for original header , due to which api's are failing.
Existing Header:
h = {
"Host": "www.nseindia.com",
"Referer": "https://www.nseindia.com/get-quotes/equity?symbol=SBIN",
"X-Requested-With": "XMLHttpRequest",
"pragma": "no-cache",
"sec-fetch-dest": "empty",
"sec-fetch-mode": "cors",
"sec-fetch-site": "same-origin",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36",
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8",
"Cache-Control": "no-cache",
"Connection": "keep-alive",
}
Below Header: Only working in Local Machine:
The issue with this header: headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:80.0) Gecko/20100101 Firefox/80.0'} is API request are getting blocked by NSE as its giving IP of Cloud Providers, NSE Blocks all IP's of Cloud Providers (AWS, GCP, Azure).
Hi All, I resolved this by navigating to live.py file in installation directory and locating live.py file.(use pip show jugaad-data) In that file locate header dictoniary by 'h' name. and replace "Accept-Encoding": "gzip, deflate, br" with this one "Accept-Encoding": "gzip, deflate", It starts running for me.
I hope this helps temporarily until fix is released.
Hi amritjain123, Great!!! Thanks for temp solution. Could you please elaborate how removing "br" for Accept-Encoding solve this issue? I'm still in learning phase, just curious.
To all, Do you think this api is loophole for NSE, will the NSE api will get remove sooner or later? as we know the api is not free by NSE, also NSE endpoint contain api in it, making me doubt.
Do we need to limit API access per second to avoid getting blocked?
The fix doesn't work well with batch commands. JSON Decode Error popping up if trying to iterate historical data retrieval for over 15 tickers.
Seems like this repository is not being actively managed anymore. Any contributors subscribed to this thread? There are several open issues on this repository.
What does it take for someone to push this change and to get it approved? I am not sure how pushing this workaround to production would cause issues, but something simple as github.com/jugaad-py/jugaad-data/issues/70 should be doable by anyone. It's just updating text in a file.
PR#88 should help. brotli python module is required to handle encoded responses (check requests to also handle it)
Hi All, I resolved this by navigating to live.py file in installation directory and locating live.py file.(use pip show jugaad-data) In that file locate header dictoniary by 'h' name. and replace "Accept-Encoding": "gzip, deflate, br" with this one "Accept-Encoding": "gzip, deflate", It starts running for me.
I hope this helps temporarily until fix is released.
if this does not work, then move to use nsepython
