node-bovespa
node-bovespa copied to clipboard
nihey
Original Bovespa's API is dead
~~As of now, there is no open current way to scrape it that I know of.~~
~~It seems there are however some paid alternatives, like: Bolsa Financeira~~
Upon doing some search, there seems to be way, this page gets it using a WebSocket protocol, the data seems to appear like this:
There is also this API, it seems to return just stock codes, but I do not know the full extent of it:
$ curl "https://data-bovespa.tradingview.com/search/?text=ABEV3&exchange=BMFBOVESPA&type=&domain=bovespa" -H "Origin: https://s.tradingview.com"
Whenever I get some free time, I'll try to migrate node-bovespa to use this (it won't be as smooth wrapping a REST API, but it is doable).
This is weird. TradingView uses data from ICE data services. My main concern is due to the reliability of data, so I wanted to have data as close as possible from bovespa's services.
Do you guys got any progress on finding another source for the data?
Alpha Vantage seems to be a great alternative. It is free (although it needs registration to obtain a API Key), and have quotes from B3 (with some delay, of course). It is also worth noting that some of the assets listed on B3, like subscription rights, are not available in Alpha Vantage.
And of course, depending on your goal, it might not be a good idea to rely on a third party for stock quotes.
@lenilsonjr Thanks for the information, Alpha Vantage might be the easiest way to got from now. I'll work on it as soon as possible.
@lenilsonjr @nihey I could not find B3 quotes in Alpha. How you should pass in the URL? CSNA:B3? This is not working for me
@lenilsonjr @nihey I could not find B3 quotes in Alpha. How you should pass in the URL? CSNA:B3? This is not working for me
Don't know if it's still relevant to you, but I could get appending .SA to the stock symbol, similar how Yahoo Finance does.
Example:
Hey everyone, I'm planning to make this project alive once more.
I've been thinking about the plans on how to do it and found out that we can download historical series on these files.
My plan to is to make an API out of it for historical data querying. After doing it, the plan is to offer it publicly (or self hosted via the package).
Once the historical data API is done, I can add a realtime API via webscraping from various sources. It may be hard to maintain it, but we'll see the best way to handle it once we reach this point.
I've managed to setup a new API:
https://bovespa.nihey.org/api/quote/ABEV3/2018-02-14
I'll rewrite the code for node-bovespa so that it uses this brand new API, until then, I recommend using this API directly for whoever needs consuming this data.
Note: it currently do not have realtime data, just historic ones.
Hey, this is awesome. I just read your email and I am all up to make this project be big. How can I help? Do we have a roadmap?
@djalmaaraujo Great to see you're interested :smile: !
I'll get everything organized this week and possibly create issues, but as a brief explanation on everything I wanted for this project, we should add:
-
Extract realtime data via webscraping from some of these platforms
- https://finance.yahoo.com/quote/abev3.sa?ltr=1
- https://www.tradingview.com/symbols/BMFBOVESPA-ABEV3/
- https://www.infomoney.com.br/ambevsa-abev3
- https://economia.uol.com.br/cotacoes/bolsas/acoes/bvsp-bovespa/abev3-sa/
-
Unit testing to receive notifications when the realtime API stop working because one of those sites changed.
-
Include the realtime data into the API
-
regularly update the historic series with official data from bovespa
-
a CLI utility to extract data from the API
# Something like:
$ bovespa ABEV3 -d 2018-01-08
<outputs the data>
- an easy way to access the data
// Something like:
const bovespa = require('bovespa');
bovespa.get("ABEV3", "2018-01-08").then(...)
:arrow_up: That would be it for now, but I'm open for suggestions too.
@nihey A few days ago I received an email for a legal department of XP and a while ago also from bank safra, because I had open source projects with their name or part of it. I am pretty sure this will be a near future for this project as-well, which sucks.
Also, I really love the idea but while I am very interested I am on zero time now, so I am sorry if I passed the wrong impression in my last comment.
I think the roadmap is great but I would ignore the CLI for now since this is not very used compared to an API. That's my opinion.
I will try to find time for this anytime soon. tks
@djalmaaraujo
A few days ago I received an email for a legal department of XP and a while ago also from bank safra, because I had open source projects with their name or part of it. I am pretty sure this will be a near future for this project as-well, which sucks.
That sucks a lot, I really hope that if this happens here, it takes a long time to happen.
Also, I really love the idea but while I am very interested I am on zero time now, so I am sorry if I passed the wrong impression in my last comment.
No problems!
I think the roadmap is great but I would ignore the CLI for now since this is not very used compared to an API. That's my opinion.
I think you're right, although the CLI can be quite handy sometimes, it is not nearly as useful as the API
I should definitely add some documentation not only for the node API, but for the REST API too, and this should be the main focus for the project (ideally aiming into the real time API).
I will try to find time for this anytime soon. tks
Alright, thanks!
I was searching for some solution to get real time and endup here...
Any thoughts in get real-time data?
I'm doing one dashboard for myself my current solution is using google sheet, is not the best one as it does not reflect the current stock price (they say it can be like 20min delay), but at least I get something close to the current stock price like using the formula =GOOGLEFINANCE("ITUB4"; "price")
Maybe for historical data, you can use the same? I think it's going to be pretty easy to maintain...
@krolow but how do you get the data from google sheets? I mean, is there a way to get the sheet generated data from code running outside (not using google script)?
BTW, I was playing with Alpha Vantage (using Node), but they now have some limits ("our standard API call frequency is 5 calls per minute and 500 calls per day") and, in this case, it would work just to get historical data - for example, updating daily a database with 500 assets at most.
so the way that i'm using right now I have a public sheet with all the stock codes and I put it to export as CSV and I consume that CSV url when I need to fetch it... not perfect but for my own purposes it's working well...
and what I did was to also have several columns so I can have the history of the stock in the last week, 30d ago, 60d ago etc...
@krolow This might be useful too: https://sheetsu.com/ I have not used them yet, but its looks like a good alternative to extract the data from a spreadsheet via API.
nice I did a small demo just to test the idea https://github.com/krolow/carteira/ it's basically consuming the CSV of a public sheet
Hey guys, not sure if this helps at some point, but I am building a integrated database with all the companies/papers in b3.
https://www.tradertax.com.br/api/v1/b3_companies.json?query=xp
At some point this will be private, or not. I get info from a bunch of sources. Some companies don't have the CNPJ attached, I am working on that.
@djalmaaraujo I'm not sure if node-bovespa could use it, but I can see some uses of it on some of my personal projects.
Seeing these last updates I think I've came up with a reliable solution to keep the real time data. I'll take some time to implement it, but it surely will be a major enhancement on this project.
@nihey That's what I thought. Are you using the spreadsheet solution or alphavantage?
@nihey I could not find in the GOOGLEFINANCE formula ETF's: SMLL11 for example. Can you?
@djalmaaraujo I could not find SMLL11 indeed, even though some other ETF's could be found (like XBOV11). Seems like we may have to extract these data from AlphaVantage (or another source anyway).
have u tried SMAL11? it worked for me here, also I have tried ETF IVVB11 works as well
@krolow Interesting. The hard part is to find all possibilities, because it's not the ticker name.
yeah, at least the FIIS and stocks itself all the codes that I have tried work just fine, maybe only the ETF ones that have this difference? 🤔
On Mon, Jul 22, 2019 at 11:56 AM Djalma Araújo [email protected] wrote:
@krolow https://github.com/krolow Interesting. The hard part is to find all possibilities, because it's not the ticker name.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nihey/node-bovespa/issues/5?email_source=notifications&email_token=AAALNZVVUSIRRRNNHOXE4UDQAXDCTA5CNFSM4CI2UNJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2QGFQY#issuecomment-513827523, or mute the thread https://github.com/notifications/unsubscribe-auth/AAALNZVA23ER2OSI2NSVNDTQAXDCTANCNFSM4CI2UNJQ .
--
Vinícius Krolow - Software Engineer http://krolow.com.br
So, I'm finally coming up with a solution, I've Google Finance API Proxy, consuming data from spreadsheets, storing them in the database. If any of you want to try it out, I'm keeping the API at: https://bovespa.nihey.org/api/realtime (or https://bovespa.nihey.org/api/realtime/ABEV3).
I'll also add AlphaVantage API proxying at some point, but it seems like this should be enough for the next version.
So, finishing a couple of things will mark the next version of node-bovespa:
- Add data for the current day back to the
cli - Document and add these new methods to
README.md