BETS
BETS copied to clipboard
nmecsys
Why BETS is so much slower than similar packages?
Hey @GreedBlink , @luccasr73 . I really like BETS but one big issue that I noted is that it is around 10 times slower than other packages like wilsonfreitas/rbcb , Quandl, fredr, ...
Taking this simple example for comparison, while rbcb takes 0.25 seconds, BETS is taking 30 seconds:
library(tictoc)
tic()
BETS::BETSget(24369)
toc()
tic()
rbcb::get_series(24369)
toc()
Any idea why such a difference? Thank you.
@GitHunter0 , I'm glad to hear that you use the BETS package.
If you do not know the initial idea of the package was to offer not only access to time series data, but also access to data analysis tools.
The package needs an update to its most basic structures. I am working on a new version, lighter (in terms of dependencies, features and functionalities) and faster.
Hey @GreedBlink , a new version would be awesome, I'm looking forward to that!
Hey @GreedBlink , I see here that BETS provides a lot of functionalities divided in 4 groups, which makes it comprehensive but possibly a little bit heavy.
Nonetheless, I suspect the speed issue is directly related to the database server, have you guys inspected that?
PS: Although BETS database might be something hard to maintain updated and optimized for external access, it provides useful metadata that no other package offers, which makes BETS a unique tool (also because it makes it possible to retrieve data from multiple Brazilian sources).
Thank you
@GitHunter0 , you are right! The problem is the database connection.
I will migrate the database to AWS RDS and release a new development version. Another way to improve this is create a rest api for metadata with public access. What to you think? A simple request::GET() would be necessary and not DBI and RMySQL that is, less two packages and thus decreasing the BETS namespace.
Hi @GreedBlink! If you are going to continue with the idea of creating an api rest, I am interested in collaborating with the project if there is a possibility, an api with AWS Lambda would be really cool
@GreedBlink , awesome news! I believe migrating to AWS RDS is the right call. And I think the rest api for metadata with public access is really a great idea! Thank you for the feedback
Yeah i liked too!
I will create a new repository for the api and I would like to ask for your collaboration on this project (@GitHunter0 and @luccasr73), how about it? The RDS ideia is great , but any possibility to decrease the BETS imports list would be more important.
Wait for another contact in your e-mail !
Thank you for your interest in the BETS, @GitHunter0 and @luccasr73
Hey @GitHunter0 , what your email?
I just sent you an email now. Let me know if you don't receive it
