stockdex icon indicating copy to clipboard operation
stockdex copied to clipboard

Macrotrends Frequency Parameter on statement URLs

Open wittling opened this issue 4 months ago • 10 comments

Right now, there is no frequency parameter on the statement urls. But, if you supply one (freq=Q or freq=A), it will toggle the javascript dropdown select for "Annual" and "Quarterly".

Macrotrends gives a lot more data back on the time-series than Yahoo does currently. I am using your stockdex macrotrends for Annual data, but for Quarterly data I am having to rely on Yahoo. But since Yahoo only returns back 4-5 quarters, there's just not enough data to run a model.

So, if I can run this with the freq parm, I can get both Annual AND Quarterly data back and ensemble those together.

I could probably hack this in my virtualenv and make it work, but I wanted to see if you would like to add that, or if you would like me to do a fork and pull request and do it. It looks like it would involve just adding a freq parm, validating it is Q or A.

From what I can tell, the data that comes back is the same, just with different time period.

wittling avatar Aug 21 '25 22:08 wittling

Thanks for reporting it @wittling, I was aware of this shortcoming but did not think many people would use it so I did not implement it. I dive into it in the coming days or would appreciate it if you could create a pull request with your solution.

ahnazary avatar Aug 22 '25 17:08 ahnazary

Not a problem! I am just glad to be able to get Quarterly data with it and not have to use difference data sources!

I forked it and made the changes. Will test them with a run later and hit you up for a pull request!

On Fri, Aug 22, 2025, 1:31 PM Amir Nazary @.***> wrote:

ahnazary left a comment (ahnazary/stockdex#32) https://github.com/ahnazary/stockdex/issues/32#issuecomment-3215097024

Thanks for reporting it @wittling https://github.com/wittling, I was aware of this shortcoming but did not think many people would use it so I did not implement it. I dive into it in the coming days or would appreciate it if you could create a pull request with your solution.

— Reply to this email directly, view it on GitHub https://github.com/ahnazary/stockdex/issues/32#issuecomment-3215097024, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHPXQLBLT3RGMIDYT65TCC33O5HWFAVCNFSM6AAAAACEQCPIBWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEMJVGA4TOMBSGQ . You are receiving this because you were mentioned.Message ID: @.***>

wittling avatar Aug 22 '25 22:08 wittling

Made changes and the job looks like it is running fine. Submitted Pull Request.

On Fri, Aug 22, 2025, 6:25 PM Mark Wittling @.***> wrote:

Not a problem! I am just glad to be able to get Quarterly data with it and not have to use difference data sources!

I forked it and made the changes. Will test them with a run later and hit you up for a pull request!

On Fri, Aug 22, 2025, 1:31 PM Amir Nazary @.***> wrote:

ahnazary left a comment (ahnazary/stockdex#32) https://github.com/ahnazary/stockdex/issues/32#issuecomment-3215097024

Thanks for reporting it @wittling https://github.com/wittling, I was aware of this shortcoming but did not think many people would use it so I did not implement it. I dive into it in the coming days or would appreciate it if you could create a pull request with your solution.

— Reply to this email directly, view it on GitHub https://github.com/ahnazary/stockdex/issues/32#issuecomment-3215097024, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHPXQLBLT3RGMIDYT65TCC33O5HWFAVCNFSM6AAAAACEQCPIBWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEMJVGA4TOMBSGQ . You are receiving this because you were mentioned.Message ID: @.***>

wittling avatar Aug 25 '25 18:08 wittling

don't approve the pull request. reject it.

it is still pulling annuals even if you supply ?freq=Q to the url. I am looking into it.

On Mon, Aug 25, 2025 at 2:54 PM Mark Wittling @.***> wrote:

Made changes and the job looks like it is running fine. Submitted Pull Request.

On Fri, Aug 22, 2025, 6:25 PM Mark Wittling @.***> wrote:

Not a problem! I am just glad to be able to get Quarterly data with it and not have to use difference data sources!

I forked it and made the changes. Will test them with a run later and hit you up for a pull request!

On Fri, Aug 22, 2025, 1:31 PM Amir Nazary @.***> wrote:

ahnazary left a comment (ahnazary/stockdex#32) https://github.com/ahnazary/stockdex/issues/32#issuecomment-3215097024

Thanks for reporting it @wittling https://github.com/wittling, I was aware of this shortcoming but did not think many people would use it so I did not implement it. I dive into it in the coming days or would appreciate it if you could create a pull request with your solution.

— Reply to this email directly, view it on GitHub https://github.com/ahnazary/stockdex/issues/32#issuecomment-3215097024, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHPXQLBLT3RGMIDYT65TCC33O5HWFAVCNFSM6AAAAACEQCPIBWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEMJVGA4TOMBSGQ . You are receiving this because you were mentioned.Message ID: @.***>

wittling avatar Aug 26 '25 13:08 wittling

Okay I finally figured out why the issue with it pulling annuals came from.

You don't know the "mnemonic" for the ticker symbol, so you are stubbing TBD in there. Well, as it turns out, TBD will still work - but it will only and always return JUST the annual statement. For the Quarterly statement to come back properly, you MUST put the right mnemonic into the url. https://www.macrotrends.net/stocks/charts/BRK.B/berkshire-hathaway /stock-price-history

I don't know if they're using the same strings as you would see from the SEC and at a glance, I suspect not. So - what I did, was to build a cache, and if the symbol isn't in it, it makes a creative fetch to ascertain what the mnemonic is and then saves it in the cache.

Hitting their site often enough to get the mnemonics is risky, so I put in a time sleep and the cache as a way of reducing the hits.

But - lo and behold - it is now working properly, it appears.

Let me let this run a while, and I will re-check it, and then check this code in and do a new pull request.

Mark

On Tue, Aug 26, 2025 at 9:12 AM Mark Wittling @.***> wrote:

don't approve the pull request. reject it.

it is still pulling annuals even if you supply ?freq=Q to the url. I am looking into it.

On Mon, Aug 25, 2025 at 2:54 PM Mark Wittling @.***> wrote:

Made changes and the job looks like it is running fine. Submitted Pull Request.

On Fri, Aug 22, 2025, 6:25 PM Mark Wittling @.***> wrote:

Not a problem! I am just glad to be able to get Quarterly data with it and not have to use difference data sources!

I forked it and made the changes. Will test them with a run later and hit you up for a pull request!

On Fri, Aug 22, 2025, 1:31 PM Amir Nazary @.***> wrote:

ahnazary left a comment (ahnazary/stockdex#32) https://github.com/ahnazary/stockdex/issues/32#issuecomment-3215097024

Thanks for reporting it @wittling https://github.com/wittling, I was aware of this shortcoming but did not think many people would use it so I did not implement it. I dive into it in the coming days or would appreciate it if you could create a pull request with your solution.

— Reply to this email directly, view it on GitHub https://github.com/ahnazary/stockdex/issues/32#issuecomment-3215097024, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHPXQLBLT3RGMIDYT65TCC33O5HWFAVCNFSM6AAAAACEQCPIBWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEMJVGA4TOMBSGQ . You are receiving this because you were mentioned.Message ID: @.***>

wittling avatar Aug 27 '25 21:08 wittling

New pull request submitted. I had to do a bit of an overhaul to how this worked, after getting deeper into it.

Let me know if you would like my fetch_statements.py module. It has some intelligence and efficiencies built into it.

On Wed, Aug 27, 2025 at 5:41 PM Mark Wittling @.***> wrote:

Okay I finally figured out why the issue with it pulling annuals came from.

You don't know the "mnemonic" for the ticker symbol, so you are stubbing TBD in there. Well, as it turns out, TBD will still work - but it will only and always return JUST the annual statement. For the Quarterly statement to come back properly, you MUST put the right mnemonic into the url. https://www.macrotrends.net/stocks/charts/BRK.B/berkshire-hathaway /stock-price-history

I don't know if they're using the same strings as you would see from the SEC and at a glance, I suspect not. So - what I did, was to build a cache, and if the symbol isn't in it, it makes a creative fetch to ascertain what the mnemonic is and then saves it in the cache.

Hitting their site often enough to get the mnemonics is risky, so I put in a time sleep and the cache as a way of reducing the hits.

But - lo and behold - it is now working properly, it appears.

Let me let this run a while, and I will re-check it, and then check this code in and do a new pull request.

Mark

On Tue, Aug 26, 2025 at 9:12 AM Mark Wittling @.***> wrote:

don't approve the pull request. reject it.

it is still pulling annuals even if you supply ?freq=Q to the url. I am looking into it.

On Mon, Aug 25, 2025 at 2:54 PM Mark Wittling @.***> wrote:

Made changes and the job looks like it is running fine. Submitted Pull Request.

On Fri, Aug 22, 2025, 6:25 PM Mark Wittling @.***> wrote:

Not a problem! I am just glad to be able to get Quarterly data with it and not have to use difference data sources!

I forked it and made the changes. Will test them with a run later and hit you up for a pull request!

On Fri, Aug 22, 2025, 1:31 PM Amir Nazary @.***> wrote:

ahnazary left a comment (ahnazary/stockdex#32) https://github.com/ahnazary/stockdex/issues/32#issuecomment-3215097024

Thanks for reporting it @wittling https://github.com/wittling, I was aware of this shortcoming but did not think many people would use it so I did not implement it. I dive into it in the coming days or would appreciate it if you could create a pull request with your solution.

— Reply to this email directly, view it on GitHub https://github.com/ahnazary/stockdex/issues/32#issuecomment-3215097024, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHPXQLBLT3RGMIDYT65TCC33O5HWFAVCNFSM6AAAAACEQCPIBWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEMJVGA4TOMBSGQ . You are receiving this because you were mentioned.Message ID: @.***>

wittling avatar Aug 29 '25 13:08 wittling

There is a new version of the package published (version 1.2.0) which covers fetching quarterly data from macrotrends. Install the latest version with pip install stockdex -U and fetch quarterly data following the example below:

from stockdex import Ticker

ticker = Ticker(ticker="AAPL")

income_statement = ticker.macrotrends_income_statement(frequency="quarterly")
balance_sheet = ticker.macrotrends_balance_sheet(frequency="quarterly")

print(income_statement)
print(balance_sheet)

This gives a full history of quarterly reports as seen in the result example below:

Image

Also thanks for the PR @wittling, I have added some comments on it.

ahnazary avatar Aug 30 '25 01:08 ahnazary

I will do.

BTW - I just put in another pull request, which has some additional enhancement functionality. I noticed that if symbols are delisted, macrotrends returns a lower url with the word "delisted" in it. I added some code on the statement fetches to raise a new DelistedException I created in exceptions.py file if that circumstance happens.

Changed 2 files for that - macrotrends_interface.py and exceptions.py.

Unfortunately, I had not synched to mainstream before making those changes and issuing the new pull request. Hopefully not too much of an effort to merge those changes in. ? If so I do apologize for not getting synched before I did that.

Mark

On Fri, Aug 29, 2025 at 9:08 PM Amir Nazary @.***> wrote:

ahnazary left a comment (ahnazary/stockdex#32) https://github.com/ahnazary/stockdex/issues/32#issuecomment-3238799550

There is a new version of the package published (version 1.2.0) which covers fetching quarterly data from macrotrends. Install the latest version with pip install stockdex -U and fetch quarterly data following the example below:

from stockdex import Ticker

ticker = Ticker(ticker="AAPL")

income_statement = ticker.macrotrends_income_statement(frequency="quarterly") balance_sheet = ticker.macrotrends_balance_sheet(frequency="quarterly")

print(income_statement) print(balance_sheet)

This gives a full history of quarterly reports as seen in the result example below: image.png (view on web) https://github.com/user-attachments/assets/95211623-189b-47b5-bef4-d7f2d1175fb0

Also thanks for the PR @wittling https://github.com/wittling, I have added some comments on it.

— Reply to this email directly, view it on GitHub https://github.com/ahnazary/stockdex/issues/32#issuecomment-3238799550, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHPXQLC3Z37NOWQQJARHPO33QD2PFAVCNFSM6AAAAACEQCPIBWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEMZYG44TSNJVGA . You are receiving this because you were mentioned.Message ID: @.***>

wittling avatar Sep 04 '25 15:09 wittling

I just pulled in your v1.2.0 with a pip install.

I don't think it is going to work properly without my other changes. If you try to query with just the parm of freq=Q, you will always get an Annual back (emailed you earlier about this).

Macrotrends seems to have a list of names(mnemonics) that go with the symbols. They do NOT match perfectly in all cases with the name registered w SEC.

So, I had to change the code to check a local cache and if not present, I fetch that mnemonic and put it on the cache. If the symbol is delisted in response lower url, I add it to a delisted cache (no sense fetching stuff you know cannot be fetched).

On my latest pull request, you will see this mnemonic added to the base class (as was in prv pull request), because one needs it to form the URL. Again it won't work for quarterly if that mnemonic is not in the URL string. You cannot use TBD as you were doing before. TBD will get you an annual but you cannot get a quarterly using TBD.

I also added a Delisted exception which I raise if I see the delisted in the response URL string. I noticed sometimes you see it on different http response codes so I check for it first before a raise.

On Thu, Sep 4, 2025, 11:48 AM Mark Wittling @.***> wrote:

I will do.

BTW - I just put in another pull request, which has some additional enhancement functionality. I noticed that if symbols are delisted, macrotrends returns a lower url with the word "delisted" in it. I added some code on the statement fetches to raise a new DelistedException I created in exceptions.py file if that circumstance happens.

Changed 2 files for that - macrotrends_interface.py and exceptions.py.

Unfortunately, I had not synched to mainstream before making those changes and issuing the new pull request. Hopefully not too much of an effort to merge those changes in. ? If so I do apologize for not getting synched before I did that.

Mark

On Fri, Aug 29, 2025 at 9:08 PM Amir Nazary @.***> wrote:

ahnazary left a comment (ahnazary/stockdex#32) https://github.com/ahnazary/stockdex/issues/32#issuecomment-3238799550

There is a new version of the package published (version 1.2.0) which covers fetching quarterly data from macrotrends. Install the latest version with pip install stockdex -U and fetch quarterly data following the example below:

from stockdex import Ticker

ticker = Ticker(ticker="AAPL")

income_statement = ticker.macrotrends_income_statement(frequency="quarterly") balance_sheet = ticker.macrotrends_balance_sheet(frequency="quarterly")

print(income_statement) print(balance_sheet)

This gives a full history of quarterly reports as seen in the result example below: image.png (view on web) https://github.com/user-attachments/assets/95211623-189b-47b5-bef4-d7f2d1175fb0

Also thanks for the PR @wittling https://github.com/wittling, I have added some comments on it.

— Reply to this email directly, view it on GitHub https://github.com/ahnazary/stockdex/issues/32#issuecomment-3238799550, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHPXQLC3Z37NOWQQJARHPO33QD2PFAVCNFSM6AAAAACEQCPIBWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEMZYG44TSNJVGA . You are receiving this because you were mentioned.Message ID: @.***>

wittling avatar Sep 04 '25 17:09 wittling

Ah. I noticed you pulled in and implemented the logic using the term "slug". Which I had not searched for (I had renamed that mnemonic in latest changes).

I guess that will work. I will test it. But the way you have it always calling that extra API call, we will get rate limited or penalized if we double the successive calls per symbol. This is why I chose to add mnemonic as an attribute, so that I could consult a cache and not always hit the API twice for every statement you want. If you are doing bulk fetches across 1-2k symbols both annual and quarterly, those calls add up.

On Thu, Sep 4, 2025, 1:31 PM Mark Wittling @.***> wrote:

I just pulled in your v1.2.0 with a pip install.

I don't think it is going to work properly without my other changes. If you try to query with just the parm of freq=Q, you will always get an Annual back (emailed you earlier about this).

Macrotrends seems to have a list of names(mnemonics) that go with the symbols. They do NOT match perfectly in all cases with the name registered w SEC.

So, I had to change the code to check a local cache and if not present, I fetch that mnemonic and put it on the cache. If the symbol is delisted in response lower url, I add it to a delisted cache (no sense fetching stuff you know cannot be fetched).

On my latest pull request, you will see this mnemonic added to the base class (as was in prv pull request), because one needs it to form the URL. Again it won't work for quarterly if that mnemonic is not in the URL string. You cannot use TBD as you were doing before. TBD will get you an annual but you cannot get a quarterly using TBD.

I also added a Delisted exception which I raise if I see the delisted in the response URL string. I noticed sometimes you see it on different http response codes so I check for it first before a raise.

On Thu, Sep 4, 2025, 11:48 AM Mark Wittling @.***> wrote:

I will do.

BTW - I just put in another pull request, which has some additional enhancement functionality. I noticed that if symbols are delisted, macrotrends returns a lower url with the word "delisted" in it. I added some code on the statement fetches to raise a new DelistedException I created in exceptions.py file if that circumstance happens.

Changed 2 files for that - macrotrends_interface.py and exceptions.py.

Unfortunately, I had not synched to mainstream before making those changes and issuing the new pull request. Hopefully not too much of an effort to merge those changes in. ? If so I do apologize for not getting synched before I did that.

Mark

On Fri, Aug 29, 2025 at 9:08 PM Amir Nazary @.***> wrote:

ahnazary left a comment (ahnazary/stockdex#32) https://github.com/ahnazary/stockdex/issues/32#issuecomment-3238799550

There is a new version of the package published (version 1.2.0) which covers fetching quarterly data from macrotrends. Install the latest version with pip install stockdex -U and fetch quarterly data following the example below:

from stockdex import Ticker

ticker = Ticker(ticker="AAPL")

income_statement = ticker.macrotrends_income_statement(frequency="quarterly") balance_sheet = ticker.macrotrends_balance_sheet(frequency="quarterly")

print(income_statement) print(balance_sheet)

This gives a full history of quarterly reports as seen in the result example below: image.png (view on web) https://github.com/user-attachments/assets/95211623-189b-47b5-bef4-d7f2d1175fb0

Also thanks for the PR @wittling https://github.com/wittling, I have added some comments on it.

— Reply to this email directly, view it on GitHub https://github.com/ahnazary/stockdex/issues/32#issuecomment-3238799550, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHPXQLC3Z37NOWQQJARHPO33QD2PFAVCNFSM6AAAAACEQCPIBWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEMZYG44TSNJVGA . You are receiving this because you were mentioned.Message ID: @.***>

wittling avatar Sep 04 '25 17:09 wittling