portfolio-performance
Trim historical quotes
Since the historical quotes occupy the majority of the storage in the XML file, I'd like to propose a solution to trim the quotes. This is a solution that can be combined with the work being done in #2363.
When a new security is added, by default the complete historical quotes are downloaded. However, at least in my case, I will be adding recent transactions and from a performance point of view historical quotes do not matter; only those between the first and last transaction. In my case, a file with 14MB is reduced to around 7MB using the strategy below.
The proposal is:
- Add a preference "Exclude quotes outside trades"
- When that preference is enabled use the date of the first trade (and if it exists, the date of the last trade) as boundaries
- Alternatively, use as boundary the first day of the year corresponding to the date of the first trade; which allows the user to maintain a relevant view of the quotes chart that still uses minimal storage
- Do not store quotes that are outside those boundaries
- If there are no trades, it downloads as quotes as usual; but as soon we have a trade the next "Update quotes" will remove unneeded historical quotes
I don't understand why the last trade should be a boundary. Newer quotes than the last trade are interesting to see how the stock developed and if it might be time to buy or sell. Or am I getting something wrong? Alternatively one could weed out daily quotes after a certain time like for instance quotes older than n years are only kept for the first of a month so one has a rough picture of what happened. This in combination with other weeding out strategies.
Trimming the data seams to me as a subsequent problem, created by the primary problem, which is slow loading and saving. Or is your issue really about space on your disk?
Why should we put effort into trimming data (with individual configurations,...) when we could solve this problem (and many more) by speeding up the loading and saving?
I don't understand why the last trade should be a boundary. Newer quotes than the last trade are interesting to see how the stock developed and if it might be time to buy or sell. Or am I getting something wrong?
I should have been more specific: last sell trade and where there are no longer any shares held. Since there are no shares, the quotes are irrelevant. I do agree that if anyone uses the tool to track new buying opportunities they do need the recent quotes.
Alternatively one could weed out daily quotes after a certain time like for instance quotes older than n years are only kept for the first of a month so one has a rough picture of what happened.
Keeping quotes is important for performance calculation, and that is why I propose only to trim quotes that do not affect that calculation. The quotes older than the first buy and quotes newer than the last sell (where sell caused shares held to be 0).
Or is your issue really about space on your disk?
It is actually about storage. My case might be "special", but I do record all the stocks I buy/sell, even if they are just day trading. Maybe, I'm just a perfectionist and want to get an accurate performance calculation. This leads to huge file, that then I keep a copy in a cloud storage.
Another consequence of a big file – due to having many securities – is the sync up time. Everywhere we open the file, quotes are synced. For example, when we reach a point with 1000 securities, downloading quotes for all, even if we don't have any shares held for some time seems kind of a waste.
Why should we put effort into trimming data (...) by speeding up the loading and saving?
That is a very good question and why I still consider the work to optimise loading/saving a priority. The proposal to trim quotes, is more of a hygiene which combined with the above will lead to even better results both in terms of storage as well as quote updates.
I did a quick calculation and applying this strategy in my case can reduce my current file to 1/40 of it's current size.
Thanks for the information. Now I understand your situation better.
For me three solutions come to mind, which bring benefit to a lot of people and may solve some of your problems:
- Thinning out quotes: when I look at charts over 5 or 10 years I don't need daily data. Maybe weekly data would be enough. Only for the last two or three years I need daily data.
- Faster loading and saving compressed files. (For my private data the solution of #2363 reduces the file size from 114 MB to 11 MB. I don' know if this solution is really the best idea, but it shows that there is a lot of room for improvements.)
- Stopping the automatic updates of all stock quotes. Maybe a simple checkbox to stop the updates for a security would help? Or a select box for weekly quote updates?