QuickFolders
QuickFolders copied to clipboard
RealRaven2000
Feature Request: Make all folders in IMAP folder structure visible to quickMove/quickJump with non-synced folder tree
(Thunderbird 78.7.1 on Kubuntu Linux 20.04)
We have a small multi-user environment with tens of thousands of mail-containing (on IMAP) folders, in a deep and wide folder/directory structure up to 12 levels deep. We do not download the contents of mail messages to the local user's profile structure.
Currently (and for the last 10+ years), whenever we install a significant Thunderbird upgrade or when a user's Thunderbird panacea.dat file gets corrupted (typically, pre- TB 78, corrupted or became too large every 18-20 months), we have to rebuild the user's profile IMAP structure so that .msf files are created for each folder.
Having the .msf files exist has been the only way that I have found for "filing programs" like QuickFolder or Nostagy/Nostalgy++ to "see" the structure and the folders so that the user can, in such a program, search for the folders into which the user wishes to move or copy emails, or to go-to the searched-for folder.
The only way that I have found in the last 10+ years to build/re-build that structure [user's profile IMAP structure containing the .msf files] is to...
... Manually, using the left-side folder pane in the Thunderbird GUI, open each folder that contains either folders-containing-emails and/or folders-that-thus-contain-more-folders. One I manually open/expose each folder, then Thunderbird creates the .msf file for the folders contained within the folder just opened.
In our environment, prior to TB 78.7.1, we were using a much older TB version and this opening/exposing process could take one person several hours (at least 4-5 hours if one sits in front of the program and waits until they can do each one) for each user/seat. Under TB 78.7.1 and with faster new computers and more memory, it now only takes about 2 hours. However, currently, with 4 users/seats, that is still more than a full day of clicking on folders to open them.
Important: When I say "open/expose", I mean click on any folder that contains folders. It is not necessary to click on the actual mail-containing folders that don't contain folders.
Example of structure: There are far more than 10,000 folders that actually contain mail themselves. Other folders contain either folders-that-do-not-contain-mail themselves and/or folders-that-do-contain-mail. [When I say "contain mail", the mail message content stays on the IMAP server and is not downloaded to the user's computer in our environment.]
[email protected]
[Local User Folders (Inbox, Sent, etc.) ... NOT of concern for this issue]
IMAP folder, i.e. public/ ......
folder-level-1-company-abc
...folder-level-2-marketing
......folder-level-3-current-projects
........folder-level-4-project-x
..............folder-level-5-meeting-plans (contains mail)
..............folder-level-5-meeting-proposals (contains mail)
..............folder-level-5-contracts
...................folder level-6-contract-1 (contains mail)
...................folder level-6-contract-2 (contains mail)
.........folder-level-4-project-y
..............folder-level-5-meeting-plans (contains mail)
..............folder-level-5-meeting-proposals (contains mail)
...folder-level-2-personnel
..................[lots of structure]
...folder-level2-customers
..................[lots of structure]
For years, I have been operating under the assumption that the .msf files have to be created -- but mail-contents of a folder (if any) does not have to be viewed -- in order for the search functions in programs like QuickFolders to list them in the search function.
It may be now that all that is required is for the folders to be "Subscribed to", but that still requires a lot of time/work, though it is easier than it used to be.
I am open to any possibilities, but based on my previous experience, it seems that what is needed is the a function that walks / iterates through the entire structure and "opens/exposes", in turn, all the folders so as to create .msf files for them all.
To start I changed the title to be less based on implementation detail (whether they are stored in msf files is a detail which isn't really part of your requirement, and also the Add-on cannot see these files anyway). The front-end code currently works with XPCOM interfaces that are built on top of GLODA (Thunderbird's global database) which is updated through complicated asynchronous code that cannot be changed / overwritten by an Add-on - it's mainly written in C++ and compiled (hard-baked) into the Thunderbird platform. So if it has hard limitations then we will have to find other workarounds (such as forcing the folder tree to update itself by connecting to the server and creating the missing data)
The raw data (messages of a folder) are stored in a file without extension. msf files are database summaries that help searching the mails, but they are not necessary for the folder to be visible in the tree. If a folder has an (even empty) subfolder this is named with the ".sbd" extension, which doesn't necessarily have to contain any files (it can be an empty folder with no messages or it can just contain subfolders).
The "Repair Folder" command which can be initiated from the folder properties dialog will parse the raw message file and build a database index and as such delete and rebuild the .msf file so that contents can be accessed quickly and the Thunderbird search functions can work on it. The process is called "indexing folders" and this activity can be witnessed in Tools / Activity Manager. Large amounts of data / folders will result in long processing times which can be seen there.
For years, I have been operating under the assumption that the .msf files have to be created -- but mail-contents of a folder (if any) does not have to be viewed -- in order for the search functions in programs like QuickFolders to list them in the search function.
This is correct - but I have no internal knowledge how exactly Thunderbird builds it's tree in Imap when you are not subscribed to a folder. My guess is that it will not generate any subfolders for any unsubscribed folders, so there is no way for an Add-on to see these folders either. The Add-ons cannot talk directly to the Server. A quick and dirty approach would be to write a maintenance routine that temporarily subscribes to missing folders in order to create the structure, but I am not sure if it is possible to suppress the part where it starts to download (sync) the data, which is probably not helpful in this case as it may take days and lead to a large amount of data being stored locally.
As I was digging around I did find a Server Setting (clicking on the Advanced button) with a checkbox [x] Show only subscribed folders which seems to imply that it should be possible to show all folders without subscription. The hope is to find a Thunderbird function (or set of functions) that updates this information from the remote data into the folder tree.
Unfortunately, experimenting with what you have described has not resulted in a solution for me.
The raw data (messages of a folder) are stored in a file without extension. msf files are database summaries that help searching the mails, but they are not necessary for the folder to be visible in the tree.
In our environment, in the user's profile IMAP structure such as:
/d/home/USER/.thunderbird/12345abc.default-release/ImapMail/mail.FOO.com/public.sbd/...
-
There are ONLY .sbd directories and .msf files. There are NO files or directories without extensions.
-
It does not matter in that regard (no files without extensions) whether or not the folders have or have not been subscribed too.
-
I am not sure how it is accomplished, but NO mail messages (in the IMAP public structure) are actually downloaded to the user's computer or home directory on the server. That is not a problem or concern for us -- it may be a concern for other environments, however.
-
There seem to be different states of the "tree". In the Subscribe dialog one can see the entire tree, even though .msf files have not been yet created. Even though you can see the entire tree in the Subscribe dialog, you cannot see the tree in the left-side folder tree pane of Thunderbird, until you open each folder-that-contains-folders. See more about this below...
This is correct - but I have no internal knowledge how exactly Thunderbird builds it's tree in Imap when you are not subscribed to a folder.
I do not know either (of course). However....
-
The IMAP (same as above) folders in the tree are not visible in the left-side folder pane until the user manually opens folders-that-contain-folders-or-files, using the GUI "left-side folder pane".
-
The entire IMAP (same as above) folder tree is visible in the Subscribe dialog box. For our large (many thousands of folders) structure, the entire list is populated in about 2-3 seconds.
-
In old (very old?) versions of Thunderbird, the folder tree in the Subscribe dialog box was first presented (in a new installation) in closed (un-expanded) state. However, at some point along the line, and certainly by 78.7.1, the entire IMAP folder tree in the Subscribe dialog box in a new installation is presented to the user in fully expanded / opened state. Also the user can select all folders in the tree (or a range of folder) by clicking on the first desired folder and shift-clicking on the last desired folder. And then the user can click the Subscribe button to subscribe to a range of folders. This process creates .msf files (and the necessary .sbd directories needed to contain the .msf files), but they are just initial skeleton" versions of the files and they are NOT "visible" to either QuickFolders (or Nostalgy++) search.
-
Apparently the skeleton folders created by the Subscribe dialog are not fully registered by the Thunderbird database -- or are not registered in a manner that QuickFolders (or Nostalgy++) is looking for them. Perhaps it would be worth considering how and what QuickFolders is looking for in the Thunderbird database -- maybe there is something more/different that QuickFolders could be looking for in the database to qualify the folders to show up in QuickFolders search results. I say this because as a result of subscribing to folders the .msf files DO exist (but they are just skeletons) -- and also Thunderbird does know the folders exist in the IMAP structure (because they are shown in the Subscribe dialog).
The "Repair Folder" command which can be initiated from the folder properties dialog will parse the raw message file and build a database index and as such delete and rebuild the .msf file so that contents can be accessed quickly and the Thunderbird search functions can work on it. The process is called "indexing folders" and this activity can be witnessed in Tools / Activity Manager. Large amounts of data / folders will result in long processing times which can be seen there.
Unfortunately, the "Repair Folder" feature is not a solution to this issue. In order to use "Repair Folder", one must -- as you say -- use the folder properties dialog. However, if you are using the folder properties dialog that means that you have gone to that folder using the GUI. And if you have gone to the folder using the GUI, then the complete .msf file (not just the skeleton of .msf from subscribing) has already been created by the process of going to the folder. Sorry!
A quick and dirty approach would be to write a maintenance routine that temporarily subscribes to missing folders in order to create the structure, but I am not sure if it is possible to suppress the part where it starts to download (sync) the data, which is probably not helpful in this case as it may take days and lead to a large amount of data being stored locally.
As stated above...
-
For my installations, no emails are being downloaded locally regardless subscribing or not, so that is not an issue for us. It could be an issue for other installations.
-
But, in my testing, simply subscribing to a folder creates the .msf file (and any necessary .sbd directories), but the .msf file is only an initial skeleton file and thus (for reasons unknown to me) those initial skeleton .msf files (and whatever is happening by that action in the Thunderbird database) does NOT make the folder visible to QuickFolder's search function.
As I was digging around I did find a Server Setting (clicking on the Advanced button) with a checkbox [x] Show only subscribed folders which seems to imply that it should be possible to show all folders without subscription. The hope is to find a Thunderbird function (or set of functions) that updates this information from the remote data into the folder tree.
As you and I had discussed previously (before this issue was posted), I do NOT have "Show only subscribed folders" check-marked. That (not checked -- i.e. show ALL) has been our standard for more than 10 years.
a) For me, all folders are shown in the Subscribe dialog box within 2-3 seconds after opening the dialog box.
b) In a new installation, the Thunderbird left-side-folder-pane shows (for the IMAP structure) only the first-level "public/" folder. Then when the user clicks to opens that, all existing folders (i.e. ALL folders at the second level) become visible (one, by one, by one; you can see them being created/read, one at a time on a slower system) and .msf and .sbd are created. Then when the user clicks to open any of the folders-that-contain-folders, those become visible in the same way, and so forth as you navigate deeper and deeper. It is only when that manual "opening / expanding" is done does "something happen" that makes the folders visible to the Search function of QuickFolders.
My thought about the solution is either... (I fully admit that I have no knowledge about the internal workings of Thunderbird, add-ons, or what the add-ons can and cannot access, etc. However my sense of the situation is that one or both the following "should" be possible...)
1a) First, the user (or an add-on) actually subscribes to all (desired) folders -- even if this has to be manually done by the user (it is not difficult). Thunderbird does create skeleton .msf files (and the necessary .sbd directories) when the user subscribes to folders. Thus there "should" be a "hook" that QuickFolders can use.
1b) Then, determine what QuickFolders is looking at/for in the Thunderbird database to decide what shows up in the QuickFolders search results list. Thunderbird knows what is and is not subscribed to -- and thus there "should" be a way for QuickFolders to look/for at that subscribed-folders information (in addition to QuickFolder's previous method, so no previously-working method becomes broken).
OR
- Determine what action Thunderbird is taking in the database, when the user, in the left-side-folder-pane, clicks on a folder-that-contains-folders to open/expand such folders. Whatever action Thunderbird is taking at that time, could possibly be replicated by an add-on which would iterate through the structure. While the add-on cannot access the IMAP server itself, as the add-on opens/expands each folder-that-contains-folders, that resulting information should, at that moment, become available to the add-on from the Thunderbird database.
Note: Even if this add-on-driven open/expand folders process is slow and clumsy, if the user can just walk away and let it run overnight, that is a tremendous improvement over the user having to spend HOURS manually clicking on all those folders-that-contain-folders ... for every user's seat.
Note: Perhaps it would be possible for an add-on to create it's own mirror structure of the folders in such a way that it is more efficient for the add on to search in its own structure. Of course such a structure would have to be updated/refreshed to bring it up to date (because of deletions and additions of folders), but that is a process the user could trigger every so often; or perhaps there is data in the Thunderbird database that the add-on could check to see if a refresh is needed (and then the add-on could inform the user that there is a change to the folder structure and a refresh should be done).
Unfortunately, experimenting with what you have described has not resulted in a solution for me.
The raw data (messages of a folder) are stored in a file without extension. msf files are database summaries that help searching the mails, but they are not necessary for the folder to be visible in the tree.
In our environment, in the user's profile IMAP structure such as:
/d/home/USER/.thunderbird/12345abc.default-release/ImapMail/mail.FOO.com/public.sbd/...
- There are ONLY .sbd directories and .msf files. There are NO files or directories without extensions.
That's because there are no emails in these folders? An old school advice for people who wanted to force rebuilding their databases was to delete msf files manually (while Thunderbird was not running of course). It would then reindex those folders and build fresh msf files. I suspect either there are no emails only subfolders on the server or no email has been downloaded.
- It does not matter in that regard (no files without extensions) whether or not the folders have or have not been subscribed too.
It really depends on whether there are mails in a folder on the server I think. If there are mails they are not stored in the msf files, that's just metadata; it could well include header data, but it can be rebuilt by downloading the headers from the server.
- I am not sure how it is accomplished, but NO mail messages (in the IMAP public structure) are actually downloaded to the user's computer or home directory on the server. That is not a problem or concern for us -- it may be a concern for other environments, however.
The only setting with Imap I know is that you can restrict mails to be not downloaded restricted by size or only download the mail headers (the actual mail is then downloaded from the server for viewing when you click on the entry in the list).
- There seem to be different states of the "tree". In the Subscribe dialog one can see the entire tree, even though .msf files have not been yet created. Even though you can see the entire tree in the Subscribe dialog, you cannot see the tree in the left-side folder tree pane of Thunderbird, until you open each folder-that-contains-folders. See more about this below...
Yes. My example code tries to do that for you, but it's not yet fully successful and my cause all mail to be downloaded but not exactly 100% of the folders to be downloaded, that's why I didn't upload it here yet.
This is correct - but I have no internal knowledge how exactly Thunderbird builds it's tree in Imap when you are not subscribed to a folder.
I do not know either (of course). However....
- The entire IMAP (same as above) folder tree is visible in the Subscribe dialog box. For our large (many thousands of folders) structure, the entire list is populated in about 2-3 seconds.... ... by 78.7.1, the entire IMAP folder tree in the Subscribe dialog box in a new installation is presented to the user in fully expanded / opened state. Also the user can select all folders in the tree (or a range of folder) by clicking on the first desired folder and shift-clicking on the last desired folder. And then the user can click the Subscribe button to subscribe to a range of folders. This process creates .msf files (and the necessary .sbd directories needed to contain the .msf files), but they are just initial skeleton" versions of the files and they are NOT "visible" to either QuickFolders (or Nostalgy++) search.
Looks like a Thunderbird bug to me.
- Apparently the skeleton folders created by the Subscribe dialog are not fully registered by the Thunderbird database -- or are not registered in a manner that QuickFolders (or Nostalgy++) is looking for them. **Perhaps it would be worth considering how and what QuickFolders is looking for in the Thunderbird database -- maybe there is something more/different that QuickFolders could be looking for in the database to qualify the folders to show up in QuickFolders search results.
I am using the low level XPCOM interfaces nsIMsgFolder and nsIMsgAccountManager.allFolders for iterating but they will eventually be replaced by even higher level API functions. They abstract away part of the Imap complexities and return a simple Array of items.
Unfortunately, the "Repair Folder" feature is not a solution to this issue. In order to use "Repair Folder", one must -- as you say -- use the folder properties dialog. However, if you are using the folder properties dialog that means that you have gone to that folder using the GUI. And if you have gone to the folder using the GUI, then the complete .msf file (not just the skeleton of .msf from subscribing) has already been created by the process of going to the folder. Sorry!
I did not suggest it as a solution, I merely explained that it rebuilds the metadata in the msf file. I don't think the msf files are necessary for Thunderbird to "see" the folders, but they have to exist on disk. Your problem seems to be that the folders are not generated automatically?
A quick and dirty approach would be to write a maintenance routine that temporarily subscribes to missing folders in order to create the structure, but I am not sure if it is possible to suppress the part where it starts to download (sync) the data, which is probably not helpful in this case as it may take days and lead to a large amount of data being stored locally.
As stated above...
- For my installations, no emails are being downloaded locally regardless subscribing or not, so that is not an issue for us. It could be an issue for other installations.
- But, in my testing, simply subscribing to a folder creates the .msf file (and any necessary .sbd directories), but the .msf file is only an initial skeleton file and thus (for reasons unknown to me) those initial skeleton .msf files (and whatever is happening by that action in the Thunderbird database) does NOT make the folder visible to QuickFolder's search function.
Looks like a bug in Thunderbird, I will ask the thunderbird developers core team why empty folders would be invisible to Thunderbird.
a) For me, all folders are shown in the Subscribe dialog box within 2-3 seconds after opening the dialog box.
The subscribe dialog is obviously not directly connected to the folder tree. That Thunderbird doesn't rebuild the tree looks like a Thunderbird bug to me, but it may be by design. Certainly not something QuickFolders can fix, that's why I suggested the helper function that forces visiting every folder and expanding it so you don't have to do it manually.
b) In a new installation, the Thunderbird left-side-folder-pane shows (for the IMAP structure) only the first-level "public/" folder. Then when the user clicks to opens that, all existing folders (i.e. ALL folders at the second level) become visible (one, by one, by one; you can see them being created/read, one at a time on a slower system) and .msf and .sbd are created. Then when the user clicks to open any of the folders-that-contain-folders, those become visible in the same way, and so forth as you navigate deeper and deeper. It is only when that manual "opening / expanding" is done does "something happen" that makes the folders visible to the Search function of QuickFolders.
And I bet there is a reason for precisely large profiles with many folders or slow connections, since the JavaScript was still single threaded I could imagine starting up Thunderbird and rebuilding the whole tree every time (Imap works more or less in real time) would severely delay it. But that's just a speculation, who knows why the design stuff in a certain way?
My thought about the solution is either... (I fully admit that I have no knowledge about the internal workings of Thunderbird, add-ons, or what the add-ons can and cannot access, etc. However my sense of the situation is that one or both the following "should" be possible...)
1a) First, the user (or an add-on) actually subscribes to all (desired) folders -- even if this has to be manually done by the user (it is not difficult). Thunderbird does create skeleton .msf files (and the necessary .sbd directories) when the user subscribes to folders. Thus there "should" be a "hook" that QuickFolders can use.
We have no direct access to the msf files in the future. only the API methods. At the moment I can still use XPCOM, which the frontend code of Thunderbird also uses, but they are eventually going to take this away. So it is important that Thunderbird must support the building of the folder tree through some command or mechanism we still need to discover. I already did debug the subscribe dialog last Monday for an hour but it is not easy to understand because it is event driven and the C++ parts that run in the background are hidden to me.
1b) Then, determine what QuickFolders is looking at/for in the Thunderbird database to decide what shows up in the QuickFolders search results list.
Even though Thunderbird uses a database internally we have no direct access here.
Thunderbird knows what is and is not subscribed to -- and thus there "should" be a way for QuickFolders to look/for at that subscribed-folders information (in addition to QuickFolder's previous method, so no previously-working method becomes broken). OR
- Determine what action Thunderbird is taking in the database, when the user, in the left-side-folder-pane, clicks on a folder-that-contains-folders to open/expand such folders. Whatever action Thunderbird is taking at that time, could possibly be replicated by an add-on which would iterate through the structure. While the add-on cannot access the IMAP server itself, as the add-on opens/expands each folder-that-contains-folders, that resulting information should, at that moment, become available to the add-on from the Thunderbird database.
I propose that Thunderbird should be able to build / refresh the tree from some command that's what we need to find. But it may well be event driven such as "we don't bother downloading deeply nested folders until the user expands a node on the tree, which will save time and make the UI more responsive". They only recently started introducing multi-process browsing (fissure) and then the speed penalty of Thunderbird rebuilding a tree in the background would be less punitive for the user.
Note: Even if this add-on-driven open/expand folders process is slow and clumsy, if the user can just walk away and let it run overnight, that is a tremendous improvement over the user having to spend HOURS manually clicking on all those folders-that-contain-folders ... for every user's seat.
I have already written the function but it didn't always reliable build all folders which is annoying. I will probably re-ask the question on the Tb developer matrix chat next week but I cannot do that too often because they will just ignore me as they have a lot of their own code to worry about. I don't even need to ask on the support and Add-on developers forums because that information is so incredibly low-level that even Core developers do not have it in their head - they would have to look up the flow in the code and or maybe try and find a developer who worked on this code over the last 10 years.
Note: Perhaps it would be possible for an add-on to create it's own mirror structure of the folders in such a way that it is more efficient for the add on to search in its own structure. Of course such a structure would have to be updated/refreshed to bring it up to date (because of deletions and additions of folders), but that is a process the user could trigger every so often; or perhaps there is data in the Thunderbird database that the add-on could check to see if a refresh is needed (and then the add-on could inform the user that there is a change to the folder structure and a refresh should be done).
I am not going to do that. It's hard enough synchronizing the tabs with their folders (which users can rename and move) without losing track and I have 10,000 lines of code to rewrite to make QuickFolders compatible as a web extension.
Here is a test version - you get the command to refresh the tree by holding down Shift while clicking the settings menu. Then select Debug > Force Tree Refresh
This will iterate all folders known to the tree at the time and brute force expand them in the tree, the hope is that it will expand all contained folders from the remote server. I did a few tests with my gmail account and closed the parent folder in the tree before I added children and grandchildren, but I am not sure if it is fully reliable. Also at the moment in the end all folders will be expanded in the tree, it's hard to remember which ones should be closed again (and difficult to predict whether that would impact on expanding). To get additional debug information you can enable "debug mode" on the QuickFolders Advanced Settings tab:
To install, download the zip file. then drag the zip file into Thunderbird Add-ons Manager. github doesn't allow xpi files in comments.
Unfortunately, experimenting with what you have described has not resulted in a solution for me.
The raw data (messages of a folder) are stored in a file without extension. msf files are database summaries that help searching the mails, but they are not necessary for the folder to be visible in the tree.
In our environment, in the user's profile IMAP structure such as:
/d/home/USER/.thunderbird/12345abc.default-release/ImapMail/mail.FOO.com/public.sbd/...
- There are ONLY .sbd directories and .msf files. There are NO files or directories without extensions.
That's because there are no emails in these folders? An old school advice for people who wanted to force rebuilding their databases was to delete msf files manually (while Thunderbird was not running of course). It would then reindex those folders and build fresh msf files. I suspect either there are no emails only subfolders on the server or no email has been downloaded.
I think we are having a communication problem, probably because we are each accustomed to different environments??? There are email messages in all the folders (on the IMAP server!!) that are "email containing". Sometimes only one or two email messages and sometimes 10,000 in one folder. However in each user's profile, if the folder tree has been expanded/opened so that the .msf files have been created, there are only .msf files and .sbd directories and NO files that do not have either .msf or .sbd on them. If you use a text editor to view the contents of those .msf files, the contents correctly describe the summaries of the email messages that they represent and for which the complete messages are on the IMAP server.
Our Thunderbird is configured to NOT download messages to the local user's profile space or hard disk. This is intentional. In a multi-user environment, where users are ALWAYS "ONLINE" (with the IMAP server), the data (email messages) should only be stored on the IMAP server.
- It does not matter in that regard (no files without extensions) whether or not the folders have or have not been subscribed too.
We have NO files (folders) in the user's profile area that do not have extensions.
It really depends on whether there are mails in a folder on the server I think. If there are mails they are not stored in the msf files, that's just metadata; it could well include header data, but it can be rebuilt by downloading the headers from the server.
There are email folders, containing email messages, physically on the IMAP server. None locally in the user's profile space or hard disk. The local user only has summaries (.msf and .sdb).
- I am not sure how it is accomplished, but NO mail messages (in the IMAP public structure) are actually downloaded to the user's computer or home directory on the server. That is not a problem or concern for us -- it may be a concern for other environments, however.
The only setting with Imap I know is that you can restrict mails to be not downloaded restricted by size or only download the mail headers (the actual mail is then downloaded from the server for viewing when you click on the entry in the list).
Correct. And that is what is happening in our environment.
- There seem to be different states of the "tree". In the Subscribe dialog one can see the entire tree, even though .msf files have not been yet created. Even though you can see the entire tree in the Subscribe dialog, you cannot see the tree in the left-side folder tree pane of Thunderbird, until you open each folder-that-contains-folders. See more about this below...
Yes. My example code tries to do that for you, but it's not yet fully successful and my cause all mail to be downloaded but not exactly 100% of the folders to be downloaded, that's why I didn't upload it here yet.
I really don't think that downloading of actual mail messages is a serious risk. I would be testing using a test user and I can simply delete that test user's Thunderbird profile directory.
This is correct - but I have no internal knowledge how exactly Thunderbird builds it's tree in Imap when you are not subscribed to a folder.
I do not know either (of course). However....
- The entire IMAP (same as above) folder tree is visible in the Subscribe dialog box. For our large (many thousands of folders) structure, the entire list is populated in about 2-3 seconds.... ... by 78.7.1, the entire IMAP folder tree in the Subscribe dialog box in a new installation is presented to the user in fully expanded / opened state. Also the user can select all folders in the tree (or a range of folder) by clicking on the first desired folder and shift-clicking on the last desired folder. And then the user can click the Subscribe button to subscribe to a range of folders. This process creates .msf files (and the necessary .sbd directories needed to contain the .msf files), but they are just initial skeleton" versions of the files and they are NOT "visible" to either QuickFolders (or Nostalgy++) search.
Looks like a Thunderbird bug to me.
- Apparently the skeleton folders created by the Subscribe dialog are not fully registered by the Thunderbird database -- or are not registered in a manner that QuickFolders (or Nostalgy++) is looking for them. **Perhaps it would be worth considering how and what QuickFolders is looking for in the Thunderbird database -- maybe there is something more/different that QuickFolders could be looking for in the database to qualify the folders to show up in QuickFolders search results.
I am using the low level XPCOM interfaces nsIMsgFolder and nsIMsgAccountManager.allFolders for iterating but they will eventually be replaced by even higher level API functions. They abstract away part of the Imap complexities and return a simple Array of items.
Unfortunately, the "Repair Folder" feature is not a solution to this issue. In order to use "Repair Folder", one must -- as you say -- use the folder properties dialog. However, if you are using the folder properties dialog that means that you have gone to that folder using the GUI. And if you have gone to the folder using the GUI, then the complete .msf file (not just the skeleton of .msf from subscribing) has already been created by the process of going to the folder. Sorry!
I did not suggest it as a solution, I merely explained that it rebuilds the metadata in the msf file. I don't think the msf files are necessary for Thunderbird to "see" the folders, but they have to exist on disk. Your problem seems to be that the folders are not generated automatically?
The .msf files and .sbd directories are NOT -- and never have been in 10+ years -- generated automatically for the folders that are on the IMAP server.
If I "subscribe" to folders on the IMAP server (which I can do), Thunderbird DOES create .msf files skeleton files only and the necessary .sbd directories that are needed. However, QuickFolders (and Nostalgy++) still don't "see" those folders when doing searches.
The only way I know of for QuickFolders (or Nostalgy++) to "see" the folders when searching is if I manually, in the left-side-folder-pane expand/open the folder structure to expose all folders. (I do NOT have to click on terminal [mail-containing] folders; I just have to expand the folder that contains them so that they become exposed/visible.)
A quick and dirty approach would be to write a maintenance routine that temporarily subscribes to missing folders in order to create the structure, but I am not sure if it is possible to suppress the part where it starts to download (sync) the data, which is probably not helpful in this case as it may take days and lead to a large amount of data being stored locally.
As stated above...
- For my installations, no emails are being downloaded locally regardless subscribing or not, so that is not an issue for us. It could be an issue for other installations.
- But, in my testing, simply subscribing to a folder creates the .msf file (and any necessary .sbd directories), but the .msf file is only an initial skeleton file and thus (for reasons unknown to me) those initial skeleton .msf files (and whatever is happening by that action in the Thunderbird database) does NOT make the folder visible to QuickFolder's search function.
Looks like a bug in Thunderbird, I will ask the thunderbird developers core team why empty folders would be invisible to Thunderbird.
The Susbcribe-created .msf folders are NOT "empty" -- they just have a skeleton contents that is essentially only the name of the folder plus about 20 header lines that are common to all folders, but NOT including the summary of the contents of the folder that is on the IMAP server. Would you like me to post an example??
a) For me, all folders are shown in the Subscribe dialog box within 2-3 seconds after opening the dialog box.
The subscribe dialog is obviously not directly connected to the folder tree. That Thunderbird doesn't rebuild the tree looks like a Thunderbird bug to me, but it may be by design. Certainly not something QuickFolders can fix, that's why I suggested the helper function that forces visiting every folder and expanding it so you don't have to do it manually.
It depends upon what you mean by "tree". The Subscribe dialog correctly represents the "tree" on the IMAP server. However, the left-side-folder-pane of Thunderbird does not even know that "tree" exist until the user manually expands/opens the folders-that-contain-folders.
I agree that it is a Thunderbird bug/feature -- but it seems like bad design to me!
The "helper function" sounds like a wonderful solution as you described.
b) In a new installation, the Thunderbird left-side-folder-pane shows (for the IMAP structure) only the first-level "public/" folder. Then when the user clicks to opens that, all existing folders (i.e. ALL folders at the second level) become visible (one, by one, by one; you can see them being created/read, one at a time on a slower system) and .msf and .sbd are created. Then when the user clicks to open any of the folders-that-contain-folders, those become visible in the same way, and so forth as you navigate deeper and deeper. It is only when that manual "opening / expanding" is done does "something happen" that makes the folders visible to the Search function of QuickFolders.
And I bet there is a reason for precisely large profiles with many folders or slow connections, since the JavaScript was still single threaded I could imagine starting up Thunderbird and rebuilding the whole tree every time (Imap works more or less in real time) would severely delay it. But that's just a speculation, who knows why the design stuff in a certain way?
I think you are correct. Our 10-year-old server and gigabit network CAN handle the load -- at least if only done from time to time. However, our PREVIOUS Thunderbird (on older workstations, but the older workstations still had more power than the old server), would not have been able to handle that. I don't think the problem was hardware -- I think that OLD Thunderbird was not up to that task.
My thought about the solution is either... (I fully admit that I have no knowledge about the internal workings of Thunderbird, add-ons, or what the add-ons can and cannot access, etc. However my sense of the situation is that one or both the following "should" be possible...) 1a) First, the user (or an add-on) actually subscribes to all (desired) folders -- even if this has to be manually done by the user (it is not difficult). Thunderbird does create skeleton .msf files (and the necessary .sbd directories) when the user subscribes to folders. Thus there "should" be a "hook" that QuickFolders can use.
We have no direct access to the msf files in the future. only the API methods. At the moment I can still use XPCOM, which the frontend code of Thunderbird also uses, but they are eventually going to take this away. So it is important that Thunderbird must support the building of the folder tree through some command or mechanism we still need to discover. I already did debug the subscribe dialog last Monday for an hour but it is not easy to understand because it is event driven and the C++ parts that run in the background are hidden to me.
Understood. Everything is movng to the API methods. But that means the Thunderbird developers have to do more/better than they have been doing. If they restrict only to API methods, they must also accept the responsibility for annoying issues like this one!
1b) Then, determine what QuickFolders is looking at/for in the Thunderbird database to decide what shows up in the QuickFolders search results list.
Even though Thunderbird uses a database internally we have no direct access here.
My apologies. I thought you earlier said that you did have access to the Thunderbird database.
Thunderbird knows what is and is not subscribed to -- and thus there "should" be a way for QuickFolders to look/for at that subscribed-folders information (in addition to QuickFolder's previous method, so no previously-working method becomes broken). OR
- Determine what action Thunderbird is taking in the database, when the user, in the left-side-folder-pane, clicks on a folder-that-contains-folders to open/expand such folders. Whatever action Thunderbird is taking at that time, could possibly be replicated by an add-on which would iterate through the structure. While the add-on cannot access the IMAP server itself, as the add-on opens/expands each folder-that-contains-folders, that resulting information should, at that moment, become available to the add-on from the Thunderbird database.
I propose that Thunderbird should be able to build / refresh the tree from some command that's what we need to find. But it may well be event driven such as "we don't bother downloading deeply nested folders until the user expands a node on the tree, which will save time and make the UI more responsive". They only recently started introducing multi-process browsing (fissure) and then the speed penalty of Thunderbird rebuilding a tree in the background would be less punitive for the user.
I agree with you on what is needed.
I agree that it may indeed be "event driven" and only through that left-side-folder-pane.
Note: Even if this add-on-driven open/expand folders process is slow and clumsy, if the user can just walk away and let it run overnight, that is a tremendous improvement over the user having to spend HOURS manually clicking on all those folders-that-contain-folders ... for every user's seat.
I have already written the function but it didn't always reliable build all folders which is annoying. I will probably re-ask the question on the Tb developer matrix chat next week but I cannot do that too often because they will just ignore me as they have a lot of their own code to worry about. I don't even need to ask on the support and Add-on developers forums because that information is so incredibly low-level that even Core developers do not have it in their head - they would have to look up the flow in the code and or maybe try and find a developer who worked on this code over the last 10 years.
Understood. You don't want to annoy the TB developers. :-)
Note: Perhaps it would be possible for an add-on to create it's own mirror structure of the folders in such a way that it is more efficient for the add on to search in its own structure. Of course such a structure would have to be updated/refreshed to bring it up to date (because of deletions and additions of folders), but that is a process the user could trigger every so often; or perhaps there is data in the Thunderbird database that the add-on could check to see if a refresh is needed (and then the add-on could inform the user that there is a change to the folder structure and a refresh should be done).
I am not going to do that. It's hard enough synchronizing the tabs with their folders (which users can rename and move) without losing track and I have 10,000 lines of code to rewrite to make QuickFolders compatible as a web extension.
Understood.
Here is a test version - you get the command to refresh the tree by holding down Shift while clicking the settings menu. Then select Debug > Force Tree Refresh
This will iterate all folders known to the tree at the time and brute force expand them in the tree, the hope is that it will expand all contained folders from the remote server. I did a few tests with my gmail account and closed the parent folder in the tree before I added children and grandchildren, but I am not sure if it is fully reliable. Also at the moment in the end all folders will be expanded in the tree, it's hard to remember which ones should be closed again (and difficult to predict whether that would impact on expanding). To get additional debug information you can enable "debug mode" on the QuickFolders Advanced Settings tab:
To install, download the zip file. then drag the zip file into Thunderbird Add-ons Manager. github doesn't allow xpi files in comments.
I will work on testing this over the next couple of days. I will be using a testing user login, so no harm can be done. I will also do it at night when a spike in network load won't inconvenience anybody.
Unless you have already thought of it, one issue that I can see could be a problem is that the process may need to be "throttled" so that it only does X quantity per second or something like that. Otherwise, I can imagine "too many simultaneous processes" or "too many simultaneous hits to the IMAP server". If this helper is just forcing Thunderbird to do certain tasks, one could potentially over-run the capability to keep track of (remember) the tasks being requested -- maybe that could be the cause of the irregular results you noticed in your testing.
I sure appreciate the serious attention that you are giving this issue.
I am replying (and formatting) directly on github. I cannot handle a message that is longer than 100 lines long. So I am replying from memory. You refer to msf as "skeletons" - they are meta-data and contain the info contained in the mail headers. And the can be rebuild by querying the server. But they have nothing to do with the folders as such so I don't why we keep talking about them. If you delete all your msf files in your IMap folder you will still see the same folders in the Thunderbird folder tree; it is not going to make folders disappear.
You where asking about the "tree". I am referring to the folders that are accessible on the left hand side of the 3pane window, which is called "Folder Tree". These folders seem to correspond with the ones that are accessible to the "allFolders" iterator.
Of course the tree just attempts to represent the real folder hierarchy on the remote server (and the subscribe dialog is able to pull that whole information in a different, less bureaucratic way - probably using a tenth or a hundreth of the memory and event listeners that are generated in the "folder tree" (see above) displayed in the Thunderbird main window.
I am replying (and formatting) directly on github. I cannot handle a message that is longer than 100 lines long. So I am replying from memory.
Since you told me to use GitHub, I have only replied/formatted on GitHub. I have NOT replied by email.
You refer to msf as "skeletons" - they are meta-data and contain the info contained in the mail headers. And the can be rebuild by querying the server. But they have nothing to do with the folders as such so I don't why we keep talking about them. If you delete all your msf files in your IMap folder you will still see the same folders in the Thunderbird folder tree; it is not going to make folders disappear.
We are having two types of communication problems:
-
Terminology: we are using different words for the same or different things.
-
There are factual differences in our experiences.
We were talking about .msf files because...
A) .msf files. There are TWO different forms of .msf files. They are both .msf files, but how they have been created results in TWO different forms of them. I do NOT know if it is the form of .msf file that makes a difference in my issue OR if it is some OTHER aspect of the different process that creates the two forms of .msf files that cause the difference in my issue.
Aa) .msf files created by using the Subscribe dialog box. This creates .msf files (and the needed .sbd directories), but FOR ME they are only "skeletons". FOR ME they only have the INITIAL header of the .msf file itself -- they do NOT contain any mail-specific data. FOR ME, the process that created the subscribe-created .msf files, or perhaps the presence of these files, does NOT allow QuickFolders to "see" the corresponding folder structure represented by those .msf files. Below I will paste in an example of a Subscribe-created .msf file -- you will see that it contains no mail-specific data.
Ab) .msf files (and the needed .sbd directories) created by (or updated by) manually opening/expanding the folder structure in the left-side-folder-pane. FOR ME these .msf files DO contain mail-specific data such as information from the mail message headers, etc. FOR ME the process that creates THESE .mfs files, or perhaps the presence of these files, DOES allow QuickFolders to "see" the folder structure that THESE .msf files represent.
You where asking about the "tree". I am referring to the folders that are accessible on the left hand side of the 3pane window, which is called "Folder Tree". These folders seem to correspond with the ones that are accessible to the "allFolders" iterator.
Okay. There is also a "tree" in the Subscribe dialog.
The tree in the Subscribe dialog DOES correctly represent the tree that on the IMAP server.
The tree in the left-folder-pane does NOT correctly represent the tree that is either in the Subscribe dialog on the IMAP server. The tree in the left-folder-pane is MISSING 99.9% of the folder structure until each folder-that-contains-folders is manually opened/expanded.
Of course the tree just attempts to represent the real folder hierarchy on the remote server (and the subscribe dialog is able to pull that whole information in a different, less bureaucratic way - probably using a tenth or a hundreth of the memory and event listeners that are generated in the "folder tree" (see above) displayed in the Thunderbird main window.
Understood.
The problem seems to be that Thunderbird requires those "event listeners" to be triggered by manually opening/expanding, in the left-side-folder-pane, all the folders-that-contain-folders.... before QuickFolders can "see" them.
We just don't know why this is true and why programs such as QuickFolders CAN see the the folder structure resulting from the left-pane actions, but CANNOT see the folder structure when subscribing is done using the Subscribe dialog.
Hopefully the test version you posted will give us some useful answers. :-) I have already downloaded it.
I will be testing it over the next couple of days. (I will not be able to work on this tomorrow.) I will report back when I have some results.
=-=-= SAMPLE of an ENTIRE .msf "skeleton" file that was created by subscribing using the Subscribe dialog:
This is all that is in this "skeleton" .msf file (created by subscribing). There is no mail-specific data.
From this location: (obfuscated)
USER@SERVER:/PATH-TO/USER/.thunderbird/ABCD1234.default-release/ImapMail/mail.FOO.com/public.sbd/__jsa.sbd/___File_by_Person.sbd/a-f.sbd/a.sbd
COMMAND LINE: more sensitive_name_of_mail_folder (obfuscated)
// < <(a=c)> // (f=iso-8859-1) (80=ns:msg:db:row:scope:msgs:all)(81=subject)(82=sender)(83=message-id) (84=references)(85=recipients)(86=date)(87=size)(88=flags)(89=priority) (8A=label)(8B=statusOfset)(8C=numLines)(8D=ccList)(8E=bccList) (8F=msgThreadId)(90=threadId)(91=threadFlags)(92=threadNewestMsgDate) (93=children)(94=unreadChildren)(95=threadSubject)(96=msgCharSet) (97=ns:msg:db:table:kind:msgs)(98=ns:msg:db:table:kind:thread) (99=ns:msg:db:table:kind:allthreads) (9A=ns:msg:db:row:scope:threads:all)(9B=threadParent)(9C=threadRoot) (9D=msgOffset)(9E=offlineMsgSize) (9F=ns:msg:db:row:scope:dbfolderinfo:all) (A0=ns:msg:db:table:kind:dbfolderinfo)(A1=numMsgs)(A2=numNewMsgs) (A3=folderSize)(A4=expungedBytes)(A5=folderDate)(A6=highWaterKey) (A7=mailboxName)(A8=UIDValidity)(A9=totPendingMsgs) (AA=unreadPendingMsgs)(AB=expiredMark)(AC=version)(AD=forceReparse) (AE=fixedBadRefThreading)(AF=onlineName)(B0=folderName) (B1=highestModSeq)(B2=highestRecordedUID)> {1:^80 {(k^97:c)(s=9)} } {FFFFFFFD:^9A {(k^99:c)(s=9)} }
<(80=1)(81=0)(82=public/__jsa/___File_by_Person/a-f/a/sensitive_name_of_mail_folder)> {1:^9F {(k^A0:c)(s=9u)} [1(^AC=1)(^AD=0)(^AE=1)(^AF^82)]}
=-=-= END OF SAMPLE
We were talking about .msf files because...
A) .msf files. There are TWO different forms of .msf files. They are both .msf files, but how they have been created results in TWO different forms of them. I do NOT know if it is the form of .msf file that makes a difference in my issue OR if it is some OTHER aspect of the different process that creates the two forms of .msf files that cause the difference in my issue.
Aa) .msf files created by using the Subscribe dialog box. This creates .msf files (and the needed .sbd directories), but FOR ME they are only "skeletons". FOR ME they only have the INITIAL header of the .msf file itself -- they do NOT contain any mail-specific data. FOR ME, the process that created the subscribe-created .msf files, or perhaps the presence of these files, does NOT allow QuickFolders to "see" the corresponding folder structure represented by those .msf files. Below I will paste in an example of a Subscribe-created .msf file -- you will see that it contains no mail-specific data.
Ab) .msf files (and the needed .sbd directories) created by (or updated by) manually opening/expanding the folder structure in the left-side-folder-pane. FOR ME these .msf files DO contain mail-specific data such as information from the mail message headers, etc. FOR ME the process that creates THESE .mfs files, or perhaps the presence of these files, DOES allow QuickFolders to "see" the folder structure that THESE .msf files represent.
I don't think the msf files create folders or cause folders to be created it's the other way round. Also I cannot access msf files so they are irrelevant to me.
The tree in the Subscribe dialog DOES correctly represent the tree that on the IMAP server.
The tree in the left-folder-pane does NOT correctly represent the tree that is either in the Subscribe dialog on the IMAP server. The tree in the left-folder-pane is MISSING 99.9% of the folder structure until each folder-that-contains-folders is manually opened/expanded.
I know.
I recommend wrapping code etc with the code tags, using the button:
//
< <(a=c)> // (f=iso-8859-1)
(80=ns:msg:db:row:scope:msgs:all)(81=subject)(82=sender)(83=message-id)
(84=references)(85=recipients)(86=date)(87=size)(88=flags)(89=priority)
(8A=label)(8B=statusOfset)(8C=numLines)(8D=ccList)(8E=bccList)
(8F=msgThreadId)(90=threadId)(91=threadFlags)(92=threadNewestMsgDate)
(93=children)(94=unreadChildren)(95=threadSubject)(96=msgCharSet)
(97=ns:msg:db:table:kind:msgs)(98=ns:msg:db:table:kind:thread)
(99=ns:msg:db:table:kind:allthreads)
(9A=ns:msg:db:row:scope:threads:all)(9B=threadParent)(9C=threadRoot)
(9D=msgOffset)(9E=offlineMsgSize)
(9F=ns:msg:db:row:scope:dbfolderinfo:all)
(A0=ns:msg:db:table:kind:dbfolderinfo)(A1=numMsgs)(A2=numNewMsgs)
(A3=folderSize)(A4=expungedBytes)(A5=folderDate)(A6=highWaterKey)
(A7=mailboxName)(A8=UIDValidity)(A9=totPendingMsgs)
(AA=unreadPendingMsgs)(AB=expiredMark)(AC=version)(AD=forceReparse)
(AE=fixedBadRefThreading)(AF=onlineName)(B0=folderName)
(B1=highestModSeq)(B2=highestRecordedUID)>
{1:^80 {(k^97:c)(s=9)} }
{FFFFFFFD:^9A {(k^99:c)(s=9)} }
<(80=1)(81=0)(82=public/__jsa/___File_by_Person/a-f/a/sensitive_name_of_mail_folder)>
{1:^9F {(k^A0:c)(s=9u)}
[1(^AC=1)(^AD=0)(^AE=1)(^AF^82)]}
=-=-= END OF SAMPLE
I don't think the msf files create folders or cause folders to be created it's the other way round.
You are, of course, correct.
The presence of folders (on the IMAP server) results/allows the .msf files to be created. I thought that is what I had said, but perhaps I did not say it clearly.
.msf files are only created when either:
-
The user subscribes (FOR ME that only creates "skeleton" .msf files and the Subscribe process does NOT result in QuickFolders being able to see those subscribed folders in the tree.)
-
The user manually opens/exposes folders in the left-side-folder-pane tree. Until that open/expand action is taken, the .msf files don't exist (unless previously subscribed, but those are "skeletons"). That open/expand action must, I assume, read the IMAP server and thus creates/updates the .msf file.
From what you are saying, since you say QuickFolders does not do anything with the .msf files... the process of opening/expanding in the left-folder-pane, in addition to creating the .msf files, must do something else in the Thunderbird database (?) that does allow QuickFolders to see those folders in the folder tree. Until that action (opening/expanding) is taken, the Thunderbird database apparently does not know about the existence of the folders on the IMAP server (even though the Subscribe dialog does know about the folders on the IMAP server) -- thus QuickFolders is not able to know about the folders.
I suspect that if the Thunderbird developers were to "reinvent" these processes today, they would do it differently. :-)
I will be testing the "helper function" in the test version you supplied.
Here is a new version, which tries to restore all the folders that should be collapsed / invisible after the whole sync operation - this will prevent all items in the Imap tree to be expanded at the very end. Hopefully should make testing a little easier as well:
Simple tests look encouraging so far. I added some (empty) folders on the Imap server with the parent collapsed in the local tree. Once it was finished I could jump to the new folders using quickMove immediately, even they were collapsed in the final tree.
And another version - I decided to take out the routine that counts subfolders (I hoped to catch an instance when that changed but apparently it wasn't happening / helpful) which should make it perform faster:
One thing it has trouble with is with folders that were removed on the server - they are still displayed in the local tree. I had a few instances that only throw an error when I click the "orphaned" folder. Not sure what is going on here.
Hi Axel,
Thanks for the further new versions. As previously mentioned, I can't test until tomorrow, though last night I did get the test bed set up. Your own testing sounds promising. I am looking forward to testing it. I will come back to you as soon as I can.
Jay
TESTING of "pre38" for "FORCE TREE REFRESH" function
SUMMARY:
-
The function initially works (to open the folder structure), but only for a few upper levels. Then it silently dies. 95% (or more) of the folders-that-contain-folders are never opened.
-
As part of its function, Force Tree Refresh is actually causing the reading of the mail messages (to get the count), as if a user manually clicked on a folder-that-contains-mail, which results in displaying the message count for folders-that-contain-mail. The counting is UNNECESSARY (for me) and could be part of the reason that the function dies. I do not need the messages to be counted -- I only need the folders to be opened to expose the levels of folders below (so that QF "go-to, move, and copy" functions will be able to "see" the possible folders). I do not know how many total messages our IMAP server is holding, but it is certainly over 100,000 (more than 20 years of mail), and it is not necessary to count all of them (even if the "Force Tree Refresh" did not die before getting to the bottom levels of folders).
=-=-=-=
This testing was done with a new/fresh TB 78 installation, against our actual IMAP server folder tree.
I am using the latest version you supplied: QuickFolders-wx-5.6pre38.zip
Under "Subscribe", I have (as always) turned OFF "only show subscribed". The Subscribe dialog does correctly show the entire structure. Thunderbird IS seeing the whole IMAP structure in the Subscribe dialog. (But that does not seem to help QF see the IMAP structure.)
=-=-=
Force Tree Refresh starts, but appears to stop after one or two minutes. During that couple of minutes, 99% of the activity seems to be taken up by counting messages in folders-that-contain-mail. Even after 20 minutes, there is no indication of any further activity.
-
There are no error messages that I can see.
-
The Activity Manager does NOT show any activity related to this function. (It does show activity such as when I am writing a draft of this posting, using Thunderbird's Compose, and the Compose is auto-saved to Drafts, so Activity Manager IS basically functioning.)
-
I did turn on QF Debug Mode, but I am not sure what that is supposed to do. Debug mode did not do anything or show anything that I could see.
-
While the "Force Tree Refresh" is operating...
-
I did see activity in Thunderbird's, bottom status bar.
-
I did see folders being visibly created in the left-side-folder-pane.
-
I did see mail messages being counted <<< THIS IS NOT NECESSARY AND IT REQUIRES THAT THE CONTENTS OF EVERY FOLDER IS ATTEMPTING TO BE READ. If it worked to open all the levels of folders, it would attempt to read probably at least 100,000++ emails to count them. That is UNNECESSARY.
-
NOTE: AFTER the test, In the left-side-folder-pane-tree, when a user MANUALLY clicks on a folder-that-contains-folders, the folder opens and Thunderbird reads [the IMAP?] and displays the folders inside the folder that was just clicked (just as normal). TB does NOT count mail messages at that time (that is normal -- TB only normally counts mail messages when the user manually clicks on a folder-that-contains-mail. I am only trying to replicate the function of opening the folders, but without counting messages.
-
"Force Tree Refresh" opens a few levels of folders, but ONLY a few. There exists in the IMAP several more levels below the level at which the function dies.
-
It appears that "Force Tree Refresh" is dying before it gets very deep. Perhaps it is running out of allocated task/tree memory or the iterator simply can't handle the depth, or ???. (The tree is both broad and deep.) I do not see any error messages.
The following is a brief summary of what the final result looks like. If you would like me to email directly to you screen shots of what it really looks like, I will be happy to do that.
[email protected]
Inbox ALREADY VISIBLE, UPDATED COUNT
Sent ALREADY VISIBLE, UPDATED COUNT
(etc.)
public ALREADY VISIBLE (CLOSED), OPENS IT & CREATES THE FOLLOWING
folder-that-contains-mail-1 CREATES & COUNTS
folder-that-contains-mail-2 CREATES & COUNTS
folder that-contains-folders-X CREATES & OPENS
folder that-contains-folders-Y CREATES & OPENS
folder-that-contains-mail-a CREATES & COUNTS
folder-that-contains-mail-b CREATES & COUNTS
folder that-contains-folders-x CREATES, does **NOT** OPEN
folder that-contains-folders-y CREATES, does **NOT** OPEN
NOTHING BELOW/INSIDE THIS IS CREATED, OPENED, OR DISPLAYED
Several levels exist below this point.
If I manually click on one of those last "folder that-contains-folders-y" level of folders, it WILL open and display the folders inside it, as usual when manually clicking. Thus they do exist and Thunderbird can see them on the IMAP when I click them with the mouse.
However, the "Force Tree Refresh" does not open or display the folders inside that level or any farther down.
I have repeatedly tried running "Force Tree Refresh" and it does not appear to do anything further, though there is some extremely fast text displayed in the bottom status bar (much too fast to read).
Restarting Thunderbird and trying again to do "Force Tree Refresh" does not result in any further folders (displayed in the tree) created or counted.
Running Thunderbird from the command line (just simply "thunderbird" in our installation) typically results in displaying (at the command line) a few error and debug messages (nothing important). However, NO additional error/debug messages are displayed as a result of running "Force Tree Refresh".
=-=-=
I hope that this information is useful. If you have specific questions, I will do my best to answer them.
If you wish me to run a different test, I will be happy to do so.
Jay
Interesting test results - I would have guess that running it repeatedly it would "go deeper" - but maybe the tree nodes have to be actually expanded for this. I was hoping that this would not be necessary. You may have noticed that I managed to collapsed the originally collapsed nodes on the tree, which I figure is cleaner and should be better performance wise. I think the status bar is updated every time another folder is expanded. I don't think Thunderbird Activity Manager is aware (or cares) that I am the user is opening folders, and it doesn't necessary build the index file unless it is the current folder. I need ti rethink, maybe there is a different thing I can do.
Force Tree Refresh starts, but appears to stop after one or two minutes. During that couple of minutes, 99% of the activity seems to be taken up by counting messages in folders-that-contain-mail. Even after 20 minutes, there is no indication of any further activity.
the idea is that it adds missing nodes while iterating the folders it already knows. But if you do it repeatedly it would iterate the new folders, too and I was hoping that it would then drill deeper. you can watch what it does if you enable "debug mode" in QF advanced and keeping the JS error console open.
A couple updates...
-
I experimented further and manually opened a folder (just one level) that contains many more folders (some containing messages and some containing further-deep folders). I then again did "Force Tree Refresh". When I manually opened the folder, that triggered Thunderbird to read the IMAP and make its contents visible in the tree. THUS the effect in this state of running Force Tree Refresh was only to count the messages.
-
I was very wrong about the number of messages on our system. The first level of the folder (that is the only level that opened; and I was the one who opened it, not QF) just mentioned above, contained folders-containing-mail that had over 140,000 messages in it. I think the counting just done took about 90 minutes (I was not watching closely). Thus our total message count for more than 20 years is probably closer to 250,000 messages. That's a lot to count.
Interesting test results - I would have guess that running it repeatedly it would "go deeper" - but maybe the tree nodes have to be actually expanded for this. I was hoping that this would not be necessary. You may have noticed that I managed to collapsed the originally collapsed nodes on the tree, which I figure is cleaner and should be better performance wise. I think the status bar is updated every time another folder is expanded. I don't think Thunderbird Activity Manager is aware (or cares) that I am the user is opening folders, and it doesn't necessary build the index file unless it is the current folder. I need ti rethink, maybe there is a different thing I can do.
!!! YES, YES, YES: "...but maybe the tree nodes have to be actually expanded for this." That is what I am trying to do this whole time. Expand ALL TREE NODES and then TB should "see" them and make them visible to QF for move-to/copy-to/go-to.
Counting messages is completely optional for me (and would take a long time).
the idea is that it adds missing nodes while iterating the folders it already knows. But if you do it repeatedly it would iterate the new folders, too and I was hoping that it would then drill deeper.
Unfortunately, it is not drilling deeper. TB seems not to recognize that it is being asked to open the next level of nodes.
What is needed is a recursive command: "If folder is a folder-that-contains-folders, open it...".
You may have noticed that I managed to collapsed the originally collapsed nodes on the tree, which I figure is cleaner and should be better performance wise.
Actually, the function did not close any folders. At this time, it leaves open any folders that it opened (or which were already open). I am sorry that I failed to say this in my original test results report.
I think the function dies before it gets to the stage of closing folders.
Actually, closing the folders is not very important to me (but perhaps it would be to other users). Our hierarchical structure is such that I can close/hide thousands of folders by closing only a very few folders at a higher level.
Thank you for telling me about the JS error console. I had not used it in years and had forgotten about it.
That showed a couple interesting things:
When running "Force Tree Refresh"...
MESSAGE #1 in the JS error console:
QuickFolders 18:39:47.379 [141301 ms]
Parent panel {QuickFolders-PreviewToolbarPanel} is not on screen; moving current folder button for tabMode: folder
[Exception... "Component returned failure code: 0x80520001 (NS_ERROR_FILE_UNRECOGNIZED_PATH) [nsIXPCComponents_Utils.readUTF8URI]" nsresult: "0x80520001 (NS_ERROR_FILE_UNRECOGNIZED_PATH)" location: "JS frame :: resource://gre/modules/L10nRegistry.jsm :: L10nRegistry.loadSync :: line 658" data: no] L10nRegistry.jsm:658:19
MESSAGE #2 in the JS error console:
QuickFolders 18:39:59.67 [11688 ms]
switched TreeView mode= all
ALL THE REST OF THE MESSAGES (HUNDREDS)... in the JS error console: All the following (this is just a small sample) are terminal folders containing mail NONE of the following are folders that contain folders !!!
First the user's own area:
QuickFolders 18:39:59.94 [27 ms]
theTreeView.selectFolder(Inbox) => index = 1, hasSubFolders = false, open = false
QuickFolders 18:39:59.118 [24 ms]
theTreeView.selectFolder(Trash) => index = 5, hasSubFolders = false, open = false
QuickFolders 18:39:59.141 [23 ms]
........ etc.
Then the "public/" IMAP folders:
theTreeView.selectFolder(jay_orders) => index = 21, hasSubFolders = false, open = false
QuickFolders 18:39:59.434 [17 ms]
theTreeView.selectFolder(_spam_that_shouldnt_be) => index = 12, hasSubFolders = false, open = false
QuickFolders 18:39:59.448 [14 ms]
theTreeView.selectFolder(stock) => index = 25, hasSubFolders = false, open = false
QuickFolders 18:39:59.471 [23 ms]
theTreeView.selectFolder(jay) => index = 19, hasSubFolders = false, open = false
QuickFolders 18:39:59.489 [18 ms]
theTreeView.selectFolder(auctions) => index = 15, hasSubFolders = false, open = false
........ etc.
Again, ALL the folder names in this listing (of hundreds) are only the "terminal" (final in the structure) folders that actually contain mail.
There are NO folders that contain folders in this list.
Either the function is simply not logging the "opening" of folders, or it is not trying to open them???
Just to document what I am doing - the folder iterator has to start from somewhere - so it iterates all folders that are seen and tries to open them in the tree. The "open=false" log actually shows whether the folder is expanded in the tree. The code I am using to attempt a "drill down" is taken from Thunderbird's _toggleRow function
https://searchfox.org/comm-esr78/source/mail/base/content/folderPane.js#1284
In my tests I was able to search for (empty, so not mail containing) folders I added on Imap straight away even when the parent folder was collapsed. I don't really know what else to do, guess I need to find a Thunderbird Core developer who is interested in this topic
Interesting test results - I would have guess that running it repeatedly it would "go deeper" - but maybe the tree nodes have to be actually expanded for this. I was hoping that this would not be necessary. You may have noticed that I managed to collapsed the originally collapsed nodes on the tree, which I figure is cleaner and should be better performance wise. I think the status bar is updated every time another folder is expanded. I don't think Thunderbird Activity Manager is aware (or cares) that I am the user is opening folders, and it doesn't necessary build the index file unless it is the current folder. I need ti rethink, maybe there is a different thing I can do.
!!! YES, YES, YES: "...but maybe the tree nodes have to be actually expanded for this." That is what I am trying to do this whole time. Expand ALL TREE NODES and then TB should "see" them and make them visible to QF for move-to/copy-to/go-to.
That's exactly what the code tries to do. It should at least expand all the folders that are logged with "open=false" and thus expose their direct children. Anything that is listed in your log file like this should have a complete list of children afterwards. but I guess maybe I need to force to "download messages" on the folder (even if it is empty apart from subfolders). I have a download all messages command on context menu for the Imap folders, but at least on gmail it always tries to update my Inbox and Sent folders, which takes ages.
the idea is that it adds missing nodes while iterating the folders it already knows. But if you do it repeatedly it would iterate the new folders, too and I was hoping that it would then drill deeper.
Unfortunately, it is not drilling deeper. TB seems not to recognize that it is being asked to open the next level of nodes.
What is needed is a recursive command: "If folder is a folder-that-contains-folders, open it...".
it won't work that way around. because you do not know the fires part "is a folder-that-contains-folders" without opening it first. So my algorithm actually opens everything that has "open=false" in the log in order to force expanding it from the remote server - at least that's my theory. The actual communication with the Imap server is hidden to me as it happens super low level - one would need a C++ compiler to see what happens there and I just haven't got the time to set up a complete Thunderbird build environment.
There are NO folders that contain folders in this list. Either the function is simply not logging the "opening" of folders, or it is not trying to open them???
It tries to open all folders, regardless of whether they are expanded or not. It just works with the data that Thunderbird volunteers to give it.
It tries to open all folders, regardless of whether they are expanded or not. It just works with the data that Thunderbird volunteers to give it.
This is very puzzling because on the GUI, I can visually see the first-level-at-that-point-in-the-process of folders-that-contain-folders that the function is not "seeing", thus which the function is not trying to open.
Is it possible that some how the function's code is dying because of some environmental factor, such as limit to the number of threads, processes, forks, memory, etc. available to that function. As the tree spreads out, the management of the iteration becomes much, much more complex. The iterator has to "protect" itself and thus can't just open everything in sight, and then open everything below that, and everything below that, because in a very large structure either the iterator, or the subsequent "count messages" processes would run out of resources.
Of course, I don't have the knowledge to know, but my thinking is that the reason that the iterator is not "seeing" the deeper folders-that-contain-folders is that the iterator has already died by that time. Perhaps not a problem that Thunderbird is not making the information available, but that the iterator can't see it because it is already dead. ???
Is it possible to instruct the iterator, or to use an iterator designed for deeper/broader structures, to follow only a few branches of the tree at a time, and at each subsequent branch, have another such limit on the number of branches? Then when it hits the terminal end of the branches, it rises back up to the previous level and looks for more unopened branches?
It tries to open all folders, regardless of whether they are expanded or not. It just works with the data that Thunderbird volunteers to give it.
This is very puzzling because on the GUI, I can visually see the first-level-at-that-point-in-the-process of folders-that-contain-folders that the function is not "seeing", thus which the function is not trying to open.
it won't "die" without an exception (which my code catches and reports) or Thunderbird hanging. New version:
I added an endpoint to make sure. at which point it should say something like
QuickFolders 15:47:41.316 [26 ms]
refreshTree() iterated all accessible (62) folders.
Run it and then save the complete log output in a text file (that way you can search for the folder's name you refer to) using the "export visible messages to..." command.
P.S. feel free to upload the log file as an attachment here if you are comfortable with it.
It should at least expand all the folders that are logged with "open=false" and thus expose their direct children. Anything that is listed in your log file like this should have a complete list of children afterwards. but I guess maybe I need to force to "download messages" on the folder (even if it is empty apart from subfolders).
Just to be sure that there is no miscommunication in my test results report, the JS error console messages (of which I gave an example) contain absolutely NO listings for folders-that-contain-folders.
Thus, when you say "It should at least expand all the folders that are logged with "open=false" and thus expose their direct children." ... I want to be sure you are aware that it is NOT logging ANY folders-that-contain-folders.
Thus, if it is not logging them, perhaps there is some aspect of the function that is not behaving as expected.
I don't really know what else to do, guess I need to find a Thunderbird Core developer who is interested in this topic.
There are a couple things that support the possibility that there is something wrong (in regard to folder depth) either with the Thunderbird code that communicates with the IMAP, or something wrong with Thunderbird's [ _toggleRow] (https://searchfox.org/comm-esr78/source/mail/base/content/folderPane.js#1233) function (that you mentioned above).
-
In at least 10 years, "nobody" has had an answer to this -- though you have certainly done more work on it than anybody has ever tried to do.
-
On OLD Thunderbird (long before 78) the "Compact All Folders" function dies (silently, though I have not ever tried to watch in the JS error console). I have not wanted to test it on TB 78 as it can take a few hours before it dies and it puts a very heavy load on the server.
I hope it is possible to interest a Thunderbird Core developer in this topic.
Did you run the new version? A complete log would be helpful. "folders that contain folders" would mean the function doesn't even iterate any second level folders at all? so you basically only see folers at the root and nothing any deeper? e.g. f1 / f2
if you see f2 then you see f1 (which is a folder that contains f2). Not quite sure what a "folder that contains folder" means. Probably a folder without emails... But in my test these show too. So a complete log file and the path of one of these folders that is not seen would bring some clarity.
Did you run the new version?
I have not yet run QuickFolders-Lic-5.6pre93.zip. It may be a day before I can run it.
A complete log would be helpful. "folders that contain folders" would mean the function doesn't even iterate any second level folders at all? so you basically only see folers at the root and nothing any deeper? e.g. f1 / f2
I will be happy to supply the complete log, but I will either have to email it to you directly (I have your address already from our previous conversation) or I can post it in my webspace where you can download it. I do not want to post it publicly here.
if you see f2 then you see f1 (which is a folder that contains f2).
I do see f1, f2, f3, f4 ... but I see NO more below f4 (unless I manually click to open f3, which then properly reads f4 and then manually click to open f4, which then reads f5, etc, etc.).
Not quite sure what a "folder that contains folder" means. Probably a folder without emails... But in my test these show too. So a complete log file and the path of one of these folders that is not seen would bring some clarity.
Unfortunately, we are continuing to have the communication problem, that we had from the very beginning, with the meaning of "folders". This communication problem may be at the core of why it is difficult to make progress?
There are TWO types of folders, or you could say that folders can have TWO statuses.
- Folders that only contain mail (results in .msf files)
- Folders that contain other folders (results in .sbd directories)
Using your (f1, f2, etc.) illustration I will try again to describe:
In this example below, if I use the "Force Tree Refresh" function, the f4 level is visible, but f4 folders have NOT been opened, thus all the f5 folders (and below) are NOT visibile.
f1 ([email protected])
f2a Inbox (that users folders)
f2b Sent " "
f2c "public/" (the shared IMAP structure)
f3a [only contains folders]
f4a [only contains folders]
f5a (folder that contains mail) NOT VISIBLE
f5b (folder that contains mail) NOT VISIBLE
f5c (folder that contains mail) NOT VISIBLE
f4b [only contains folders]
f5m (folder that contains mail) NOT VISIBLE
f5n (folder that contains mail) NOT VISIBLE
f5n (folder that contains mail) NOT VISIBLE
f3b [only contains folders]
f4c [only contains folders]
f5x (folder that contains mail) NOT VISIBLE
f5y (folder that contains mail) NOT VISIBLE
f5z (folder that contains mail) NOT VISIBLE
f3c [only contains folders]
f4d [only contains folders]
f5r [only contains folders] NOT VISIBLE
f6a [only contains folders] NOT VISIBLE
f7a [only contains folders] NOT VISIBLE
f8a [only contains folders] NOT VISIBLE
f9a (folder that contains mail) NOT VISIBLE
f9b (folder that contains mail) NOT VISIBLE
f9c (folder that contains mail) NOT VISIBLE
Good example. But f3a and f4d also "only contain folders" and they are displayed, so what is different to f4a / f4b? The depth of level maybe?
Good example. But f3a and f4d also "only contain folders" and they are displayed, so what is different to f4a / f4b? The depth of level maybe?
In your question, all of f3a, f4d and f4a, f4b ARE visible. There is no difference between the type of folder they are; but f3a is a level above f4a, fb4 folders [f3a CONTAINS f4a, f4b]. (I only wrote "NOT VISIBLE" for the items that were not visible. All the others ARE visible).
In that illustration,the digits (3,4) represent the levl from the top. The letters (a,b,c) are just identifiers to be able to talk about, in this conversation, a particular line in the structure.
All the f3 folders are at a certain level, i.e. one level deeper than f2 . All the f4 folders are one deeper lever than f3 folders. (All the f3 folders are contained in one or more f2 folders; all the f4 folders are contained in one or more f3 folders.)
In that illustration, all the f3 and f4 folders are by coincidence folders-that-only-contain-folders. It is just representation of one sample hierarchical structure. Most private users probably do not have a structure that is so deep, but large companies probably have an even-deeper structure; we are only a small company.
You could add items f3d and/or f4e that are folders-that-contains-mail, last two lines in the illustration below:
f3c [only contains folders]
f4d [only contains folders]
f5r [only contains folders] NOT VISIBLE
f4e (folder that contains mail)
f3d (folder that contains mail)
Unfortunately, I don't know anything about other possible methods of installation, but in our installation, we CANNOT put mail directly into a "folder-that-contains-folders". In our installation it is only possible to put mail into a "terminal", mail-containing, folder-that-contains-mail, that is the final folder at the "end of the path".
=-=-=
Perhaps If we used the terminology of computing that I was taught, then we might not have so many complications in discussion.
However, I suspect that the communication problem is that there are types of Thunderbird installations which do allow "folders-that-contain-folders" that are ALSO ALLOWED to DIRECTLY CONTAIN MAIL. (If that is true, that is unfortunately very complicated.)
"Folders-that-contain-mail" are "files" in computing (and on the IMAP server) in our installation. These result in ONLY .msf files being created.
"Folders-that-contain-folders" are "directories" in computing (and on the IMAP server) in our installation. These result in BOTH .msf files AND .sbd directories being created.