Thunderbird 115.12.2 and 115.13 freeze with FiltaQuilla 4.1 on Windows 11 / 21H2

[b00Ta]

I use Thunderbird 115.13.0 (115.12.2) + quickFilters 6.5.1 + Filta Quilla 4.0 (4.1) on Windows 11 21H2 Enterprise.

I have configured two IMAP accounts, the first one is active and is now hosted by Microsoft Outlook365 with our university domain, the second one is inactive now (it is old server and I keep it for an archive purposes).

With FiltaQuilla 4.0 everything I need works. But after update to 4.1 the Thunderbird within a while (it is random from several seconds to several minutes) starts freezing for many seconds and then run a little bit (like a second or two) and then freezes again. I am unable even to close it, I must kill it through task manager.

When it freeze whole GUI is unresponsive, the TB window cannot be moved at all. The left bottom status shows that it freeze in different states, sometimes is in "Looking for folder" state, in another freezing is in "Downloading headers..." or "Downloading messages...".

If it is running for a while after TB start and I run filters manually by [quickFiters] > [Run filters], it shows message that filters are running, like usually, and it takes some times, like usually, and then disappears, like usually, but nothing has been actually done.

The one thing I observed during freezing was when I look on Thunderbird processes by System Informer and looked at threads. The most CPU intensive thread points to some sound related call but no sound was produced.

I am now on FiltaQuilla 4.0 version with updates disabled as I need to work - I lost several hours by solving the problem (and needs to filter several tens of thousands of emails by plenty of rules after return from vacation).

[RealRaven2000]

can you check whether this is related to

• regular expression search in body
• multiple regular expression search terms in a single filter

there is some slightly changed code that uses the length of the message according to the same methods that Thunderbird uses. I will try to post a new version here that behaves closer the old way within TB115, but I am currently out of office (on annual leave until Aug 1st) so i cannot promise real deep patches within taht period of time. If I haven't come up with a solution by then, please ping me again here so I can prioritize this work.

[b00Ta]

I will try to investigate that on weekend, I have not so much free time during working days.

[RealRaven2000]

When testing, you can use debug mode and the additional settings you get when you right-click the debug mode checkbox for additional info in JS error console:

if there are operations related to reading / saving email body, you can activate

extensions.filtaquilla.debug.mimeBody = true

[cociweb]

Even on ubuntu 22.04 (v115.13.0/64 with the add-on version of 4.1)

Here is a standard log (without debug):

console.log: "FiltaQuilla [logTime init]\n"
JavaScript error: chrome://filtaquilla/content/filtaquilla-util.js, line 596: InternalError: allocation size overflow
console.log: "FiltaQuilla 15:14:53.440  [45534 ms]   \n"
JavaScript error: chrome://filtaquilla/content/filtaquilla-util.js, line 596: InternalError: allocation size overflow
console.log: "FiltaQuilla 15:22:1.163  [427723 ms]   \n"

Then after the crash: Exiting due to channel error.

Usually the crash is happening after console.log: "FiltaQuilla [logTime init]\n" even if the debug logging is activated.There is no additonal log even if debug (incl. mimeBody) logs are turned on

I've also catched in the TB crash report, that all of the available 16GB is allocated....

The root cause is unclear: it may or may not be regex-related. For testing purposes, I've disabled all rules containing regex. Now, TB does not crash. Again for testing purposes I've added a simple rule with BCC and the console returns the following: JavaScript error: chrome://filtaquilla/content/filtaquilla.js, line 1356: TypeError: headerParser.parseHeadersWithArray is not a function The rule does not apply... I'm not sure if it worked before or not, as it is a new rule for me.

[RealRaven2000]

Can you try the latest prerelease? filtaquilla-4.3pre66.zip

there are some improvements in handling anything that's related to streaming the message (e,g, to regex body)

---

To try the version above download zip file and drag the file into Thunderbird Add-ons Manager (do not extract contents, it won't install like that)

[cociweb]

The 128.4.3esr (64-bit) is currently undergoing testing on ubuntu 24.02. Over the past hour, it appears that the issue has not reoccurred following the application of the patch and reenable regex filters. ;)

[b00Ta]

I apologize for long delay. I use BetterBird 115.17.0-bb35 (64-bit) on Windows 11 now as it is more stable and responsive then Thunderbird and also supports logical terms grouping in filters which allows more complex filters which I need.

I finally did some tests, but on that BetterBird instead of ThunderBird:

• FiltaQuilla 4.0
  • all my filters works as expected
  • no BetterBird hanging, but after several days of running on laptop, where I move a lot and network are changed frequently, the filters stops silently suddenly working but BetterBird continue to works, restart of BetterBird solve the issue
  • times to times happens that filters doesn't works even after BetterBird restart, so then I start Thunderbird 115.17 with the same filters and it works - to be precise, now the filters differs a little bit as I use advanced filtering in BetterBird but in near past the filters where exactly the same and the same problem was there too, so it might be related to BetterBird itself - I didn't debug that at that time when it happens and usually have not time to do that, when it happens - I usually need to solve the issue ASAP
• FiltaQuilla 4.1
  • BetterBird hangs within a while
  • in the debug console it stops on exception "InternalError: allocation size overflow" on file filtaquilla-util.js, line number 596. The values are not available as the whole debug console window disappears before it shows it and whole BetterBird hangs (for screenshot see files section).
• FiltaQuilla 4.2
  • BetterBird hangs within a while
  • in the debug console it stops on exception "InternalError: allocation size overflow" on file filtaquilla-util.js, line number 625. In this case the debug console stays and shows values. It looks like it repeatedly match the same string over and over and add it to the end of current txtResults variable. On exception the variable count has value 20468923, which is a lot, I would expect just one match and just one match (for screenshot see files section).
• FiltaQuilla 4.3pre66
  • BetterBird doesn't hang
  • but some filters doesn't works
  • these filters have "Subject Regex Match" filter and also "Body Regexp match" filter among others (for redacted real example see JSON fragment in files section)
  • in the debug console it shows (for screenshot see files section):
    • subjectRegex RESULT: true
    • MailStringUtils is not defined
    • currentPart is undefined
    • mime parser retrieved no data!

Files section

redacted problematic-filter-on-FQ-4.3pre66.utf-8.json

[RealRaven2000]

thanks - can you send me a matching email off list (export as eml format which is easiest to import) so I can test your regular expression with it. I noticed that you are not using the /regex/switches syntax (theoretically FiltaQuilla then adds its own switches unless we introduced a regression in the last round of changes.

by the way I cannot zoom into your screenshots, they are marked private somehow (?). If you want to share console.log, text format is preferred, just use "copy all messages" from the context menu:

For the MailtStringUtils not being defined (not sure why it works in Thunderbird, maybe it's in a different scope there) try this quickfix version:

filtaquilla-4.3.1pre2.zip

(it may also fix the follow on problems for you)

[b00Ta]

It is strange that you cannot zoom the images. I tried to access them as a guest (without being logged in and from another browser) and have no issue. Moreover image thumbnails contains link to full sized PNG file (it may be saved as a file), which is actually what is loaded when you click on the thumbnail, but it is resized to fit the browser window until you click on it again to obtain full size.

Anyway, I attached redacted email you were asking for (Redacted RegEx Test Email.zip).

I also quick test the FiltaQuilla 4.3.1pre2 version and it again cannot import MailStringUtils. But I took closer look to source code and changed line 33 to var { MailStringUtils } = ChromeUtils.import("resource:///modules/MailStringUtils.jsm"); and then it starts working. May be that is the issue.

[b00Ta]

It looks like there is an use after free bug in "Body RegExp Match", when it matches more emails in the row (one call on whole message folder). See the log (console-export-2024-11-20_8-41-41.txt).

Due to transition from standard jsm to mjs it would be better to use try { Import mjs } catch { ImportESModule sys.jsm }.

[RealRaven2000]

I also quick test the FiltaQuilla 4.3.1pre2 version and it again cannot import MailStringUtils. But I took closer look to source code and changed line 33 to var { MailStringUtils } = ChromeUtils.import("resource:///modules/MailStringUtils.jsm"); and then it starts working. May be that is the issue.

Yes but that will break in Betterbird / Thunderbird 128 - the jsm files are slowly going away as part of "ESMifying" the sources - since I am currently catering for 115 + 128 in the same build, the correct way to patch for both (which I will include in the next test version) is:

  var { AppConstants } = ChromeUtils.importESModule("resource://gre/modules/AppConstants.sys.mjs");
  FiltaQuilla.ESM = parseInt(AppConstants.MOZ_APP_VERSION, 10) >= 128;

  var { MailStringUtils } = FiltaQuilla.ESM
    ? ChromeUtils.importESModule("resource:///modules/MailStringUtils.sys.mjs")
    : ChromeUtils.import("resource:///modules/MailStringUtils.jsm");

Here is a new patched version:

filtaquilla-4.3.1pre3.zip

[RealRaven2000]

It looks like there is an use after free bug in "Body RegExp Match", when it matches more emails in the row (one call on whole message folder). See the log (console-export-2024-11-20_8-41-41.txt).

not sure what I am looking at here, just a pure log? Looks like an email was printed that contained a log in it as well - confusing...

[b00Ta]

It looks like there is an use after free bug in "Body RegExp Match", when it matches more emails in the row (one call on whole message folder). See the log (console-export-2024-11-20_8-41-41.txt).

not sure what I am looking at here, just a pure log? Looks like an email was printed that contained a log in it as well - confusing...

This is a pure log from error console (CTRL+SHIFT+J in this case but the same is accessible in browser toolbox via CTRL+SHIFT+I). The decoded MIME BASE-64 body of an email, which is searched for Body RegExp Match, is printed there by your filtaquilla-util.js, line 1049. The problem is that garbage after printed body content of one of the two emails. I create test set (test_set.zip) with two redacted emails and included log as well. I had these two and only these two emails in one separated folder and I run the filters on that folder. The matching filter should be that one I posted here previously.

[b00Ta]

I also quick test the FiltaQuilla 4.3.1pre2 version and it again cannot import MailStringUtils. But I took closer look to source code and changed line 33 to var { MailStringUtils } = ChromeUtils.import("resource:///modules/MailStringUtils.jsm"); and then it starts working. May be that is the issue.

Yes but that will break in Betterbird / Thunderbird 128 - the jsm files are slowly going away as part of "ESMifying" the sources - since I am currently catering for 115 + 128 in the same build, the correct way to patch for both (which I will include in the next test version) is:

  var { AppConstants } = ChromeUtils.importESModule("resource://gre/modules/AppConstants.sys.mjs");
  FiltaQuilla.ESM = parseInt(AppConstants.MOZ_APP_VERSION, 10) >= 128;

  var { MailStringUtils } = FiltaQuilla.ESM
    ? ChromeUtils.importESModule("resource:///modules/MailStringUtils.sys.mjs")
    : ChromeUtils.import("resource:///modules/MailStringUtils.jsm");

Here is a new patched version:

filtaquilla-4.3.1pre3.zip

This works. Still persist the problem where there is some garbage in decoded email body, see previous post for details including test set.

[RealRaven2000]

This works. Still persist the problem where there is some garbage in decoded email body, see previous post for details including test set.

I tried the email, took out the first 3 search terms and only tested the body in Thunderbird 115.15 - log looked good here:

====> text/html:
 <html>

<head>

<meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head>

<html>
    <head>
      <title>
      </title>
      <meta content="ders_mailer" name="AUTHOR">
      <meta content="UK" name="COPYRIGHT">
      <meta content="AS Verso, DERS s.r.o., http://www.ders.cz" name="GENERATOR">
    </head>
    <body>
      <div>
        <h1 style="color:dark_green">Informace o úspěšném dokončení pumpy FNP_PUMPA (ID=62879)</h1>
        <h2>Pumpa doběhla úspěšně - doba trvání běhu: 00:00:04 (spočítaný čas běhu).</h2>
        <p>Záznam běhu pumpy:<pre>
2024-11-18 00:31:32 Start pumpy
2024-11-18 00:31:33 Odstraněny problematické záznamy, které nemají pracoviště z číselníku pracovišť FNP (761 záznamů)
2024-11-18 00:31:33 Ošetření duplicitních RČ osob, problematické záznamy uloženy tabulce error_log_s_lide (žádné záznamy)
2024-11-18 00:31:33 Promazání duplicitních řádků 0
2024-11-18 00:31:33 Páruji ručně vložené osoby a úvazky
2024-11-18 00:31:33 Napárováno 22 osob a 0 úvazků.
2024-11-18 00:31:33 Aktualizace osobních údajů v S_LIDE (tabulka)
2024-11-18 00:31:34 Problém při aktualizaci osobních údajů v S_LIDE (záznamy v error_log_s_lide) (457 záznamů)
2024-11-18 00:31:34 Aktualizace údajů v S_LIDE (46 záznamů)
2024-11-18 00:31:34 Aktualizace osob v modulu GAP (žádné záznamy)
2024-11-18 00:31:34 Aktualizace osob v modulu OBD (žádné záznamy)
2024-11-18 00:31:34 Aktualizace osob v modulu PORTAL (žádné záznamy)
2024-11-18 00:31:35 Aktualizace úvazků (505 záznamů) bez chyb
2024-11-18 00:31:35 Žádné úvazky na zaniklých pracovištích
2024-11-18 00:31:35 Žádné nové úvazky
2024-11-18 00:31:35 Vložení vazeb pracovních vztahů pro moduly do S_LIDEPRAC_MODUL(žádné záznamy)
2024-11-18 00:31:35 Ukončení neplatných úvazků
2024-11-18 00:31:35 Ukončeno úvazků: 0
2024-11-18 00:31:35 Promazání duplicitních úvazků
2024-11-18 00:31:35 Smazáno duplicitních úvazků: 0
2024-11-18 00:31:35 Smazání řádků z S_LIDEPRAC_MODUL, smazáno 0 řádků.
2024-11-18 00:31:35 Odpárování osob, které již nemají platný úvazek
2024-11-18 00:31:35 Odpárováno osob: 22
2024-11-18 00:31:36 Nastavím osoby na aktivní pokud mají aktivní úvazek, nastaveno u 0 řádků.
2024-11-18 00:31:36 Konec pumpy bez fatálních chyb
</pre></div>
    </body>
  </html>
</html> filtaquilla-util.js:1051:21

so no garbage was appended here.

[RealRaven2000]

Also tested in Betterbird 115.14.0-bb31, with same result.

[cociweb]

The 128.4.3esr (64-bit) is currently undergoing testing on ubuntu 24.02. Over the past hour, it appears that the issue has not reoccurred following the application of the patch and reenable regex filters. ;)

Confirmed, Thank you for your work!

[RealRaven2000]

Ok marking as fixed in 4.3.1 - Published 21/11/2024