Mails getting moved to Invalid, but process fine if dragged to Inbox again

[Raithmir]

I've had Parsedmarc working fine for a couple of weeks now, but I've noticed that there'll be quite a few mails that get moved to Invalid when it would appear they aren't.

I periodically move them back to Inbox again whenever I see a load in Invalid, and for the vast majority of them they'll process fine this second time around. Any ideas what this is likely to be? Can I set some re-try value?

[seanthegeek]

Maybe your nameservers are timing out? try setting different ones,

[Raithmir]

Could be, but I've not specified different ones, I'm just using the default.

I'm seeing "too many open files" in the logs, yet I haven't specified n_procs either.

[Raithmir]

Jun 11 09:52:16 ukbbsldmarc01 parsedmarc[259736]: [38B blob data] Jun 11 09:52:16 ukbbsldmarc01 parsedmarc[259736]: WARNING:init.py:1092:Message with subject "Report domain: XXXXXX.com Submitter: google.com Report-ID: Jun 11 09:52:16 ukbbsldmarc01 parsedmarc[259736]: 15875890874485070008" is not a valid aggregate DMARC report: Unexpected error: [Errno 24] Too many open files Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: Traceback (most recent call last): Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: File "/usr/local/bin/parsedmarc", line 8, in Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: sys.exit(_main()) Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: File "/opt/venvs/parsedmarc/site-packages/parsedmarc/cli.py", line 614, in _main Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: process_reports(results) Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: File "/opt/venvs/parsedmarc/site-packages/parsedmarc/cli.py", line 93, in process_reports Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: number_of_replicas=replicas Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: File "/opt/venvs/parsedmarc/site-packages/parsedmarc/elastic.py", line 297, in save_aggregate_report_to_elasticsearch Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: begin_date = human_timestamp_to_datetime(metadata["begin_date"]) Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: File "/opt/venvs/parsedmarc/site-packages/parsedmarc/utils.py", line 247, in human_timestamp_to_datetime Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: return dateparser.parse(human_timestamp, settings=settings) Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: File "/opt/venvs/parsedmarc/site-packages/dateparser/conf.py", line 84, in wrapper Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: return f(*args, **kwargs) Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: File "/opt/venvs/parsedmarc/site-packages/dateparser/init.py", line 53, in parse Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: data = parser.get_date_data(date_string, date_formats) Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: File "/opt/venvs/parsedmarc/site-packages/dateparser/date.py", line 418, in get_date_data Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: locale, date_string, date_formats, settings=self._settings) Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: File "/opt/venvs/parsedmarc/site-packages/dateparser/date.py", line 196, in parse Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: return instance._parse() Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: File "/opt/venvs/parsedmarc/site-packages/dateparser/date.py", line 200, in _parse Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: date_obj = self._parsersparser_name Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: File "/opt/venvs/parsedmarc/site-packages/dateparser/date.py", line 213, in _try_freshness_parser Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: return freshness_date_parser.get_date_data(self._get_translated_date(), self._settings) Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: File "/opt/venvs/parsedmarc/site-packages/dateparser/freshness_date_parser.py", line 151, in get_date_data Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: date, period = self.parse(date_string, settings) Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: File "/opt/venvs/parsedmarc/site-packages/dateparser/freshness_date_parser.py", line 94, in parse Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: self.now = datetime.now(self.get_local_tz()) Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: File "/opt/venvs/parsedmarc/site-packages/dateparser/freshness_date_parser.py", line 46, in get_local_tz Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: return get_localzone() Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: File "/opt/venvs/parsedmarc/site-packages/tzlocal/unix.py", line 165, in get_localzone Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: _cache_tz = _get_localzone() Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: File "/opt/venvs/parsedmarc/site-packages/tzlocal/unix.py", line 86, in get_localzone Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: tz = pytz.timezone(etctz.replace(' ', '')) Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: File "/opt/venvs/parsedmarc/site-packages/pytz/init.py", line 181, in timezone Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: raise UnknownTimeZoneError(zone) Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: pytz.exceptions.UnknownTimeZoneError: 'Europe/London' Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: Traceback (most recent call last): Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: File "/opt/pypy3/lib-python/3/multiprocessing/util.py", line 262, in _run_finalizers Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: finalizer() Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: File "/opt/pypy3/lib-python/3/multiprocessing/util.py", line 186, in call Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: res = self._callback(*self._args, **self._kwargs) Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: File "/opt/pypy3/lib-python/3/shutil.py", line 486, in rmtree Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: _rmtree_safe_fd(fd, path, onerror) Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: File "/opt/pypy3/lib-python/3/shutil.py", line 408, in _rmtree_safe_fd Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: onerror(os.listdir, path, sys.exc_info()) Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: File "/opt/pypy3/lib-python/3/shutil.py", line 405, in _rmtree_safe_fd Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: names = os.listdir(topfd) Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: OSError: [Errno 24] Too many open files: '/tmp/pymp-7d4pwdqd' Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: Traceback (most recent call last): Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: File "/opt/venvs/parsedmarc/site-packages/parsedmarc/utils.py", line 46, in _cleanup Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: shutil.rmtree(tempdir) Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: File "/opt/pypy3/lib-python/3/shutil.py", line 486, in rmtree Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: _rmtree_safe_fd(fd, path, onerror) Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: File "/opt/pypy3/lib-python/3/shutil.py", line 408, in _rmtree_safe_fd Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: onerror(os.listdir, path, sys.exc_info()) Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: File "/opt/pypy3/lib-python/3/shutil.py", line 405, in _rmtree_safe_fd Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: names = os.listdir(topfd) Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: OSError: [Errno 24] Too many open files: '/tmp/tmpr8x4ipdb' Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: debug: OperationError: Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: debug: operror-type: OSError Jun 11 09:52:20 ukbbsldmarc01 parsedmarc[259736]: debug: operror-value: [Errno 24] Too many open files: '/tmp/tmpr8x4ipdb' Jun 11 09:52:20 ukbbsldmarc01 systemd[1]: parsedmarc.service: Main process exited, code=exited, status=1/FAILURE Jun 11 09:52:20 ukbbsldmarc01 systemd[1]: parsedmarc.service: Failed with result 'exit-code'.