keep icon indicating copy to clipboard operation
keep copied to clipboard

feat(mailgun): Improve email parsing with enhanced DMARC detection and reprocess functionalityFeat/mailgun email parsing improvements

Open sanyo4ever opened this issue 3 months ago • 3 comments

Summary

This PR enhances the Mailgun provider to handle DMARC reports and other emails without body content, and adds functionality to reprocess failed email events.

Problem

Mailgun provider was failing to parse DMARC reports and other emails without body content, resulting in errors:

DMARC reports (common automated emails from Google, Microsoft, etc.) have no body text - only subject and ZIP attachments - causing the provider to crash.

Solution

1. Enhanced Email Parsing

DMARC Detection:

  • Multi-indicator detection using sender, subject, content-type, and attachments
  • Properly classifies DMARC, SPF, and bounce emails

Flexible Message Handling:

  • Handles emails without body content
  • Extracts message from subject + attachment descriptions
  • Multiple fallback levels for robustness

Email Classification:

  • Tags emails with email_type metadata (dmarc_report, spf_report, bounce, alert)
  • Allows filtering by email type

Configuration Options:

  • skip_dmarc_reports (default: false) - Option to skip DMARC reports
  • skip_spf_reports (default: false) - Option to skip SPF reports
  • handle_emails_without_body (default: true) - Handle emails without body content

2. Error Alert Reprocessing

Backend:

  • New endpoint: POST /alerts/event/error/reprocess
  • Database helpers: get_error_alerts_to_reprocess(), dismiss_error_alert_by_id()
  • Reprocess single or all error alerts
  • Auto-dismiss successfully reprocessed alerts

Frontend:

  • Added "Reprocess" buttons to Error Alerts Modal
  • Loading states and toast notifications
  • Detailed success/failure feedback

Use Case: After deploying code fixes, users can reprocess previously failed events without resending emails.

Changes

Backend

  • keep/providers/mailgun_provider/mailgun_provider.py - Enhanced email parsing logic
  • keep/api/routes/alerts.py - Reprocess endpoint and imports
  • keep/api/core/db.py - Database helper functions
  • docs/snippets/providers/mailgun-snippet-autogenerated.mdx - Updated documentation

Frontend

  • keep-ui/entities/alerts/model/useAlerts.ts - Reprocess hook function
  • keep-ui/features/alerts/alert-error-event-process/ui/alert-error-event-modal.tsx - UI buttons and handlers

Testing

Tested with:

  • ✅ DMARC reports from Google (no body, ZIP attachments)
  • ✅ Regular monitoring emails with body content
  • ✅ Reprocessing error alerts
  • ✅ Email classification and metadata

Results:

  • ✅ DMARC emails create alerts successfully
  • ✅ Message extracted from subject + attachments
  • ✅ No parsing errors
  • ✅ Reprocess functionality works for both new and old errors
  • ✅ Source facet counter updates correctly

Breaking Changes

None - fully backward compatible with existing functionality.

Screenshots

(Optional: Add screenshots of the Error Alerts Modal with Reprocess buttons)


Fixes: Mailgun DMARC email parsing failures Related: #5366 (if applicable)

sanyo4ever avatar Oct 16 '25 09:10 sanyo4ever

@sanyo4ever is attempting to deploy a commit to the KeepHQ Team on Vercel.

A member of the Team first needs to authorize it.

vercel[bot] avatar Oct 16 '25 09:10 vercel[bot]

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

:white_check_mark: shahargl
:x: sanyo4ever
You have signed the CLA already but the status is still pending? Let us recheck it.

CLAassistant avatar Oct 16 '25 09:10 CLAassistant

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Preview Comments Updated (UTC)
keep Ready Ready Preview Comment Nov 16, 2025 8:01am

vercel[bot] avatar Nov 16 '25 07:11 vercel[bot]

@sanyo4ever "Test docs" check keeps failing, can you have a look? also - you must sign on CLA so I can merge this PR

shahargl avatar Nov 16 '25 07:11 shahargl

Codecov Report

:x: Patch coverage is 11.76471% with 15 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 30.53%. Comparing base (97c263a) to head (477e658). :warning: Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
keep/api/core/db.py 11.76% 15 Missing :warning:
Additional details and impacted files
@@             Coverage Diff             @@
##             main    #5371       +/-   ##
===========================================
- Coverage   46.39%   30.53%   -15.87%     
===========================================
  Files         176      101       -75     
  Lines       18412    11682     -6730     
===========================================
- Hits         8543     3567     -4976     
+ Misses       9869     8115     -1754     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

codecov[bot] avatar Nov 16 '25 08:11 codecov[bot]