Meeting detection in all apps + calendars integration

[mdmohsin7]

#3465

• [x] Detect meetings and show nub to start recording
• [x] Connect local macos calendar to detect meetings
• [x] Monitor local calendar continuously in the background
• [x] Intelligently manage when and when not to show nub
• [x] Show nub and or in menu bar for upcoming meetings (detected in calendar)
• [x] Use meeting context if it's in around 2 mins of the current recording
• [x] Pass meeting context to LLM while processing conversation
• [x] Code cleanup

https://github.com/user-attachments/assets/16df8e38-aba5-461e-99e1-df6a8156b951

[mdmohsin7]

/gemini summary

[gemini-code-assist[bot]]

Summary of Changes

This pull request delivers a significant enhancement by integrating calendar functionality and advanced meeting detection into the macOS application. The system monitors local calendars for scheduled meetings and observes microphone activity across all running applications. This dual approach enables the app to intelligently prompt users to record relevant conversations via a new 'nub' UI and automatically associate these recordings with their respective calendar events, providing richer context for subsequent AI processing.

Highlights

• Calendar Integration: Introduced a new feature to integrate with macOS calendars, allowing the application to detect upcoming meetings and provide timely prompts to start recording.
• Meeting Detection & Nub UI: Implemented a system to detect active microphone usage in meeting applications (including browser-based meetings) and display a floating 'nub' UI element. This 'nub' guides users to start recording and provides context for the meeting.
• Automatic Recording Context: Conversations started around a detected meeting time (within +/- 2 minutes) are automatically linked to the corresponding calendar event, enriching the conversation's context for processing by the LLM.
• Backend API & Data Models: Added new backend API endpoints and data models to store and manage calendar meeting information, ensuring that meeting context can be persisted and retrieved for conversation processing.
• macOS Sandboxing Disabled: Disabled application sandboxing on macOS to allow access to the system's logstream, which is crucial for detecting microphone usage across all applications.

Changelog

• app/lib/backend/http/api/calendar_meetings.dart
  • Added API calls for storing, retrieving, and listing calendar meetings.
• app/lib/backend/preferences.dart
  • Added new preferences for calendar integration settings, including enabling/disabling integration, showing events without participants, displaying meetings in the menu bar, and managing enabled calendar IDs.
• app/lib/backend/schema/calendar_meeting_context.dart
  • Introduced MeetingParticipant and CalendarMeetingContext models to structure meeting data from calendars.
• app/lib/desktop/pages/desktop_home_page.dart
  • Replaced 'Device Settings' with 'Calendar Integration' in the settings menu and updated navigation accordingly.
  • Removed unused dart:math import.
• app/lib/main.dart
  • Added CalendarProvider to the list of ChangeNotifierProviders, initialized for eager loading.
• app/lib/pages/settings/calendar_settings_page.dart
  • Created a new Flutter page for managing calendar integration settings, including toggles for macOS Calendar and a 'coming soon' placeholder for Google Calendar, along with display options.
• app/lib/providers/calendar_provider.dart
  • Added a new CalendarProvider to handle calendar permissions, monitor events, fetch meetings, and synchronize them with the backend.
• app/lib/providers/capture_provider.dart
  • Integrated CalendarProvider as a dependency.
  • Added callbacks for when recording is initiated from the 'nub' and for automatic recording stops.
• app/lib/services/calendar_service.dart
  • Added a new service to interface with native macOS calendar APIs for permissions, event monitoring, and data retrieval.
• app/lib/services/services.dart
  • Extended ISystemAudioRecorderService and DesktopSystemAudioRecorderService to include callbacks for recording initiation from the 'nub' and automatic stopping.
• app/macos/Runner.xcodeproj/project.pbxproj
  • Updated the Xcode project file to incorporate new Swift files (MeetingDetector.swift, NubManager.swift, NubWindow.swift, CalendarMonitor.swift).
  • Disabled app sandboxing to allow access to system logstream.
• app/macos/Runner/AppDelegate.swift
  • Modified applicationShouldTerminateAfterLastWindowClosed to keep the application running in the menu bar when the main window is closed.
• app/macos/Runner/CalendarMonitor.swift
  • Added a new Swift file implementing CalendarMonitor to interact with EventKit, detect upcoming meetings, apply filtering rules, and dispatch events to Flutter. Includes logic for meeting platform extraction and snooze management.
• app/macos/Runner/DebugProfile.entitlements
  • Disabled app sandboxing.
  • Added calendar access permissions.
• app/macos/Runner/Info.plist
  • Updated calendar usage descriptions to provide more detailed information to the user.
• app/macos/Runner/MainFlutterWindow.swift
  • Significantly refactored to integrate MeetingDetector and CalendarMonitor.
  • Set up new Flutter method and event channels for meeting detection and calendar events.
  • Implemented logic for tracking recording sources (manual, calendar, microphone, hybrid) and managing automatic recording stops based on meeting context.
• app/macos/Runner/MeetingDetector.swift
  • Added a new Swift file implementing MeetingDetector to monitor system logs for microphone activity.
  • Identifies meeting applications, including browser-based meetings by analyzing window titles.
  • Incorporates debouncing logic for microphone activity to prevent false positives.
• app/macos/Runner/MenuBarManager.swift
  • Modified to display upcoming meeting information directly in the macOS menu bar.
• app/macos/Runner/NubManager.swift
  • Added a new Swift file implementing NubManager to control the floating 'nub' UI element.
  • Manages different nub states (upcoming meeting, meeting started, microphone active, recording) and tracks the source of the meeting context.
• app/macos/Runner/NubWindow.swift
  • Added a new Swift file defining NubWindow, a custom NSPanel for the floating 'nub' UI.
  • Handles visual updates for various states, includes auto-dismiss timers, and interactive elements like 'Start Recording' and 'Close' buttons.
• app/macos/Runner/Release.entitlements
  • Disabled app sandboxing.
  • Added calendar access permissions.
• app/macos/Runner/RunnerDebug-dev.entitlements
  • Disabled app sandboxing.
  • Added calendar access permissions.
• app/macos/Runner/RunnerDebug-prod.entitlements
  • Disabled app sandboxing.
  • Added calendar access permissions.
• app/macos/Runner/RunnerDebug.entitlements
  • Disabled app sandboxing.
  • Added calendar access permissions.
• app/macos/Runner/RunnerProfile-prod.entitlements
  • Disabled app sandboxing.
  • Added calendar access permissions.
• app/macos/Runner/RunnerProfile.entitlements
  • Disabled app sandboxing.
  • Added calendar access permissions.
• app/macos/Runner/RunnerRelease-prod.entitlements
  • Disabled app sandboxing.
  • Added calendar access permissions.
• app/macos/Runner/RunnerRelease.entitlements
  • Disabled app sandboxing.
  • Added calendar access permissions.
• backend/database/calendar_meetings.py
  • Added new Python functions for CRUD operations on calendar meetings in Firestore, including finding meetings by calendar event ID and listing meetings within a specified time range.
• backend/database/redis_db.py
  • Added functions to store and retrieve a meeting_id associated with a conversation in Redis.
• backend/main.py
  • Included the new calendar_meetings router to expose calendar-related API endpoints.
• backend/models/conversation.py
  • Introduced MeetingParticipant and CalendarMeetingContext Pydantic models to represent detailed meeting information.
  • Updated the CreateConversation model to include an optional calendar_meeting_context.
• backend/routers/calendar_meetings.py
  • Added a new FastAPI router for calendar meeting API endpoints, enabling the storage, retrieval, and listing of calendar meetings.
• backend/routers/conversations.py
  • Modified the process_in_progress_conversation endpoint to accept calendar_meeting_context in the request body and inject it into the conversation's external data for processing.
• backend/routers/transcribe.py
  • Modified _create_new_in_progress_conversation to automatically detect and associate a calendar meeting with a new conversation if one is occurring within a +/- 2-minute window of the conversation start time.
• backend/utils/conversations/process_conversation.py
  • Modified to retrieve an associated meeting_id from Redis and fetch the corresponding CalendarMeetingContext from Firestore, then inject this context into the conversation's external data before processing.
• backend/utils/llm/conversation_processing.py
  • Updated extract_action_items and get_transcript_structure functions to accept and leverage calendar_meeting_context.
  • The LLM now uses participant names from the calendar context instead of generic 'Speaker X' labels, and incorporates meeting details to enhance the conversation title and overview.

Activity

• The author, mdmohsin7, requested a summary of the pull request.
• A bot provided a high-priority review comment suggesting a more robust if-else if structure for microphone session detection in MeetingDetector.swift to prevent state flickers.
• A bot provided a high-priority review comment highlighting a bug and unnecessary complexity in ProgressBarView within NubWindow.swift, recommending simplification by removing manual corner masking.

[mdmohsin7]

Gemini you forgot to include that we don't just rely on mic activity for detection, in case of browsers we also make use of the accessibility api of macos to check the title of the window that is open to be able to figure out which platform the user is using in browser for the meeting

Calendar events are synced to firestore to a meetings collection, whenever a conversation starts we check if there's any meeting in calendar around that time and link it (store in redis if there is). Might not be the best approach but added it for now

[beastoin]

what is the best alternative solution for keeping the app sandbox on? and its trade-offs

[mdmohsin7]

what is the best alternative solution for keeping the app sandbox on? and its trade-offs

We can rely on NSWorkspace to detect if the user is using zoom/teams etc but it will not be as exact and accurate as relying on mic usage, this approach doesn't require disabling sandboxing. This approach can be also improved (since we already take screen capture permission). Overall it won't really be as good as the current approach but it will work, we might not be able to auto stop recording when meeting ends in case of browsers.

@beastoin

[mdmohsin7]

e00a9dd replaces logstream (separate process) and Accessibility APIs usage with NSWorkspace and Core Graphics APIs to detect meetings both natively and in the browsers

(this approach is inspired by Aside which is pretty cool and AppStore safe, added additional stuff using core graphics APIs to cover a lot of browser related cases and also to make auto stop recording possible)

@beastoin

[beastoin]

i see. pls use logstream. feel free to merge it.

then open a new pr to add support:

1. auto update omi macos app (.dmg ver)
2. ci for .dmg, codemagic -> signed-dmg -> push to the download page

then we move omi macos app to .dmg only.