feat: frame-specific state based screenshots with preview

[reddygtvs]

Description

• Screenshots now target the most recently interacted frame
• Added screenshot preview functionality with smooth transitions
• Fixed context array issues with duplicate screenshot IDs

Related Issues

related to #2676

Type of Change

• [x] Bug fix
• [x] New feature
• [ ] Documentation update
• [ ] Release
• [ ] Refactor
• [ ] Other (please describe):

Testing

I switched from Firecrawl to DOM screenshotting for frame-specific screenshots because Firecrawl can't target individual iframes, only entire pages. Since we needed screenshots of the specific frame the user was interacting with (editorEngine.frames.getMostRecentlyInteractedFrame()), Firecrawl wasn't able to target it.

The DOM-to-canvas approach targets exactly the frame we want. The downside is rendering quality: fonts, spacing, and styling don't translate perfectly to canvas. Better solutions like Playwright need extra dependencies and risky permissions. I prioritized frame targeting and security over perfect visuals.

I also fixed a bug where multiple identical screenshots would break the context array by getting the same base64 ID. Added smooth preview transitions because Firecrawl screenshots aren't perfect and I couldn't tell what they were capturing without previewing.

Firecrawl still handles project thumbnails. DOM approach works for chat screenshots.

Screenshots (if applicable)

Additional Notes

P.S. Apologies on the git diffs, I had to rebase 3 different times today as there were a lot of pushes to the repo :) Although I was careful, this ended up being a much larger change than I initially expected, so I will keep iterating on it if anyone notices something wrong.

---

[!IMPORTANT] Introduces frame-specific screenshots and preview functionality, fixes duplicate screenshot ID issues, and updates frame interaction tracking.

• Behavior:
  • Screenshots now target the most recently interacted frame using getMostRecentlyInteractedFrame() in FramesManager.
  • Added screenshot preview functionality with smooth transitions in DraftImagePill.
  • Fixed context array issues with duplicate screenshot IDs in ChatContext.
• Functions:
  • captureScreenshot() in screenshot.ts now uses DOM-to-canvas rendering instead of Firecrawl.
  • addImageContext() added to ChatContext to handle image contexts.
• Classes:
  • FramesManager tracks frame interactions with updateLastInteraction() and getMostRecentlyInteractedFrame().
  • ElementsManager updates frame interaction on element click.
• Misc:
  • Minor changes in chat-input/index.tsx and input-context-pills.tsx to integrate new screenshot and preview features.

^{This description was created by for d6208c38253a463205a87e9449330469b688765d. You can customize this summary. It will automatically update as commits are pushed.}

Summary by CodeRabbit

• New Features

  • Click an image attachment pill to open a full-size preview modal.
  • Screenshots now capture the currently active frame for quicker, targeted sharing.

• Bug Fixes

  • More robust error handling and toasts when screenshots fail; prevents incomplete attachments.
  • Stabilized rendering of image attachment pills to reduce flicker.

• Refactor

  • Improved screenshot pipeline with graceful fallback image when capture isn’t possible.
  • Interaction-aware frame handling enhances consistency across selection, navigation, and capture flows.

[vercel[bot]]

@reddygtvs is attempting to deploy a commit to the Onlook Team on Vercel.

A member of the Team first needs to authorize it.

[coderabbitai[bot]]

[!NOTE]

Other AI code review bot(s) detected

CodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review.

Walkthrough

Reworks screenshot capture to use the most recently interacted frame; adds interaction tracking across frames (selection, navigation, element clicks); introduces a portal-based image preview for draft image pills; adds ChatContext.addImageContext; overhauls preload screenshot API to DOM-to-canvas with compression and fallback; minor key prop tweak; one formatting-only change.

Changes

Cohort / File(s)	Summary of Changes
Chat input screenshot targeting apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-input/index.tsx	Capture now targets frames.getMostRecentlyInteractedFrame(); single-frame capture with updated error handling; sets image context and toasts on success/failure.
Draft image preview portal apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/context-pills/draft-image-pill.tsx	Adds portal-based modal preview on pill click; measures pill position; SSR-safe portal; retains remove behavior; internal ref/state added.
Input pills key change apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/context-pills/input-context-pills.tsx	Changes key for image pills from content-based to index-based (image-${index}).
Chat context API apps/web/client/src/components/store/editor/chat/context.ts	Adds addImageContext(imageData); replaces existing IMAGE contexts with a new one using current time and image/jpeg.
Element interaction tracking apps/web/client/src/components/store/editor/element/index.ts	Logs errors with frameId; calls frames.updateLastInteraction on valid element clicks before overlay logic.
Frames manager interaction model apps/web/client/src/components/store/editor/frames/manager.ts	Adds lastInteractionTime to FrameData; integrates updateLastInteraction in select/navigation; preserves field on updates/register; adds getMostRecentlyInteractedFrame.
Screenshot manager formatting apps/web/client/src/components/store/editor/screenshot/index.tsx	Formatting-only (blank line); no functional changes.
Preload screenshot pipeline apps/web/preload/script/api/screenshot.ts	Replaces getDisplayMedia with DOM-to-canvas render plus compression; adds robust size-bounded compression, constants, and fallback canvas with error message JPEG.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  actor User
  participant ChatInput
  participant Frames as FramesManager
  participant FrameView
  participant ChatCtx as ChatContext
  participant UI as UI/Toast

  User->>ChatInput: Click "Screenshot"
  ChatInput->>Frames: getMostRecentlyInteractedFrame()
  alt Frame available
    ChatInput->>FrameView: captureScreenshot()
    alt Capture success with data
      ChatInput->>ChatCtx: addImageContext(base64 JPEG)
      ChatCtx-->>ChatInput: Context updated
      ChatInput->>UI: Show success toast
    else Failure or no data
      ChatInput->>UI: Show error toast
    end
  else No frame
    ChatInput->>UI: Show "No active frame" error
  end

sequenceDiagram
  autonumber
  actor User
  participant Element as ElementStore
  participant Frames as FramesManager

  User->>Element: Click in frame
  Element->>Frames: updateLastInteraction(frameId)
  note right of Frames: Sets lastInteractionTime = now
  Element-->>User: Proceed with overlay/select logic

  User->>Frames: select(frames)
  Frames->>Frames: updateLastInteraction(each frameId)

  User->>Frames: navigateToPath(frameId, path)
  Frames->>Frames: updateLastInteraction(frameId)

sequenceDiagram
  autonumber
  participant API as captureScreenshot
  participant Impl as captureScreenshotImpl
  participant Render as renderDomToCanvas
  participant Compress as compressImage

  API->>Impl: Invoke
  Impl->>Render: Render DOM to canvas
  Render-->>Impl: Canvas
  Impl->>Compress: Scale/quality iteration (<=5MB)
  alt Fits limit
    Compress-->>Impl: base64 JPEG
    Impl-->>API: base64 JPEG
  else Render/compress fails
    Impl-->>API: Fallback canvas JPEG with message
  end

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• onlook-dev/onlook#2665: Touches screenshot and frame-interaction paths, including most-recently-interacted frame and screenshot triggers.
• onlook-dev/onlook#2697: Modifies overlay/chat-related UI components used around chat interactions, overlapping UI surface.
• onlook-dev/onlook#2706: Also edits ChatInput.handleScreenshot with pre-checks; intersects with the new single-frame capture flow.

Suggested reviewers

• Kitenite

Poem

I tapped a frame—click, click—oh my!
A canvas bloomed beneath the sky.
JPEG whispers, small and bright,
Hopping through the chat by night.
One frame chosen, neat and keen—
A bunny bows to the screenshot scene. 🐇🖼️

[!TIP]

🔌 Remote MCP (Model Context Protocol) integration is now available!

Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats.

---

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

💡 Knowledge Base configuration:

• MCP integration is disabled by default for public repositories
• Jira integration is disabled by default for public repositories
• Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between f2256ec54e513882cb86a2c3860f8502dc35a6dd and 8b6a8298fbd1a018aed1fec9fe81399efceeb3a4.

📒 Files selected for processing (1)

• apps/web/client/src/components/store/editor/screenshot/index.tsx (1 hunks)

✅ Files skipped from review due to trivial changes (1)

• apps/web/client/src/components/store/editor/screenshot/index.tsx

✨ Finishing Touches

• [ ] 📝 Generate Docstrings

🧪 Generate unit tests

• [ ] Create PR with unit tests
• [ ] Post copyable unit tests in a comment

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

• Visit our Status Page to check the current availability of CodeRabbit.
• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[reddygtvs]

@Kitenite Hi Kiet! It's been a while since I touched this branch. Should I continue working on this by fixing the conflicts due to the more recent changes or leave it be?