onlook icon indicating copy to clipboard operation
onlook copied to clipboard

Feature/chat image ask mode

Open Rish-it opened this issue 6 months ago • 2 comments

Description

This PR introduces two major enhancements to the Onlook chat interface:

  • Ask Mode: Adds an "Ask" mode toggle to the chat, allowing users to ask questions about their project without triggering build/code changes. The mode uses a dedicated system prompt focused on explanations and guidance, not code generation.
  • Image & Screenshot Menu: Adds a dropdown menu next to the chat submit button, enabling users to either upload an image or add a screenshot of the selected frame. Screenshots are labeled and displayed as image pills above the prompt.

These features improve the chat experience by making it more flexible, educational, and visually interactive.


Related Issues

  • Fixes [#2196]
  • Fixes [#2197]

Type of Change

  • [x] New feature
  • [x] Refactor
  • [ ] Bug fix
  • [ ] Documentation update
  • [ ] Release
  • [ ] Other (please describe):

Testing

  • Manually tested toggling between "Build" and "Ask" modes; verified correct system prompt and UI changes.
  • Uploaded images and added screenshots via the new dropdown; confirmed they appear as pills and are sent with chat context.
  • Checked accessibility (keyboard navigation, ARIA labels) for the dropdown and toggle.
  • Verified that chat behavior and context are preserved across mode and image changes.

Screenshots (if applicable)

https://github.com/user-attachments/assets/255cb399-7eea-4141-81d2-686d5ceac5f2


Additional Notes

  • The "Ask" mode prompt is designed to act as a consultant, providing explanations and advice rather than direct code edits.
  • The dropdown menu for images and screenshots streamlines the chat input UI and improves user workflow.
  • All changes are backward compatible and do not affect existing chat or build functionality.

[!IMPORTANT] Adds 'Ask' mode and image/screenshot menu to chat, enhancing user interaction and guidance.

  • Behavior:
    • Adds 'Ask' mode to chat, allowing questions without code changes, using getAskModeSystemPrompt() in route.ts.
    • Adds image/screenshot menu in action-buttons.tsx, allowing uploads and screenshots.
  • UI Components:
    • ChatModeToggle in chat-mode-toggle.tsx for switching between 'Build' and 'Ask' modes.
    • ActionButtons in action-buttons.tsx for image/screenshot actions.
  • Chat Management:
    • getAskMessages() in chat/index.ts for handling 'Ask' mode messages.
    • Updates to ChatManager to support new mode and context handling.
  • Prompts:
    • New ASK_MODE_SYSTEM_PROMPT in ask.ts for 'Ask' mode guidance.
    • getAskModeSystemPrompt() in provider.ts to retrieve the prompt.

This description was created by Ellipsis for efc80034a4e92ebea560b1da1143ae2caf874e7e. You can customize this summary. It will automatically update as commits are pushed.

Rish-it avatar Jun 30 '25 04:06 Rish-it

@Rish-it is attempting to deploy a commit to the Onlook Team on Vercel.

A member of the Team first needs to authorize it.

vercel[bot] avatar Jun 30 '25 04:06 vercel[bot]

@Kitenite hey! Could you drop your review on this #2306 when you get a chance? Take a quick look especially with Anthropic and verify end-to-end ask mode flow. Let me know if anything needs fixing or bugs!

Rish-it avatar Jun 30 '25 04:06 Rish-it