Feature/chat image ask mode
Description
This PR introduces two major enhancements to the Onlook chat interface:
- Ask Mode: Adds an "Ask" mode toggle to the chat, allowing users to ask questions about their project without triggering build/code changes. The mode uses a dedicated system prompt focused on explanations and guidance, not code generation.
- Image & Screenshot Menu: Adds a dropdown menu next to the chat submit button, enabling users to either upload an image or add a screenshot of the selected frame. Screenshots are labeled and displayed as image pills above the prompt.
These features improve the chat experience by making it more flexible, educational, and visually interactive.
Related Issues
- Fixes [#2196]
- Fixes [#2197]
Type of Change
- [x] New feature
- [x] Refactor
- [ ] Bug fix
- [ ] Documentation update
- [ ] Release
- [ ] Other (please describe):
Testing
- Manually tested toggling between "Build" and "Ask" modes; verified correct system prompt and UI changes.
- Uploaded images and added screenshots via the new dropdown; confirmed they appear as pills and are sent with chat context.
- Checked accessibility (keyboard navigation, ARIA labels) for the dropdown and toggle.
- Verified that chat behavior and context are preserved across mode and image changes.
Screenshots (if applicable)
https://github.com/user-attachments/assets/255cb399-7eea-4141-81d2-686d5ceac5f2
Additional Notes
- The "Ask" mode prompt is designed to act as a consultant, providing explanations and advice rather than direct code edits.
- The dropdown menu for images and screenshots streamlines the chat input UI and improves user workflow.
- All changes are backward compatible and do not affect existing chat or build functionality.
[!IMPORTANT] Adds 'Ask' mode and image/screenshot menu to chat, enhancing user interaction and guidance.
- Behavior:
- Adds 'Ask' mode to chat, allowing questions without code changes, using
getAskModeSystemPrompt()inroute.ts.- Adds image/screenshot menu in
action-buttons.tsx, allowing uploads and screenshots.- UI Components:
ChatModeToggleinchat-mode-toggle.tsxfor switching between 'Build' and 'Ask' modes.ActionButtonsinaction-buttons.tsxfor image/screenshot actions.- Chat Management:
getAskMessages()inchat/index.tsfor handling 'Ask' mode messages.- Updates to
ChatManagerto support new mode and context handling.- Prompts:
- New
ASK_MODE_SYSTEM_PROMPTinask.tsfor 'Ask' mode guidance.getAskModeSystemPrompt()inprovider.tsto retrieve the prompt.This description was created by
for efc80034a4e92ebea560b1da1143ae2caf874e7e. You can customize this summary. It will automatically update as commits are pushed.
@Rish-it is attempting to deploy a commit to the Onlook Team on Vercel.
A member of the Team first needs to authorize it.
@Kitenite hey! Could you drop your review on this #2306 when you get a chance? Take a quick look especially with Anthropic and verify end-to-end ask mode flow. Let me know if anything needs fixing or bugs!