omi icon indicating copy to clipboard operation
omi copied to clipboard

Solve "Remember to order milk" issue

Open smian1 opened this issue 1 year ago β€’ 10 comments

This simple phrase is very powerful and important. This should never be discarded. How do we solve for this use case?

image

smian1 avatar Oct 31 '24 06:10 smian1

I’ve set up Zapier to use AI to analyze my speech in order to create and send to do items to my to-do app. I would love to be able to just speak something out loud and have the rest taken care of.

maxfahl avatar Oct 31 '24 07:10 maxfahl

Originally this device was intended to record dialogs, that's why it it's discarding short stuff like commands. The v2 has a button and a speaker, so it might work better for this type of flow, press a button, tell command, listen for response.

e3e6 avatar Oct 31 '24 09:10 e3e6

Originally this device was intended to record dialogs, that's why it it's discarding short stuff like commands.

The v2 has a button and a speaker, so it might work better for this type of flow, press a button, tell command, listen for response.

I think this is the way to Go, because the user experience is just better. Having the action button be the trigger for the short reminders gives users immediate feedback on whether their command was saved or their reminder was saved. That is a much better experience than just saying a keyword out loud and hoping that your system caught the keyword.

andresgomezsar avatar Nov 01 '24 03:11 andresgomezsar

I'm quite new to the device. Is there something I need to do in order to make this work?

maxfahl avatar Nov 03 '24 21:11 maxfahl

Does the memory's Action Items / Events work for you @maxfahl

beastoin avatar Nov 12 '24 04:11 beastoin

Im gonna create a webhook endpoint for raw audio data and transcribe it myself, since the built-in modems suck for Swedish. Then I will use OpenAI API to let the model decide on actions to call. One of them will be executing Apple Script to add to-do items to Things 3.

That's the plan anyway

maxfahl avatar Nov 15 '24 06:11 maxfahl

Still an issue. Need better prompting to handle tasks. This should not be discarded.

image

smian1 avatar Jan 06 '25 08:01 smian1

@beastoin

smian1 avatar Jan 06 '25 08:01 smian1

Hey πŸ‘‹

According to the current prompt logic, the memory gets discarded based on this condition:
πŸ”— [Prompt in code]

 You will be given a conversation transcript, and your task is to determine if the conversation is worth storing as a memory or not.
    It is not worth storing if there are no interesting topics, facts, or information, in that case, output discard = True.

🧠 Logic:

  • If ≀ 100 words β†’ ask AI to evaluate content quality
  • If AI says no interesting content β†’ ❌ discard
  • If AI says there is interesting content β†’ βœ… keep

That said, the prompt itself could definitely be improved β€” someone could iterate on it via the linked logic

Personally, I think we should move these hardcoded prompts into a config/settings file, so users/devs can tweak them as needed. Otherwise, we’ll keep running into corner cases β€” and without tests, we won’t catch them all.

skywinder avatar Apr 07 '25 14:04 skywinder

βœ… Suggested next steps:

  • Move all prompts to a separate file
  • Later: add tools for performance testing or let users adjust prompts directly
  • This also partially addresses: [#1825](https://github.com/BasedHardware/omi/issues/1825)

Let me know what you think and which direction we should take! πŸ’‘

skywinder avatar Apr 07 '25 14:04 skywinder