mediar-ai
[feature] Make screenpipe able to use the computer
describe the feature Same as computer use from https://www.anthropic.com/news/3-5-models-and-computer-use
why is this needed? Why would computer users do things on computers if they can just ask the AI to do it?
alternatives considered Integrate with the lower level APIs of every possible app. This is a much harder problem.
![linear[bot] avatar](https://avatars.githubusercontent.com/in/20150?v=4 "Published by a pub.dev verified publisher"){kind=small}
haha yeah
i was thinking in creating API so people can use JS API to interact with mouse, keyboard, etc. in our plugin system
then later we can just generate automation of their work day
i think this is still early, openinterpreter and adept are on this for a while and there is still no real use case, nobody use this
for screenpipe we're more focused on building solid data infrastructure and let people experiment with this, but I think LLM are still a bit early for something really reliable and actually useful, in addition to the fact that an AI taking control over my computer will actually reduce my bandwidth, but maybe at night
Yeah agreed it's early, I think it becomes real in 6 months - one year
I think it might make more sense in a VM, so it automates boring things without taking the user time. That or truly be in sync with the user intent and work together but that's harder
Yeah agreed it's early, I think it becomes real in 6 months - one year
I think it might make more sense in a VM, so it automates boring things without taking the user time. That or truly be in sync with the user intent and work together but that's harder
hmm i like the idea of the VM
the problem of tools like Zapier, gumloop, etc.
- cost 10x more because it runs on the server
- needs very sensitive access like gmail etc. needs tons of tokens and things like this that are there on the client
- basically the vm can copy paste the tokens easily
- no interruption
i can imagine that you do some work and then there is 100 AI VMs doing other things for you on the internet based on your activity
Yeah I agree it's not easy to solve the security topic here.
you do some work and then there is 100 AI VMs doing other things for you on the internet based on your activity
Yeah I think that would be great. Given full context of one person work, I think it should be possible even for current LLM to follow up on some of the most trivial but time intensive tasks independently. For example doing all kind of redtape filling for the user, checking the state of various system of interest (PRs/issue solved, release of dependencies, state of projects of interest...), testing out projects of competition and aggregating the info...
On Sat, Nov 2, 2024, 00:12 Louis Beaumont @.***> wrote:
Yeah agreed it's early, I think it becomes real in 6 months - one year
I think it might make more sense in a VM, so it automates boring things without taking the user time. That or truly be in sync with the user intent and work together but that's harder
hmm i like the idea of the VM
the problem of tools like Zapier, gumloop, etc.
- cost 10x more because it runs on the server
- needs very sensitive access like gmail etc. needs tons of tokens and things like this that are there on the client
- basically the vm can copy paste the tokens easily
- no interruption
i can imagine that you do some work and then there is 100 AI VMs doing other things for you on the internet based on your activity
— Reply to this email directly, view it on GitHub https://github.com/mediar-ai/screenpipe/issues/567#issuecomment-2452705280, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAR437QUZHRZX7OQ6BLFSVLZ6QDHNAVCNFSM6AAAAABQNRGM22VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINJSG4YDKMRYGA . You are receiving this because you authored the thread.Message ID: @.***>