Create a Share-OpenDevin for people to share their interaction trajectories

[xingyaoww]

What problem or use case are you trying to solve?

Similar to ShareGPT where people can share their interaction with ChatGPT, we can create "ShareOpenDevin", where users can contribute their interaction trajectories with OpenDevin.

Describe the UX of the solution you'd like

We need:

• [x] A "Share" button on the front-end UI
  • [x] When clicked, the backend controller can send over the current State (which already contains all the history interactions) to the remote ShareOpenDevin API (we probably need a better way to figure out how to manage each session - maybe with uuid - so that same session at different steps won't get submitted multiple times)
  • [x] for reproducibility, maybe we can also send over metadata, like OpenDevin version, agent name, model name, etc.
  • [ ] (low-priority) Figure out a way for users to authenticate themselves (e.g., via GitHub login) before Share, so it is easy to track who shared what.
• [x] "ShareOpenDevin" server -- maybe we want to put this in a separate repo
  • [x] A functioning website that hosted an API endpoint /share receiving people's share requests.
  • [ ] Render the user-submitted trajectories, similar to what we did for evaluation now. We can improve it when https://github.com/OpenDevin/OpenDevin/issues/1743 is implemented.
  • [ ] (low-priority) Allow users to log in to the website via GitHub and manage (remove) their submitted trajectories.

Do you have thoughts on the technical implementation?

• ShareGPT's code is actually open-source with MIT license which can provide a good starting point.

Describe alternatives you've considered

Additional context

[JustinLin610]

This is really a terrific idea! I think we should collect more data for fine-tuning to make things work better.

[Jiayi-Pan]

happy to take care of the logging stuff (state, action, agent input/output, meta data, ...), will try to finish this in a few days. plan to do more after nips ddl :)

[frankxu2004]

Awesome idea! I wonder if it would be nice if we can let the users interact with the environment as well, e.g. terminal, editor, etc. so that we can not only record user chat history and agent actions, but also user correction actions that may sometime happen when dealing with dumb agents.

Might be a bit far fetched for now, but I think recording user demonstrations when the agent is not acting correctly are super useful as well.

[rbren]

My goal for EventStream is that it's a serializable representation of an entire session/workflow. I'd suggest leaning on that.

I'm currently working through persistent sessions, which is pretty similar technologically here

[neubig]

OK! I figured out how to set up a place on AWS where we can dump trajectories to share. Here's an example script of how it can be done. We can use this to add a button that allows users to share trajectories.

import requests

url = "https://kttkfkoju5.execute-api.us-east-2.amazonaws.com/od-share-trajectory"

data = {
    "email": "[email protected]",
    "token": "exampletoken",
    "trajectory": [{"x": 1, "y": 2, "z": 3}, {"x": 4, "y": 5, "z": 6}],
    "feedback": "positive"
}
response = requests.post(url, json=data)
print(response.status_code)
print(response.json())

We can refine this more later as we know more about our requirements.

[neubig]

This is mostly implemented through #2020, but there are a few rough edges #2033 and #2032

[neubig]

OK, I have made two pieces of progress towards the final TODO in this, which is a place where users can view and share their trajectories:

1. The backend server now returns a feedback_id and password. Right now the password isn't used for anything, but in the future it can be used for things like deleting feedback that you don't want to have shared.
2. I made an endpoint that can be used to retrieve feedback. We can use this as a backend for a javascript frontend that displays the feedback for instance. Below is an example of using the backend to retrieve a submitted piece of feedback:

curl -X POST https://show-od-trajectory-3u9bw9tx.uc.gateway.dev/show-od-trajectory      -H "Content-Type: application/json"      -d '{"feedback_id": "9b91f7052be2e34b59e003aacaf2a0ad043fea3666b223981ea8dd67b7ef5fca"}'

@amanape, @xingyaoww, or any others interested in visualization, would you like to make some frontend code to visualize these submitted feedback items?

[neubig]

I'm working on this.