Next Iteration Meeting (Friday, May 5, 2023 7:00pm UTC)

[AbdBarho]

Topics for the next meeting

[jjmachan]

First, have to say this is an awesome initiative to get the whole community involved kudos to you guys 🙌🏽 ❤️

I have two questions /topics

1. Plans to make OA sustainable in the long run?
2. What is the unique value we can provide to the wider community based on the type of people and experiences we have here? in short, what are our unique advantages that we can capitalize on to build something unique. what are we uniquely positioned to do?

Looking to catch you guys in the call :)

[RainRat]

Data cleaning website:

1. Fix code formatting, correct minor errors. Possible reranking by experts.
2. Tag every variant of "as a language model", identifying as a specific AI, or implementation detail ("cutoff date of"). Users can then choose to leave them out if they desire to give their chatbot its own personality.
3. Tag every example of refusing to do something a chatbot with a plugin can do. Users can leave them out so their plugin enabled chatbot has less chance of refusing task.

[echo0x22]

1. More context length (recent paper)
2. Multimodal (adding capability to get an information from images, audios, videos etc.)

[sedthh]

1. more info on plugins and how to contribute in this area
2. options for AutoGPT alternative
3. updates on safety and community guidelines --> are there any new decisions on firewalls, etc.?
4. requested functions from user feedback survey, which one to work on and when to expect them
5. plans for API
6. rethinking high score list to motivate non-whales

[notmd]

Discuss how we gonna do with auth service.

• Should we just implement a SSO or just a simple cookie-sharing solution (this should be easy since we are using only one domain)
• How we gonna migrate the existing data to the new service.

After this we can easily extends the UI to other platform if we want eg: desktop app, mobile app or browser extension. Also make people easier to deploy their own inference server.

Review inference report type

Here is how it looks currently

class ReportType(str, enum.Enum):
    spam = "spam"
    offensive = "offensive"
    feeback = "feedback"

While this is what ChatGPT does

This is harmful / unsafe
This isn't true
This isn't helpful

Data collection on boarding

A lot of new user don't know what they have to do, resulting a lot of bad message recently

• Add a interactive guide or make a short video.
• Add a simple test system to verify they know what they have to do.
• Limit task for new users, new user should only do simple task like classify, they need to reach X level before can do complicated tasks like writing prompt and response.

Collect feedback from chat

• Discuss how we can encourage user feedback with chat things like voting, ask users to rank when they regenerate a message.

[andreaskoepf]

High level organization proposal

• define the raodmap & main goals for v2 (tabula rasa, focus on the high-impact features)
• review all existing issues, decide set to be addressed in v2 & assign priorities (create new project board and milestones for v2)
• hire & onboard devs who want to help building the next iteration (all channels: discord, twitter, mails, youtube)
• expand leadership team (address time constrains of Yannic and Andreas)

Working areas

• model:
  • larger and better models (apply all OA tuning steps to LLaMA 65B), tune newly released open-source models ASAP
  • master the RM & RL steps (find ways to avoid losing capabilities, e.g. retain instruction following capabilities for plugins)
• plugins (provide: examples, documentation, video)
• retrieval (access wikipedia & custom document knowledge-bases or uploaded documents)
• sparse-distributed-models: use multiple models to answer requests (sparse-disributed, mixture of experts, cBTM, integrate results of other groups etc.)
• multi-modal: adding image as input, (maybe design general apporoach that can handle audio, video, 3d data)
• support non latin-cyrillic languages: help communities create open assistant versions in their language (e.g. Chinese)
• datasets:
  • gathering corpora with domain experts
  • store high quality general-knowledge indexed in vector-db and make i availale via retrieval (wikipedia, all papers etc.)

Possible Sketch of a v2 Vision

• run multiple iterations of RLHF re-finement, use best model to generate data, rank, review and correct results
• build a (fully automated) model factory: think continuous integration but for human-feedback tuned models
  • new datasets and preference data would be collected in parallel to training runs and automatically applied to next iterations
  • currently the OA training and evaluation process involves many manual steps and is somewhat brittle & and GPU training resources are sometimes not optimally utilized
  • a lot of groups are intersted in fine-tuning models on their proprietory data (going beyond prompt-engineering and zero shot), a fully open-source "model factory apporach" could make this more accessible
• transform the chat into a model-teacher UI:
  • add various feedback options:
    • survey about response quality and why a reply was good/bad, e.g. as follow-up on thumb up/down vote
    • generate multiple replies (e.g. 3) and let users rank
    • let users edit (correct) model resonses
    • collective conversations: show trees of other users for ranking or rating, especially to confirm edits & ranking
    • add report command (flag) to report inacceptable resonses
    • throttle responses for uses who don't provide feedback (cross validate feedback among peers to prevent cheating)
    • use OA inference to assist humand in writing assistant responses
  • evaluate model resonses in blind reviews (human + syntetic replies)
• focus on feedback collection and model improvements, try to deliver the best open-source conversational models
  • potentially form a sub-group which builds a network of OA inference nodes, possibly including payment options (people don't like it but someone needs to pay for the quite enourmouse hardware and energy costs and IMO it is reasonable to expose the costs for ecological reasons)

Inference System

• decouple the OA inference system from human demonstration collection and our website, create alternative to kobolt AI and other solutions
• offer it as stand-alone open-source customizable inference product (logos etc.)
• make it easily deployable in colab, at home, on private servers, single server or with distributed workers so that everyone can quickly deploy a chat interface for a our models model

Backend

• let users decide on which prompts they want to work on
• implement most wanted features: enter prompt-response pairs, edit messages (e.g. correct typos)
• potentially switch to a message-board search-based format (more like stack-overflow)

General thoughts

• OA has (very) limited resources, I recommend to focus on a high-impact roadmap, commit to few goals and prioritize tasks accodringly (we have hundreds of feature wishes - but we don't have developers to implement them)

[echo0x22]

3. Add hints for prompts (like in Bing)

4. Add ability to save custom presets and name it.

1. More context length (recent paper)
2. Multimodal (adding capability to get an information from images, audios, videos etc.)

[MattAlexMiracle]

Utilize feedback from deployed assistant (needs additional considerations wrt filterting etc...)

[sedthh]

+1 usage statistics: how the number of active users / contributors changes since the release

[theblackcat102]

Expand language support to include non latin languages ( chinese, japanese, korean, thai )

Reasons:

• our llama 13B trained on oasst with the aforementioned languages performs quite well

• the chinese discord channel has been sharing how to "hack" our 30B sft model to response in chinese, so basically even without training llama model has been able to perform pretty well

• this would increase positive exposure in Asia pacific, thereby increase the flywheel size

[elijah-kulpinski]

The future steps towards becoming an assistant rather than just a chatGPT replacement (plugins in the near future but what else, what's in the FAR future, etc).

[echo0x22]

• Add Discord OpenAssistant bot

[theblackcat102]

Open Assistant Meetup notes - 06/05/2023

Here's a list of points that we covered during the meeting:

• multi-modalities are mentioned multiple time

  • storage might be an issue

• we also need a RLHF on the results generated by human, not on human generated response

  • mainly the issue is oasst lack of bad example

• what if I want a open assistant for gardening, finance

  • a lot of people asking if we could use in our
  • expert LLM for a specific domain

• model size

  • model parameters is still a major factor in perf

• extensions

  • yannic will update the new tool model to hf API tomorrow
  • being able to use tool is very important
  • context length is still a problem, 8k is needed for it to work
  • dragan show case a vercel website deployment on the same prompts used by openai and OA models
    • performance are similar to gpt-3.5-turbo

• retrieval

  • able to simply use the embedding a fed directly into the LLM
  • basically it would be faster than placing it in the prompt

• meta prompt editing

  • andreas propose we need a prompt editing

• discord bot integration

  • with thumbs up and down

• API for open source development

  • yannic is working on the API with authentication

• sustainability

  • will be very ugly once we have to setup entities for this

• ask people to donate compute power in a decentralized manner, and reward would just be prioritized in queue for chat

[smeyerhot]

ask people to donate compute power in a decentralized manner, and reward would just be prioritized in queue for chat

https://github.com/bigscience-workshop/petals