Teach ChatCraft how to get more context for certain URLs

[humphd]

Let's say that I'm wanting to discuss a pice of code with ChatCraft. Further, let's say that this code is available on GitHub. Maybe this file:

https://github.com/tarasglek/chatcraft.org/blob/main/src/lib/summarize.ts

I can easily grab that file using the GitHub REST API in the browser like so:

// Regex to extract parts from URL TODO...
const owner = 'tarasglek';
const repo = 'chatcraft.org';
const path = 'src/lib/summarize.ts';

fetch(`https://api.github.com/repos/${owner}/${repo}/contents/${path}`)
  .then(response => response.json())
  .then(data => {
    // File contents are base64 encoded in data.content
    const fileContent = Buffer.from(data.content, 'base64').toString('utf8');
    console.log(fileContent);
  })
  .catch(error => console.error(error));

Now I have the raw text content of that file, and can use it as part of my chat.

I wonder if we should automate this, such that ChatCraft can do this automatically. Maybe I use a special syntax, like @github:https://github.com/tarasglek/chatcraft.org/blob/main/src/lib/summarize.ts or @github:tarasglek/chatcraft.org/blob/main/src/lib/summarize.ts etc and when I include that in a prompt, ChatCraft automatically downloads the file and adds it to the context of the chat.

There are probably a bunch of places where this would be useful:

1. PRs
2. Issues
3. Files (like above)
4. Gists

We could also do it automatically (maybe pref it?) so you don't need to prefix these URLs, but that might also lead to unexpected behaviour.

Which other URLs/domains would be useful to do this for?

You could also do it for @url:... and write a Pages Function to download the content and get the content (HTML) or text content.

Thoughts?

[tarasglek]

I'm not usually a fan of copilot, but I think they are onto something with /commands, see their latest release notes and create workspace demo https://code.visualstudio.com/updates/v1_81#_github-copilot

Maybe we should have a feature for /commands for fav functions. It's very IRC, very chat-ui

ps: I think we should copy their little demo and release notes style :)

[humphd]

Can you say more about what you mean? Are you suggesting we add something like /github ...?

[humphd]

You've got me thinking about all kinds of things now, this idea is brilliant! I wouldn't have thought to do it, but it's so obvious now that you describe it.

Where functions are custom JS functions called by LLMs, commands would be called by user's via prompts. Here are some I can already imagine:

• /new creates a new chat (like going to chatcraft.org/c/new)
• /clear clears all messages in the current chat
• /summary or /summarize or /summary 500 creates a summary of the current chat, optionally taking a number of words (e.g., 500)
• /shorten same idea as /summary, but deletes all messages and leaves you only with the summary
• /github <github-url-or-part-thereof> calls the GitHub REST API to get raw text content for a given URL or part of a GitHub URL (gist, issue, pr, code)
• /get like /github but for regular URLs. Could also do /json to download the result as json
• /share shares current chat and gives you the share URL
• /help or /help <cmd> gives help

etc. What other ones can you think of?

[tarasglek]

transcribe is one i want badly for youtube

there are commands and also bots like @fn:ssh to run commands or maybe invoke rest apis

/summarize @youtube http-youtube-link

would be pretty fly.

[tarasglek]

/translate would be another cool command for ESL users to replace whats entered with english

[humphd]

Question: how does one transcribe a YouTube video? Is that not a very long function call?

For /translate, what's the context? Some URL or something in the chat?

[tarasglek]

I don't have concrete ideas for these :)

translate should probably translate lass message to some preferred lang if run without prompt, eg /translate -lang ua

/translate <type sentence in ukrainian> would probably just translate the prompt

Re transcribe, yes, would be some rest api call probably to something that can pull down youtube, transcribe it and return text

key here would be composability with summarize

[humphd]

I've never seen composable slash commands in any app I've used before. I'm not sure you you'd do it, tbh. Have you seen this?

I feel like the Chat and its messages give you the context for composition (i.e., /translate dumps text into the chat, /summary acts on it, etc).

[tarasglek]

Your simpler suggestion makes sense.

[humphd]

/translate could be made to work in browser via a worker running an ONNX model using https://huggingface.co/spaces/Xenova/react-translator (code is https://github.com/xenova/transformers.js/tree/main/examples/react-translator).

As with a Speech-to-Text model, you need to download and cache it first. So maybe we'd need a way to do that in the Settings dialog, so users can opt into waiting on a big download.