dwata icon indicating copy to clipboard operation
dwata copied to clipboard

Generate embeddings for text sources

Open brainless opened this issue 1 year ago • 2 comments

As a user I would like to generate embedding for my text sources. Text from Markdown files will have some structure which may be useful internally.

Engineering notes

We should use APIs from OpenAI, Anthropic at the minimum. SentenceTransformers would be a great addition, either running locally or on a hosted server (can be done later).

brainless avatar May 04 '24 03:05 brainless

This will depend on #106 if we want to implement support for SentenceTransformers.

brainless avatar May 04 '24 03:05 brainless

#104 will deal with storage. Perhaps we will work on these two issues together.

brainless avatar May 04 '24 03:05 brainless