dwata
dwata copied to clipboard
Generate embeddings for text sources
As a user I would like to generate embedding for my text sources. Text from Markdown files will have some structure which may be useful internally.
Engineering notes
We should use APIs from OpenAI, Anthropic at the minimum. SentenceTransformers would be a great addition, either running locally or on a hosted server (can be done later).
This will depend on #106 if we want to implement support for SentenceTransformers.
#104 will deal with storage. Perhaps we will work on these two issues together.