VC: Smart Embeddings

[bobbykolev]

Description

We need smarter ingestion support for files and callouts to improve the quality of the results.

Must have scope

• [ ] Preserve the context when splitting chunks.
• [ ] Gather more context when reading chunks.

Improved handling of formats:

• [ ] Improve Excel ingestion - extract in separate epic

Architectural roadmap epic:

• [ ] Use elastic search ingestion instead of VectorDB?

Investigate multiple embeddings. Context increase, tokens limit?

Additional context

About both better ingesting and how the chunks are returned