Seems to forget about earlier documents

[cs96and]

If I upload a few documents, then it seems to forget about ones that I uploaded earlier. Is there a limit to the number of documents or tokens it will store per user?

[eabjab]

Having the same issue. It seems to only be able to reference the most recently uploaded document in my testing, even when running locally using my own pinecone index.

[AitoD]

Having the same issue, only the latest uploaded documents are referenced. It seems to overwrite the index whenever you upload a new file.

[Ninn0x4F]

Same issue here as well, I suspect the issue might be related to Pinecone itself, uploaded multiple files and the Vectors stopped increasing around 4k afterwards uploading any new documents will only reference the last 2 documents for me currently.

I used 1536 for Dimensions and left the rest default. I uploaded my first document (XML File with multiple Item IDs) and asked it for the first and last ItemID in the document, it got it nearly right. Then I continued to upload more files with same structure, asked periodically to reference first and last ItemID and it started to behave as if it only saw the last 2 documents uploaded.

[lonelycode]

The issue is in how the upload code assigns IDs here:

    for i, embedding := range embeddings {
		chunk := chunks[i]
		vectors[i] = PineconeVector{
			ID:     fmt.Sprintf("id-%d", i),
			Values: embedding,
			Metadata: map[string]string{
				"file_name": chunk.Title,
				"start":     strconv.Itoa(chunk.Start),
				"end":       strconv.Itoa(chunk.End),
				"title":     chunk.Title,
				"text":      chunk.Text,
			},
		}
	}

The ID of the vector is just a counter for the chunk in the single file, in a multi-file upload, the ID's will overlap (ID 001 will repeat for file 1 and file 2 etc.). The UPSERT operation will update or insert based on the ID, essentially overwriting vectors for pre-processed files.

Switching this to an ID that is a hash of the filename plus the chunk number would ensure uniqueness.

For example:

func HashFileName(filename string) string {
	hash := sha256.Sum256([]byte(filename))
	return hex.EncodeToString(hash[:])
}

//...

func (p *Pinecone) UploadEmbeddings(embeddings [][]float32, chunks []Chunk) error {
	// Prepare URL
	url := p.APIEndpoint + "/vectors/upsert"

	// Prepare the vectors
	vectors := make([]PineconeVector, len(embeddings))
	for i, embedding := range embeddings {
		vectorID := fmt.Sprintf("id-%s-%d", HashFileName(chunks[i].Title), i)
		vectors[i] = PineconeVector{
			ID:     vectorID,
			Values: embedding,
// ...