Indexing breaks at 20%-70%

[napa3um]

Before submitting your bug report

• [X] I believe this is a bug. I'll try to join the Continue Discord for questions
• [X] I'm not able to find an open issue that reports the same bug
• [X] I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS: macOS 14.4.1 (M2)
- Continue: 0.8.24, 0.9.115 (pre-release) 
- IDE: VSCode 1.88.1 (Universal)
- Ollama 0.1.32, nomic-embed-text:latest, hellord/e5-mistral-7b-instruct:Q4_0

Description

With Ollama embeddings indexing not working, break at 20%-30%, silence, without errors popup.

In DevTools with nomic-embed-text:latest:

console.ts:137 [Extension Host] Error refreshing index:  Error: Invalid argument error: Values length 2304 is less than the length (768) multiplied by the value size (768) for FixedSizeList(Field { name: "item", data_type: Float32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, 768)
	at LocalTable.add (/Users/napa3um/.vscode/extensions/continue.continue-0.9.115-darwin-arm64/out/extension.js:89724:25)
	at async addComputedLanceDbRows (/Users/napa3um/.vscode/extensions/continue.continue-0.9.115-darwin-arm64/out/extension.js:90024:15)
	at async _LanceDbIndex.update (/Users/napa3um/.vscode/extensions/continue.continue-0.9.115-darwin-arm64/out/extension.js:90097:13)
	at async CodebaseIndexer.refresh (/Users/napa3um/.vscode/extensions/continue.continue-0.9.115-darwin-arm64/out/extension.js:188516:47)
	at async _VsCodeExtension.refreshCodebaseIndex (/Users/napa3um/.vscode/extensions/continue.continue-0.9.115-darwin-arm64/out/extension.js:3316

with hellord/e5-mistral-7b-instruct:Q4_0:

console.ts:137 [Extension Host] Error refreshing index:  Error: Invalid argument error: Values length 12288 is less than the length (4096) multiplied by the value size (4096) for FixedSizeList(Field { name: "item", data_type: Float32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, 4096)

with transformers.js:

console.ts:137 [Extension Host] Error refreshing index:  TypeError: Cannot read properties of undefined (reading 'apply')
	at e2.<computed> (/Users/napa3um/.vscode/extensions/continue.continue-0.9.115-darwin-arm64/out/extension.js:54601:63)
	at wasm://wasm/0055765a:wasm-function[15]:0x29b5
	at wasm://wasm/000b54aa:wasm-function[234]:0x25635
	at _Parser.parse (/Users/napa3um/.vscode/extensions/continue.continue-0.9.115-darwin-arm64/out/extension.js:55274:29)
	at CodeSnippetsCodebaseIndex.getSnippetsInFile (/Users/napa3um/.vscode/extensions/continue.continue-0.9.115-darwin-arm64/out/extension.js:56504:28)
	at async CodeSnippetsCodebaseIndex.update (/Users/napa3um/.vscode/extensions/continue.continue-0.9.115-darwin-arm64/out/extension.js:56525:28)
	at async CodebaseIndexer.refresh (/Users/napa3um/.vscode/extensions/continue.continue-0.9.115-darwin-arm64/out/extension.js:188516:47)
	at async _VsCodeExtension.refreshCodebaseIndex (/Users/napa3um/.vscode/extensions/continue.continue-0.9.115-darwin-arm64/out/extension.js:331684:26)

Restart VSCode, Ollama, remove ~/.continue/index, - don't resolving problem.

To reproduce

No response

Log output

No response

[napa3um]

error again (check indexing on this repository - github.com/continuedev/continue) :(

console.ts:137 [Extension Host] Error refreshing index:  RuntimeError: Aborted(). Build with -sASSERTIONS for more info.
	at abort (/Users/napa3um/.vscode/extensions/continue.continue-0.8.24-darwin-arm64/out/extension.js:29833:172)
	at _abort (/Users/napa3um/.vscode/extensions/continue.continue-0.8.24-darwin-arm64/out/extension.js:30316:15)
	at wasm://wasm/000b54aa:wasm-function[135]:0x1c36c
	at wasm://wasm/000b54aa:wasm-function[128]:0x1a029
	at Language.query (/Users/napa3um/.vscode/extensions/continue.continue-0.8.24-darwin-arm64/out/extension.js:31237:29)
	at CodeSnippetsCodebaseIndex.getSnippetsInFile (/Users/napa3um/.vscode/extensions/continue.continue-0.8.24-darwin-arm64/out/extension.js:33075:29)
	at async CodeSnippetsCodebaseIndex.update (/Users/napa3um/.vscode/extensions/continue.continue-0.8.24-darwin-arm64/out/extension.js:33095:28)
	at async CodebaseIndexer.refresh (/Users/napa3um/.vscode/extensions/continue.continue-0.8.24-darwin-arm64/out/extension.js:188988:47)
	at async _VsCodeExtension.refreshCodebaseIndex (/Users/napa3um/.vscode/extensions/continue.continue-0.8.24-darwin-arm64/out/extension.js:331503:26)

[l7-ehumes]

I've also run into this error. It nearly instantly errors again if one forces the index refresh:

console.ts:137 [Extension Host] Error refreshing index:  TypeError: Cannot read properties of undefined (reading 'apply')
	at e2.<computed> (/Users/erichumes/.vscode/extensions/continue.continue-0.8.25-darwin-arm64/out/extension.js:54616:63)
	at wasm://wasm/0055765a:wasm-function[15]:0x29b5
	at wasm://wasm/000b54aa:wasm-function[234]:0x25635
	at _Parser.parse (/Users/erichumes/.vscode/extensions/continue.continue-0.8.25-darwin-arm64/out/extension.js:55289:29)
	at CodeSnippetsCodebaseIndex.getSnippetsInFile (/Users/erichumes/.vscode/extensions/continue.continue-0.8.25-darwin-arm64/out/extension.js:56519:28)
	at async CodeSnippetsCodebaseIndex.update (/Users/erichumes/.vscode/extensions/continue.continue-0.8.25-darwin-arm64/out/extension.js:56540:28)
	at async CodebaseIndexer.refresh (/Users/erichumes/.vscode/extensions/continue.continue-0.8.25-darwin-arm64/out/extension.js:188535:47)
	at async _VsCodeExtension.refreshCodebaseIndex (/Users/erichumes/.vscode/extensions/continue.continue-0.8.25-darwin-arm64/out/extension.js:331710:26)

I also see another error for a different repo in one of the first few files:

console.ts:137 [Extension Host] Error refreshing index:  Error: Invalid argument error: Values length 0 is less than the length (4096) multiplied by the value size (4096) for FixedSizeList(Field { name: "item", data_type: Float32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, 4096)
	at LocalTable.add (/Users/erichumes/.vscode/extensions/continue.continue-0.8.25-darwin-arm64/out/extension.js:89739:25)
	at async addComputedLanceDbRows (/Users/erichumes/.vscode/extensions/continue.continue-0.8.25-darwin-arm64/out/extension.js:90033:15)
	at async _LanceDbIndex.update (/Users/erichumes/.vscode/extensions/continue.continue-0.8.25-darwin-arm64/out/extension.js:90112:13)
	at async CodebaseIndexer.refresh (/Users/erichumes/.vscode/extensions/continue.continue-0.8.25-darwin-arm64/out/extension.js:188535:47)
	at async _VsCodeExtension.refreshCodebaseIndex (/Users/erichumes/.vscode/extensions/continue.continue-0.8.25-darwin-arm64/out/extension.js:331710:26)

Relevant config:

"embeddingsProvider": {
    "title": "Remote codellama embeddings provider",
    "provider": "ollama",
    "model": "codellama:7b",
    "apiBase": "http://172.16.0.14:11434"
  },

Info:

- OS: macOS 14.4.1 (M1)
- Continue: 0.8.25
- IDE: Version: 1.89.0 (Universal)
- Ollama 0.1.33, codellama:7b, codellama:7b-instruct

[napa3um]

console log on https://github.com/FlowiseAI/Flowise/ repo: vscode-app-1715407252162.log

[l7-ehumes]

So, I found at least one instance where the:

Error: Invalid argument error: Values length 0 is less than the length

style error occurs. On my system it was due to a large license file 903 characters long, but only one line long. If I format the license with a JSON formatter so that the file is broken into a few different lines the indexing process works after.

And indeed, I tested removing a bunch of the content of the license, but keeping it to one line, and I found that the indexing also functioned.

[dvcolgan]

If anyone is still running into this issue, I solved it by setting up a .continueignore file in the root of my repository. It was indeed the very long lines that the indexer was breaking on. Something like this:

.vscode/
.idea/
__pycache__
node_modules

Thanks to the devs for a fantastic plugin.

[napa3um]

Thanks to the devs :)