obsidian-smart-connections icon indicating copy to clipboard operation
obsidian-smart-connections copied to clipboard

Stuck on Error: The text contains a special token that is not allowed: <|endoftext|>

Open seanrose opened this issue 11 months ago • 6 comments

I get prompted to start embedding but it gets stuck at some point (0 tokens/sec) and I see this error in the JS console:

plugin:smart-connections:1229 Uncaught (in promise) Error: The text contains a special token that is not allowed: <|endoftext|>
    at _Tiktoken.encode (plugin:smart-connections:1229:19)
    at SmartEmbedOpenAIAdapter.count_tokens (plugin:smart-connections:1393:31)
    at eval (plugin:smart-connections:1418:36)
    at Array.map (<anonymous>)
    at SmartEmbedOpenAIAdapter.embed_batch (plugin:smart-connections:1417:36)
    at SmartNotes.ensure_embeddings (plugin:smart-connections:2121:47)
    at async SmartNotes.ensure_embeddings (plugin:smart-connections:2262:22)

Anything more I can do to help debug? It gets stuck here every time I restart/reinstall/retry the embeddings from scratch.

seanrose avatar Mar 04 '24 05:03 seanrose