obsidian-copilot icon indicating copy to clipboard operation
obsidian-copilot copied to clipboard

Embedding error

Open wwjCMP opened this issue 1 year ago • 11 comments

  • [x] Screenshot of note + Copilot chat pane + dev console added (required)

Describe how to reproduce I use local embedding model through ollama.

Expected behavior Only found the error in the lastest two version.

Screenshots image

wwjCMP avatar Oct 04 '24 11:10 wwjCMP

Chunking has not changed, indexing logic is the same except for the added rate limiter. This is most likely on the Ollama side. You can try using an embedding model with a longer context length.


One suspicion I have is the updated langchainjs package. I'll see if they introduced any chunking logic change. But most importantly I'll log the chunks in debugging mode so people can better debug this.

logancyang avatar Oct 04 '24 16:10 logancyang

When I switch the embedding model, the error still occurs, but the frequency decreases. I hope you can resolve this issue because the embedding model with good performance has such a high error frequency that it cannot be used.

wwjCMP avatar Oct 05 '24 02:10 wwjCMP

@wwjCMP first things first, how long is the context length of your embedding model?

logancyang avatar Oct 05 '24 02:10 logancyang

old acge context length 1024
embedding length 1024

new bge-m3 context length 8192
embedding length 1024

Testing has revealed that the new embedding model also encounters frequent errors.

wwjCMP avatar Oct 05 '24 02:10 wwjCMP

old acge, context length 1024

This is too short and I can see why it's seeing the error. That 8192 should not. It's hard to debug for you since it's a server running entirely on your side. One thing you can try is to turn on debug mode and check retrieved chunks for how long they are.

logancyang avatar Oct 05 '24 03:10 logancyang

image

wwjCMP avatar Oct 05 '24 09:10 wwjCMP

image

wwjCMP avatar Oct 05 '24 09:10 wwjCMP

image

wwjCMP avatar Oct 05 '24 09:10 wwjCMP

image

wwjCMP avatar Oct 05 '24 09:10 wwjCMP

Is the excessive context length here the reason for the error?

wwjCMP avatar Oct 06 '24 10:10 wwjCMP

Hi there, I hope it's not off-topic. But we have an Orama RC where you can generate embeddings offline and on-device via TensorFlow.js at insert and search time (https://www.npmjs.com/package/@orama/plugin-embeddings). We also have this module to generate chinks via NLP, always on-device and offline. Would you be interested in a PR to explore this?

micheleriva avatar Oct 07 '24 07:10 micheleriva

Hi,

It seems I found the issue "400 Bad Request: input length exceeds maximum context length". This response is returned by Ollama when trying to calculate vectors for .excalidraw files. Currently these files contain compressed-json data, and apparently this is the reason for the "400 Bad Request".

I'm using Ollama and bge-m3 embedding model with a context length 8192.

I couldn't exclude excalidraw files from indexing using the Indexing Exclusions setting. The strings .excalidraw and excalidraw had no effect.

Here is a partial example of a request to Ollama that results in a 400 Bad Request response.

{
    "model": "bge-m3",
    "input": [
        "[[План 2024-07-08 07.11.34.excalidraw]]\n\n---\n\n---\n\nexcalidraw-plugin: parsed\ntags: [excalidraw]\n\n---\n==⚠  Switch to EXCALIDRAW VIEW in the MORE OPTIONS menu of this document. ⚠== You can decompress Drawing data with the command palette: 'Decompress current Excalidraw file'. For more info check in plugin settings under 'Saving'\n\n\n# Excalidraw Data\n## Text Elements\nAI работает только OpenAI аккаунтом... ^oPrRs9Bl\n\nChristmas ^ULzCVO5y\n\nGo shopping ^awvwmZ7G\n\nLet me think ^akT8VGoQ\n\nLaptop ^LYXzjYdI\n\niPhone ^aMwx3nKJ\n\nCar ^mBPbcPVn\n\nGet money ^vhj8VqVy\n\nOne ^T0OINzEc\n\nTwo ^pfvF4R7I\n\nThree ^fd87RUh4\n\nЗато работает вставка диаграмм Mermaid ^ZyyOQfHE\n\n%%",
        "[[План 2024-07-08 07.11.34.excalidraw]]\n\n---\n\n(cont'd) ## Drawing\n```compressed-json\nN4KAkARALgngDgUwgLgAQQQDwMYEMA2AlgCYBOuA7hADTgQBuCpAzoQPYB2KqATLZMzYBXUtiRoIACyhQ4zZAHoFAc0JRJQgEYA6bGwC2CgF7N6hbEcK4OCtptbErHALRY8RMpWdx8Q1TdIEfARcZgRmBShcZQUebTiAdho6IIR9BA4oZm4AbXAwUDAi6HhxdAAzQIRPKn5ixhZ2LjQAZgBOOsgG1k4AOU4xbgBGAA42hIAWAFYABjGZzohCDmIs\n\nbghcGeSiyEJmABFUqGrucoIwxZJ1gA0ARyNNYgBpACt9tuYATQBlAFkAawA8gBJEYwEb/ACK22K5UI+Hw31gwXWgg8MIEUFIbH+CAA6iR1Nw+PlMdjcUiYCiJGirotsX5JBxwtk0ENFmw4LhsGoYMMZgtSRsOMpqahBTsIJhuM4eFMhtoEqMWkMEjMVSMAGzqqaLPloZwTHgTbRDHiTIZzEZapUdIXMLE4hAAYTY+DYpHWAGIhuMhtgeBiIJpuf9\n\nlAyVq73Z6JFjrMwuYFMkGKITJMTEm02i05gkEpqRlMphMEotJAhCMppNxpqaZhM5jMeALfVMVYswic2TwRqMhprNarFuHhHBgcRWagcgBdRaVXDpcfcDhCBH04QrZnMScrtdCzQb4gAUWC6Uyk5niyEcGIuGOxGGCWbRvaEwmg5aiyIHH+y9X+C/NhsFxB80DOfALiFOA2GWLJclJMA8h2IoJWQmYENnBCkOQ5wFR4TU2gIqYe19EZjR4X1OhQxU\n\nn2tXsRhaKZswSKYMKo7CdllbQjTGBI2hGBIWk1KYn3wzUqLABUJkY18eyVGZmNY5DMOQjiimcFpTXfTUhMIns2hmTU5QkhV+MmJ82wmVVCIEpSdhUnY1LADTuLzQs2gmcYZjVFVjIQsA4g1eSWnNIYJhGaSWLYrD/JckYFLmKYIqYkt8KmcT/M0tpfTVMj5TaNszTsooHKKJyXPGd8sybdVC0mElkLAEZtAFWZ62tEi5Wi1TYs0liWgEgqSyGX0w\n\nsMkypm0FoS3lMZmzGCKBu6xzYrifjLR4FoEmtaYKPCiYJONeJGMLeUmzbAyjOWsrYoVJUhhCwtDKqp8BwklVtAMwjDOzRjFOuxDbpa8LBqM6aHrmDLGqE7QBLI6atUeoqAfKmZuJsrbByYgc+II962imoz0obLVsobTUUditGBs1HiGJzbzez8xqGym9KeDac1jW1f7lPYqn4nzJtW0EyqeYO/zWZzS0iwMsj1WRvmYsa5w0fNbUKIK0XJnF97mr\n\nCvjDaEpGuqVnqVbVoXNZE9odYUiXoc0ozLRCpV9q24qwFKwGLe4qydJC41wsE+tS38/DYbq0GaaVa1KZVhVLRJhswrmkTexMgnhKEpV8Pfd2RnjnDTJCsL/dmdpDOZ5CHu0JKEubLMeY9ovOLiWnxizaaCM2siRgkgm31q9oRoezrW/UzSwoHXSjPmImJM1T70p+jy58u037P5lXNJz3bnemJv2X8yaRKY81BMiz3vfKzS5TVaSGKehT8yh5CTTG\n\n2bRgi/N2ev7fi6fQEgjIswlRhFhLIdFq4xSbAPrAWKKZsVqNTRiNeshlbKDh1EMdKElVZ+37BXDe/9lbITvjMUYVlGKRQoUOXq0CcEMTlK2XyJDzbISXkZN8hFD7fUUngk05dpovj7orLepCdgE0tFqMiOkg7rwEYqEs21zJFnci3JBN1GqWjrlmVs4V1TaX7vQ2mQl3zGibNNDR4j2E7BGnXTGtMqG7VTngtGXclQtC2vPTeJUAF2LiHKQqBUCp\n\neIImFKBvYizyhYttMRfiJFFFrvFF2nlzRCV7LTCSzV6YjXMQ2Taj8J6SUmoJFUfEFIWNEmHbRCQ65qJVF9ayBZC6aJ9jXOpeYbZJSLBtYsh19YChCtneK6jWk2OQTXHJ3MX7EwIk+GpZCl6eVVCxeU+FQlDGKaMWGMzxhzPGOad6yzspdPWURFUGFSSlQgPgUIUBXT6H0Goe8AAFGCyY0C7gAvaQg+hVx3gQK8wI24RBlHApBSUcAQXMDBReBCaM\n\nKGDlVBRNUvpmzqmrtRH6jE5JiUIjZY+aFTQGwtDwHs01mzpU/Ai00wlvLMQoVZJs2Y37UVIvA/29YXzERpcS5sZoKIUIMhzQcgkqJqwVjreaOM1kSviIZJilKSajEsYs6izZ8JanJegyY7k2VgFQQVCKWoHrjEmIHXUtKBXkstL6FlYq+U7Elaqe2voQaXS1PK6y5F+Ld3rPhLx3qkoh37FZNUaUGrOtNEWPi4xBxWV2nxb100IHg2ksWEK0kU0j\n\nUYm2Miz4ZhqJTc9TxeZkX9nVN68xLECrGiSgNQh3qebmkImDNUBZRjNv7NqdKPdtTnWDXGliOlwoPRzAJIdec9J8U2oRIswbDJKlAUxUi+Zq2anzBmjmqpN1mm9Tu3srKBpZsEsY4lI0nz9iKhDbOA1vVFrlArAa20i3ZjtBe0tD0cFNjnsM+VClCJeNytjclBUrXErmGB4sJqSYEQ/dG7y0sQ3IukgWN8AGByqlkTZaSBsnXYuJhtbUnlCybSVA\n\nBxKJYBRjBabxB6Errn5AAL51EKMUWAiB1iVAQNUcgtQhTdCaNwISiwhN9AGGUfsFq50GUuCsNYEhcBDCDHsQ4wR7ynHOAgS4oF0CQgok8BIwJJjOEkHAAAUgAMRmJCJ4AAVbKtxNBBjhAiSkYoIC0gfB2R0uICTECJGgNGEHigOnJAgDzZQvNujpEKBkVYtyTiJcUTk3JeT8lQpAawooyhZalDKW1poDJGhsixPUMpE2mmmh5OB75bK+Yi1GD03o\n\ncr+kDIsEMwERxCEjG6FrsZyAcATLgJMUAUxpmGIMgim7OYawumWCsVZxutAVLy/sPZw1ldCwIXjemKIZP6njIUPWxwTlyDc+ci49PfPXL14gSW/x7klAee7J40gZDgmgS8Qpry3k02yJ8AbXzvgY0Kb8v4vn/kAsBLsqAIU6agh8r7U5EmGokuhNpqM6Ur0mLyzm6d90C3VgrTyiaq5sMmZxVBwltR47nffJmbiaIc0MSIzqlOtE4TVvNQDQleVX\n\nzwYnesYUzQyXlH/Yp+CdXxS7sJQpHtmd5lZ8I3uHOpc88bHLgXiv/J3TyYz+JXt/HqTRn3QyFDZjT32g7Dh3Fz5j3SgxJViCJlc7bsDOUKyvEMziVGuxu8B35hK6K2yUvNKy6gwrLFRRmywwhjqBs6oN5S8muOv1g5my9sgeHJeED8p3ubL5KXS8KFPnyQHba2VDoEyYgpEaWodeu4SbY9SS8Eb1dfTE9O700ZWRFZWpxhUeAl+4uqTd1oPIBoJb\n\n37i9qKLaiH8XrHsVlnj+2mMBspXjuNQ5txOUHNQ/N+N2j5wa+c6T636KhDOwI5T87kM4ho+GwX839PnfyFyVKOIsWeUA5LdXQr4qzn4T5v7b435JJ1KtosTvgxyjDH43yr4tTl7Bw5id47ZJJ3R14eRq5Myc7tKcSnyFqIzqjyQeQSRQGlxMpX7bZS4mg9wd7xTkrvgKTvSfSTBkxFiMxG6IE7yKj5p8TFj3RkzjThwKqyJXqjTky+In6t7OSJAa\n\nz1gqizQQLqpJLxDzLajfrO68xu4EHqQKi05mjqgCiH42Tno1x1hWRyjxqS5AHc445067TtCG7+6x4OJ1q/o5zwH4FORSQGR/4Dg2G/xGQSSTQcz8SzCwE2g+HFKB5467TWiJTZgWGSJTSFgbbi44JEzFJp5eJ6LGhhRcHbQmQmjeRdJwxP72E7AgEGReK9jTSvwmTNTZhMRJTapoEtLFJ1JPxdzeRPrxoGoBR3zjDlJREdRMzFIEyawKReLkr9SX\n\nSHRLzkohR+hSEU7bLuKFg6RDLFiP6HSdLhLoLJ4bHVGYGKiYrzCNJkRGhuEBSHHZTHGVw5FnGSR3SXF1GkTkR3HmiKhHFJ7PEyHTjXJfj3KPLPIyDVDvKwRPY/KSisD/J3LHDAoshgpaYQSI5QowpwrwSQahRVyH6MLag5wAZO78T/5awNH4SknWjET4StqeRuxvS0pvpkSXRbQPRcEiSknZjkxyLFj4QbSYYCiCSBoeSNKWSYYb75i5zZSiIGqI\n\nr1SHavQ2GbRuGIpaxpT1icyjTKEAZ1qNrcpGRgwKnQJmQVIdQhrSQEaGrQK0yrpFpYI/w56frUL17zFFqHymmXosEhRtgazrTekL6XqH60w2HTQHo5jB45hhlmhxopoToiFFojreIprwajpbQahMHVoJTUKTCGS+jMkXrKJWQ4InpExVq0o4JeQzazB0Rvg2moJrJxrxSW4wFqFGoDQFQUJ7RNj5kYGSrSQFRGQCqeRZi24apSrmKQybpyrWp9Jh\n\nksR5yTDyoURJFWKgJb55KrlRk9z1Y9qMRg78rCKWq/q3rrpVk4zWh5iqjhQCnJoskDiPECSkFZoBGUaayvpgZAZCTkqMb2QsZsZCicbRY8Z8aUBBjibNCoCMRiZMA9AcD9AcCDBoDSo9i9j5bLCrDShKYdZChqZHBw4I66brBWYACq3w+whAtmkIkI+A+g+A+wkIFACQAASuUGwBMICK5vCIiMiNFt5kGOFk6AFkFuKHXI1k6FFqiLFj5vFsIIli\n\nyMMByFyDyLAJlosDlmKPljhagLhAZJ9IpNlEWOkhhkKPqHpQUtxCqN5CRvxKHJJbiM1jGOgD6H6AGEGF1mGBGMQM5esHGMNomJ9hNoFumGgD2NZRQpjJSnmMIotpWNWOFU7D2NEhzNtBLqEfaHtsMNmBthRDthAKduOBeJduQNdrCXdpuEpVDs9sUK9isO9meCjj9pKH9oCnpiisDh5KDgRrcssJDqgLduDkBCBOiZCsUNBLBPCnIVlihBri1CZV",
        "[[План 2024-07-08 07.11.34.excalidraw]]\n\n---\n\n(cont'd) XPxKIjHs5JbCRpusRJzLQa8TTulCNPhLlPxILkDNWb+bOjtXoU5LNfIRcRLm+KMB5MPr4UDK/G2PpHxGkt/IoklAvnVI8dkUCSbs5CaEwVumnH7hgc5JNAZMur+qdOgVLvrD+t+uSmoilHgpwq2QNC4u7AgcDbhAqFtFSrMG2GQfekgfKA2WOm+M9VLrXDpAKaqFtaHqkepJNAyugsRJfNYi3lTgYZNItElDAqFDbngmtL2K6sJNvmHq8bhPrM7E\n\nZK+vmCihjvEPNMwpskDafr6BcSopLa2mlRJCaDGQ9LlJ5M3PjafnHltOqBOjLOgdkpHAxI6bTKcVdbdATH6gtBthFNZGoZJFPC4WYlblrfTc1ANBzPFN3LNi0hNMvCWB1I2NwSPnLWFHXGep6sSRlSZEvLVn/slD5DwQTfYjInRIWKRvAhAZJCaAEYWK7KnFfFLgTDjTNt9NwprCZHhD9BXYJHIpMXLZwkHDwkxHwrro1EvMlMJJfIjOrnLbXQWJ\n\njNWTjDPmIWqPKCPFoXYR7XwWLNmFHfkYtGzWtdoHThEfFIjXLRHglLRovrTb5OLdAttARHVggi9SrCbW+oUoOCIQvKtNxItGarTItFbXIYaJ9N+i/dqAxMdROQYafQKFmLOgYpfTvThJNO+MRIBpbVDWjGfclNMAtE3XLaUjwm+fKKqD4WwdlIRHmDzaA/zc5HfCJCLrlTggrsffYp3CVreRfV0XLWbkwhGjpHug1nrh/KsfSpUe7Xze7u4ftJQm\n\nlA9KI5wyaCJLAUhiHPVuMrI/oWAKQ6YsaAVJelQ3rmtH3U2BzF9bLWgzsM1KyRguRp0dXfYgpMJNaA9CWLTtrXIfY0xPFAWE41XV3Xae0Z4yEUCSCeDmCQYBCW8sjhVb8oiYCiiaCoEGNZiRNdiSCriYhruZzMMr2o0vxFKTgrMB2k2GMCevqUEo9W+O5FER2TGq3dWSJH3eMKuUWlqFBvkT2AWPWDuTPK1GqRhZVKuf46zhaA2aOqubTgtLjA0m\n\nzWrKTLik3DVl4i6dGj2HKO0ERFbq7AOfEA0aMARBmjZWYquXEkPNlDnFergtarjJUpzHjnXuqYLHMKqhFInd4o2RocunWS7LbTjKuZ5I9PnBsjpAEyCztAKJC6koWCC4GVqKlHNBefynecaozfFD5Cudaj/iYZukEjVBEni4OP/h2k3HKZc7evtMxFeYc9s0WmaN2UaKQYRJcxaGFJdPrU+L8+rAUw3O6uapc6lMRB48RKc00yRNi/bEzO0CFJcy\n\nlOvAzHKd6cOdymbcogJOFJhs+YS0PI+pWZBpaJMGOi7OlHNpzP+SVIBfkOxpACBdxlUDUJBfBcJq0BOQwG6xJshVJu+L/N9GyksAprpRsC0KpgcIRXpsRfhXprclZv8EMC8NcKQM6MCICPZvsICGRWRfoC0P8NgAAIJHg8Xub8UyXoiOX4iTbBYSX2h+aRbls0iyVBgJZMjVWoApaQBpZqUWWWj5ZaV5aLC6W4QsTQIW1RmZkOXmWVbeTHTMqGLN\n\nI35eYNt+USBuVKgeWdahg9Z9bRj+VDYjZjYhViVf7GgLI7FYyFKevlgJUra8DUyeKFRk5jSZXwnZVoByxfNpTDgMhnYlVzhlUIBLg1VwnFA+WPagedaHiNWfYAe/Y3jtWPjPhzpvgfhfj9WJOSjuiw7RvaaLCTXni5NyPo7+SY62PwNGGG3bYY5S6BTgLZwFjy5WLH3ke6PlSJBFotgMQO1v20dX0MKPwcyppF13HUysI5jd6Q3h78EDT0yDiZLS\n\nFQIXRYOBqiMMMkfgNPrl7IN8PV11K7qPEAlVEUcg3xDlPnucwoMf4BL77ZH8TkSxECfTwiN6Q+JQ39j27SbFg/qAn02fSlwxLdVeJKqcMR0wNimVqA3zXm7BwW7Fh8d67TKgLsNWIad6N94YwDQ9kXJFTNHpEuE6TZRb2AGmemTAJRGQvqceeC1vjkxxnbX8OmdTzaSzz6QU4mRxBGj2c3FrnpdOSaTtFcz2yvR/xO2tSJ6P4yOyGMNOzkl5hLUT\n\nFE6NQYMuxWjeImfseZSn3zcdpR0rXG2wy/WEMW2zH9fbc0PhTwbGOUPWiHflxMEtha3v3vw7ebR0keQ6liPQwOJZrzDKOxzncT1j6v5T7gEmTWUXQ8Tk1A/IR1K2zUJFS2F/2+yELEk6TB502vGdIMRdKrHC1fPM5n09wCSWi6FbeNQ9EeStigJu0o8OE0x0xeIil4HFJLyXxfTJliqmO74aGHmcyk9ncvc7ClLWH1FyiL0Y0nxHfoqC/k8zckeg\n\n19kiSBrKhS/aJE23kpV0QqNs9TT7QWR1qN1wOSTw8GQ5QLHTe8HIS14VyMIWshfkF6617SRyhahPlfc6MK96O1yJokbwG0K2iZx/Ekw0ybTwJxyvGqiGXCQS4A888dL1JJQ9pEJW/A3R9Ny9Ib3VcmR1LRKZL0qAnXxRPYcxNPIvJQkJNQdJMArInZPgr4dQT1/TWIaiRCMCmZqllplLlhSMSWPULeqCRm3mq2gxkFTeoiohQ0ZEQaxGidPtR+0M\n\nTkxslLMKpCQ7EHb4SKqiH8pIZb/ntqh91Stk/zDbM4Kh98uA3Wh+nP116mlrnKHYKhx/6rlX9MLpzCSLHzmnQFNthWJAb39oM4UV8P00sRvNw+4+WmJngW5zAQWKtDeEETNCwFxmZoeWFhiXR6R7+MSB+AOD7SJRV+RjMFmGRmzZcsBaNa0JunOgewto4zAsPNwegUDBS4zbrtzEYF/9uS1qaWtMFmxGgf6WaVcqT0+pZhkW6cGPGrAFYLdO8DeN\n\nsKuRuY5g6i+YeuuKQEFyC7UaKPutMFkEpx5IRkNeI0nv5VNYqwqFUI2nAEn8RUZEc/nMCDQsl4aRkZXnNHKSNkmMRQVjHa2AqlAnWe2fjK60aCcAawXbL1n4MQqSZiQXTOhgP3woht1guACYBG3UwIAAc8ORvpKCuASB9gUAAAFrOhPg2AOABwH0DOBsAAAfQ4ATBIQZFQgAAAkOA7QUtnxSpACUW2VbUSmFXEoFVhKFIJtugEEp3ZFK24ZSkKB7\n\nYZY2QAoTSiKG0rDsZQZoO6LOUzydFvuxQCyoaEMin15YGCP6OPTCyrt+sLlCABu3ayeUd2PlNdugACpHtgqiwVMKFWJCC080tTDZMPnirLZbhioPODQUS7vs4cgNCXq0V/ajhiqF2QDguGA43Zoc8le7JB0GrgiXsMHU8HByBEId/s3woHC+G6rodwcmHavthxGpEUUhE1ZHC3xI43U2O3vbHGj2/g8IQGUNfBEWjpJlMVagPYXqbhZwRQqk5Kbx\n\njSOpgiDWiXTF3MyOchrZpgy6OYtHhpEKh2o3ZFhMXWtoxoOYPnAAqtVwgBc0aL8TkXR1hh7d+ilvenh7hfKxVlCMtWHpxEG7M9p6xqY6lsJNF1wdoi1YVFj1M7gMWUsNDkWN3oQwE5grRBLjYwp44RP658buOsV1GTwFUvoY1Kd3l7W9OIgifsFP1bKQ0lcT4Oom+FwIyiwGWUcyIgwvhpcoa19ObNnyZEaiLo7QI9I7ytFJJH215G2I5x9FkiBY\n\n1/c0F4Rz56528Yvb8nw0j6mczcPYBlCsX7p5dw4kqESI9Wp7SEBRsw+XBKwRqfDP8hhIDEoQojtdSuvo2/AA3DQzx1GfXKGhRHYLvcTW3XWsVGKKDkJKEwyW7uVkHEANd06jQ/tHjiLsFhoCnZjidW0ST1fovHQ8cDX9EFNQcv9VaqXUlY6Q1ekYr8UohgI+4Ix5YsAOz1xRvgWIao7eiuKKCTQdiDRfxv2NWpp5N8DopCdBNNB9xf4pMV9qtVBq\n\n/UyyU/VhN0X157RtootakYdyYIeJTWxEgUdMXqg0MvuF4lmC1BVZ20i8aYxhvYnCj1oGchOf8SqJ1SENLRxopJLvFYacSuRkcZninXHFlEFuTuH+JyMfoqdNUIjQsVHwkYciNQmwriThDiBRJ0oaKf/IDXHHX0k4z0UcVXEfr7Qo6FTb0TJMkiBRp6ro4MbaWJ4C8Ye44iUd9Dk7ASoJtdTxoZGJreQxRmxOuEOWfIUDY4x9FomZFGCCl9JZXBUA\n\n0QIgUJpxiw2zkUXKas8o+fUQhvalOQ0dxGbMLaBYIokCSSOnndoOzGWoc546wo4rsEXVFR8WiffVAaqEjpM49cp8ReuhU3FOdTOO4/vM+GELETSin0PkQ3TxoCipp6KUOCND/HzS6671N2NSKuRu5bkZfOJpXxhLYiwsfyWvkCnr4ZMCOzfYjraSZS0Z8qL5eNPUwAyFEWIsiFgmLickslhoMZbAZqjRoAYrmJMWYrlIEhvNPRjecJCEn+pwNUES\n\n6Mpuw1lg9iEZ1WH3Bs0qZA5tiU6DuErQKbzpDmYaZ3P2D0QMDCGzaXtNzzfTQN3wzaHvsTUijKIu2tpQhJLS8hskABq/WMUVEUE5gX0I0YmXPCiT9N6ogaHmQmljEDgSBbTX5owg+kY8yeI6YWfaP5wwZx0CLS8qLhEZPlGasVIdEg2pRjpmek6Ksg7m2LmJwMM0YNCJCQZSyk0EBVBP6XtG8llcKyYNL3AUjJ8swVuAZubPpIS42RnBHmbMGMoj\n\npLBcidoJ7KJiMJSsRRUOXAjDSupI03qEBg3j7iioiy0aE5scwCKfVe+TTE5o2AJ4lZWy6MkcXDDmCYMtYWSKsjDLGDJj407Leuf2AiiREI5+ZHmV+S+K+RNUmgqsrTVwF1k6ygTX5t5GsjBwLaCabgZRibApjTCVpbpj0mtZexbWRQe1iUC4wSAwKLrOCsENuH7yEKSFFCp2xJhboKaqQ6IUpimDxCo2N02Nv5SMAcAKAVmOAM6GYC/AhAzATUE8",
... more 35 lines
    ],
    "keep_alive": "5m",
    "truncate": false
}

ivannin avatar Oct 12 '24 14:10 ivannin

@ivannin I see, you can exclude them by folder instead.

I tried exclude by extension but if you take a look at the files they are actually md files .excalidraw.md.

logancyang avatar Oct 12 '24 16:10 logancyang

Thank you for your response, but my Excalidraw files are not in a single folder; rather, they are scattered throughout the vault in theme-based folders.

It would be wonderful if it were possible to exclude files from indexing based on a name pattern, for example, .excalidraw.md.

And returning to the 400 Bad Request error, I conducted numerous experiments and concluded that:

  • This is an Ollama problem.
  • The error occurs when Ollama encounters a long string WITHOUT SPACES. The length of the string is approximately more than 1-2 KB.
  • Ollama does not consider the \n characters as whitespace or word separator; therefore, even if they are present in the string, the absence of regular spaces triggers the error.

ivannin avatar Oct 13 '24 15:10 ivannin

On version 2.6.9, everything work fine.

wwjCMP avatar Oct 14 '24 16:10 wwjCMP