AFFiNE
AFFiNE copied to clipboard
toeverything
[Bug]: Workspace embedding queuing non-existent documents
What happened?
Main Problem
The workspace embedding is stuck at a certain percentage and is not embedding the remaining documents.
Investigation
I looked into the logs and found that the top document queued for embedding (id: 31_Iv8aUs7, see logs) does not exist. Manually typing the ID on the web URL gives:
I also queried the database directly and got nothing.
affine=# SELECT * FROM workspace_pages WHERE page_id='31_Iv8aUs7';
workspace_id | page_id | public | mode | defaultRole | summary | title | blocked
--------------+---------+--------+------+-------------+---------+-------+---------
(0 rows)
I assume this is the reason the workspace embedding got stuck.
Finding
Some documents queued for embedding do exist, and performing an embedding for that single document worked. I did this by attaching it to the AFFiNE Chat like so:
The embedding was successful, and I could see the workspace embedding file count increased by 1.
Distribution version
macOS x64 (Intel)
App Version
0.22.4
What browsers are you seeing the problem on if you're using web version?
Chrome
Are you self-hosting?
- [x] Yes
Self-hosting Version
0.22.2-canary.1
Relevant log output
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [EventBus] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Dispatch event: workspace.embedding
[Nest] 1 - 06/18/2025, 7:01:45 AM VERBOSE [EventBus] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Handle event [workspace.embedding] (CopilotEmbeddingJob.addWorkspaceEmbeddingQueue)
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [WorkspaceResolver] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Trigger embedding for 66 docs in workspace 64e2795a-a91d-41c5-abcb-313cc06afb4a
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:31_Iv8aUs7
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:m4-maZypju
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:XuSGY_JRZuFncS14LjFs4
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:4xNwDM5KRR
[Nest] 1 - 06/18/2025, 7:01:45 AM LOG [job] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job started: [copilot.embedding.docs] (CopilotEmbeddingJob.embedPendingDocs, id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:4xNwDM5KRR)
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:dSBORg4iyXXYQ9-lzOLUk
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:vZfPRmx_gwQ5mfGzaWD8q
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:fE43XQsCxi
[Nest] 1 - 06/18/2025, 7:01:45 AM LOG [job] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job started: [copilot.embedding.docs] (CopilotEmbeddingJob.embedPendingDocs, id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:vZfPRmx_gwQ5mfGzaWD8q)
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:5Ny8Bh0pqq
[Nest] 1 - 06/18/2025, 7:01:45 AM VERBOSE [WorkspaceResolver] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Check if doc 4xNwDM5KRR in workspace 64e2795a-a91d-41c5-abcb-313cc06afb4a needs embedding: true
[Nest] 1 - 06/18/2025, 7:01:45 AM VERBOSE [Locker] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Client cluster:rrpNsCNNF1f5UrdzJ75GH:PfkisxLRTkWjKNeVn9cq2 is trying to lock resource doc:update:64e2795a-a91d-41c5-abcb-313cc06afb4a:4xNwDM5KRR
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:RvhLVKVvIbqh6_fo9x_Ce
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:jZ9--UJozr
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:u5fc_8WOE4
[Nest] 1 - 06/18/2025, 7:01:45 AM VERBOSE [WorkspaceResolver] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Check if doc vZfPRmx_gwQ5mfGzaWD8q in workspace 64e2795a-a91d-41c5-abcb-313cc06afb4a needs embedding: true
[Nest] 1 - 06/18/2025, 7:01:45 AM VERBOSE [Locker] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Client cluster:rrpNsCNNF1f5UrdzJ75GH:GWuNTun9LZh9hP_Cq285N is trying to lock resource doc:update:64e2795a-a91d-41c5-abcb-313cc06afb4a:vZfPRmx_gwQ5mfGzaWD8q
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:aVHxKyUaYnj6SxKtDcKwi
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:ucwYpmbbnEiABZA96Ae6g
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:FF2A5BBoat
[Nest] 1 - 06/18/2025, 7:01:45 AM LOG [job] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job finished: [copilot.embedding.docs] (CopilotEmbeddingJob.embedPendingDocs, id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:4xNwDM5KRR), signal=undefined
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:Ilm4TrtKepO0-DZ-fTp0m
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:lExEz30NmF04LbLpOirVA
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:QijNna4or0I2w-SfMojst
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:oUVclMJl3tCUYFcg4M1ag
[Nest] 1 - 06/18/2025, 7:01:45 AM LOG [job] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job finished: [copilot.embedding.docs] (CopilotEmbeddingJob.embedPendingDocs, id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:vZfPRmx_gwQ5mfGzaWD8q), signal=undefined
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:XIE1qOfMp3WGuxQk0Srz3
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:L54OoZjGQ8qcTaNJsV3--
[Nest] 1 - 06/18/2025, 7:01:45 AM LOG [job] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job started: [copilot.embedding.docs] (CopilotEmbeddingJob.embedPendingDocs, id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:L54OoZjGQ8qcTaNJsV3--)
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:5IwDwxcGXcPLbbZePAkvK
[Nest] 1 - 06/18/2025, 7:01:45 AM VERBOSE [Locker] <selfhosted:ws:1c1b6680-b791-49dd-b8cd-0581799c492f> Client cluster:rrpNsCNNF1f5UrdzJ75GH:Ndl3LslWeK3Hu2KSpxICg is trying to lock resource doc:update:64e2795a-a91d-41c5-abcb-313cc06afb4a:64e2795a-a91d-41c5-abcb-313cc06afb4a
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:KBiXuwG4kI7ADyt_HW9Ze
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:5mgnOzT-odQZxFFak9_6O
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:N3P2NT0CbwZcK7eIQH_yo
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:tAD9CWOrl0hnAHxABxCD0
[Nest] 1 - 06/18/2025, 7:01:45 AM VERBOSE [WorkspaceResolver] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Check if doc L54OoZjGQ8qcTaNJsV3-- in workspace 64e2795a-a91d-41c5-abcb-313cc06afb4a needs embedding: true
[Nest] 1 - 06/18/2025, 7:01:45 AM VERBOSE [Locker] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Client cluster:rrpNsCNNF1f5UrdzJ75GH:2FprrZmGjLTE8zx-iaRpl is trying to lock resource doc:update:64e2795a-a91d-41c5-abcb-313cc06afb4a:L54OoZjGQ8qcTaNJsV3--
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:XrbMpptjJ3FMSflT9joYW
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:8J42PzMOkZTMx75pc-k-t
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:yW09Ehg3W4Xxb6NrD9kB1
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:asXjz69qIJRvt_nVlhEUP
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:JWASirQgI9O2xrcfYFcqc
[Nest] 1 - 06/18/2025, 7:01:45 AM LOG [job] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job started: [copilot.embedding.docs] (CopilotEmbeddingJob.embedPendingDocs, id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:asXjz69qIJRvt_nVlhEUP)
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:ZCXp4dfkp4
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:dh1AYZYr4d
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:xyratOiDlddLn4TyHcaE0
[Nest] 1 - 06/18/2025, 7:01:45 AM LOG [job] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job finished: [copilot.embedding.docs] (CopilotEmbeddingJob.embedPendingDocs, id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:L54OoZjGQ8qcTaNJsV3--), signal=undefined
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:X9_TEFgkIr9VF17x27YO_
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:nwPp82lE1XWTzqZ306YZW
[Nest] 1 - 06/18/2025, 7:01:45 AM VERBOSE [WorkspaceResolver] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Check if doc asXjz69qIJRvt_nVlhEUP in workspace 64e2795a-a91d-41c5-abcb-313cc06afb4a needs embedding: true
[Nest] 1 - 06/18/2025, 7:01:45 AM VERBOSE [Locker] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Client cluster:rrpNsCNNF1f5UrdzJ75GH:kDQp7aiCViUaOMM3yI_rX is trying to lock resource doc:update:64e2795a-a91d-41c5-abcb-313cc06afb4a:asXjz69qIJRvt_nVlhEUP
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:7YDyYeRz0rZ-JJSBT1-x-
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:3Z9eJsRKyp
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:0__-YUbNJIfVOVrirUh3J
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:24FDQWtY_uwM7rOR4TUSt
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:Pawqt6fc9q
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:DxxQ-Nu2DJ
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:xIxNZ9c2_N
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:l0COOAZ545
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:nXVYwFINJm
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:OZte8uyBOm
[Nest] 1 - 06/18/2025, 7:01:45 AM LOG [job] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job finished: [copilot.embedding.docs] (CopilotEmbeddingJob.embedPendingDocs, id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:asXjz69qIJRvt_nVlhEUP), signal=undefined
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:C2ZUMWl7L9
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:4GO4U6UG_r
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:b8nbx7yILf
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:ZM3T4y3jLl
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:QQO2-SpNKs
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:MQq2rumsK6
[Nest] 1 - 06/18/2025, 7:01:45 AM LOG [job] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job started: [copilot.embedding.docs] (CopilotEmbeddingJob.embedPendingDocs, id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:MQq2rumsK6)
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:2MvqAtzsew
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:PeLnMigg5K
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:-duQ5hy7O1
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:zZ_lCkzrSF
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:J5e6NEdWEC
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:JckAEP5P13
[Nest] 1 - 06/18/2025, 7:01:45 AM VERBOSE [WorkspaceResolver] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Check if doc MQq2rumsK6 in workspace 64e2795a-a91d-41c5-abcb-313cc06afb4a needs embedding: true
[Nest] 1 - 06/18/2025, 7:01:45 AM VERBOSE [Locker] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Client cluster:rrpNsCNNF1f5UrdzJ75GH:iN3yD5lC76RwW8MMDwq5d is trying to lock resource doc:update:64e2795a-a91d-41c5-abcb-313cc06afb4a:MQq2rumsK6
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:xyfvNb6pNu
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:n5OPYaIsHs
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:7yLQWcygVl
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:UzaU4u08iv
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:17msEJAZ1b
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:UhX1SK8njt
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:4RFzQwqX6O
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:y7TYe3XaXF
[Nest] 1 - 06/18/2025, 7:01:45 AM DEBUG [JobQueue] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:Gt3vHVAEAZ
[Nest] 1 - 06/18/2025, 7:01:45 AM LOG [job] <selfhosted:ws:fa0e7f30-a461-4c38-b977-90a06223525b> Job finished: [copilot.embedding.docs] (CopilotEmbeddingJob.embedPendingDocs, id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:MQq2rumsK6), signal=undefined
[Nest] 1 - 06/18/2025, 7:01:46 AM VERBOSE [FsStorageProvider:avatars] <selfhosted:http:4b2886f0-5871-4e48-a29f-bdf6ff326595> Read object `0c41765a-0e90-47f0-bc7f-3954a7a02767-avatar-1740546789353`
[Nest] 1 - 06/18/2025, 7:01:46 AM VERBOSE [Locker] <selfhosted:http:b4cfb8f9-eb29-4100-9b7b-99bd4c713b13> Client cluster:rrpNsCNNF1f5UrdzJ75GH:PIIBgrYB4Ralh1f6tkTZb is trying to lock resource doc:update:64e2795a-a91d-41c5-abcb-313cc06afb4a:64e2795a-a91d-41c5-abcb-313cc06afb4a
[Nest] 1 - 06/18/2025, 7:01:46 AM VERBOSE [Locker] <selfhosted:http:031569ce-8e5a-4382-a091-d8be692e7937> Client 798d4839-0eb7-4b18-867f-715f66c04c44 is trying to lock resource copilot:context:ecb5ae2e-4326-45a5-a8fd-fe25c682d7c4
[Nest] 1 - 06/18/2025, 7:01:46 AM VERBOSE [Locker] <selfhosted:http:0af8972b-e572-4ea9-9cd6-b1f245eafd08> Client 3c4bdbcf-44f2-4d3b-8bd6-fb3ee6b20fe5 is trying to lock resource copilot:context:ecb5ae2e-4326-45a5-a8fd-fe25c682d7c4
Anything else?
No response
Issue Status: 🏗 In Progress
🏗 In Progress
We’re currently working on the issue.
This is an automatic reply by the bot.
![affine-issue-bot[bot] avatar](https://avatars.githubusercontent.com/in/852626?v=4 "Published by a pub.dev verified publisher"){kind=small}
Adding more info.
I restarted the server and the progress increased, but still stuck:
Looking at the logs, these are the remaining documents:
[Nest] 1 - 06/18/2025, 10:32:07 PM VERBOSE [EventBus] <selfhosted:graphql:cfe61bd8-eb09-4d2b-8311-39ff3ec35ba3> Handle event [workspace.embedding] (CopilotEmbeddingJob.addWorkspaceEmbeddingQueue)
[Nest] 1 - 06/18/2025, 10:32:07 PM DEBUG [WorkspaceResolver] <selfhosted:graphql:cfe61bd8-eb09-4d2b-8311-39ff3ec35ba3> Trigger embedding for 5 docs in workspace 64e2795a-a91d-41c5-abcb-313cc06afb4a
[Nest] 1 - 06/18/2025, 10:32:07 PM DEBUG [JobQueue] <selfhosted:graphql:cfe61bd8-eb09-4d2b-8311-39ff3ec35ba3> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:4xNwDM5KRR
[Nest] 1 - 06/18/2025, 10:32:07 PM DEBUG [JobQueue] <selfhosted:graphql:cfe61bd8-eb09-4d2b-8311-39ff3ec35ba3> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:vZfPRmx_gwQ5mfGzaWD8q
[Nest] 1 - 06/18/2025, 10:32:07 PM DEBUG [JobQueue] <selfhosted:graphql:cfe61bd8-eb09-4d2b-8311-39ff3ec35ba3> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:L54OoZjGQ8qcTaNJsV3--
[Nest] 1 - 06/18/2025, 10:32:07 PM DEBUG [JobQueue] <selfhosted:graphql:cfe61bd8-eb09-4d2b-8311-39ff3ec35ba3> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:asXjz69qIJRvt_nVlhEUP
[Nest] 1 - 06/18/2025, 10:32:07 PM DEBUG [JobQueue] <selfhosted:graphql:cfe61bd8-eb09-4d2b-8311-39ff3ec35ba3> Job [copilot.embedding.docs] added; id=workspace:embedding:64e2795a-a91d-41c5-abcb-313cc06afb4a:MQq2rumsK6
DB query result:
@yurisasc Thank you very much for providing such detailed information. We will fix this issue as soon as possible.
@yurisasc can you check the docId in snapshots table? They should be exists.
I'm happy to help, even if it’s just by giving detailed info. Thank you for the quick response! I appreciate all of the team's hard work 🙂
Yes, the document IDs are all present in the snapshots table.
I had the same problem, but I overlooked the fact that you have to install an extra container; the indexer. Did you read this: https://docs.affine.pro/self-host-affine/administer/ai and: https://docs.affine.pro/self-host-affine/administer/indexer
