lemmy icon indicating copy to clipboard operation
lemmy copied to clipboard

Published 20 hours ago • verified publisher icon LemmyNet

[Bug]: Searching for new communities returns error: couldnt_find_object: duplicate key value violates unique constraint

Open delendum opened this issue 1 year ago • 1 comments

Requirements

  • [X] Is this a bug report? For questions or discussions use https://lemmy.ml/c/lemmy_support
  • [X] Did you check to see if this issue already exists?
  • [X] Is this only a single bug? Do not put multiple bugs in one issue.
  • [X] Is this a backend issue? Use the lemmy-ui repo for UI / frontend issues.

Summary

When searching for an external community for the first time, an "object_not_found" error is often returned and no results shown.

Subsequent searches are usually successful.

Steps to Reproduce

  1. Copy the URL of a new community that hasn't been searched before (e.g. https://rblind.com/c/main)
  2. Search for it using Community search on your instance
  3. See error

Technical Details

Jun 25 10:45:26 lemdit.com lemmy_server[27525]: 2023-06-24T22:45:26.920660Z WARN Error encountered while processing the incoming HTTP request: lemmy_server::root_span_builder: couldnt_find_object: duplicate key value violates unique constraint "community_language_community_id_language_id_key" Jun 25 10:45:26 lemdit.com lemmy_server[27525]: 0: lemmy_apub::objects::community::from_json Jun 25 10:45:26 lemdit.com lemmy_server[27525]: at crates/apub/src/objects/community.rs:126 Jun 25 10:45:26 lemdit.com lemmy_server[27525]: 1: lemmy_apub::fetcher::search::from_json Jun 25 10:45:26 lemdit.com lemmy_server[27525]: at crates/apub/src/fetcher/search.rs:137 Jun 25 10:45:26 lemdit.com lemmy_server[27525]: 2: lemmy_apub::fetcher::search::search_query_to_object_id Jun 25 10:45:26 lemdit.com lemmy_server[27525]: at crates/apub/src/fetcher/search.rs:19 Jun 25 10:45:26 lemdit.com lemmy_server[27525]: 3: lemmy_apub::api::resolve_object::perform Jun 25 10:45:26 lemdit.com lemmy_server[27525]: with self=ResolveObject { q: "https://rblind.com/c/main", auth: Sensitive } Jun 25 10:45:26 lemdit.com lemmy_server[27525]: at crates/apub/src/api/resolve_object.rs:21 Jun 25 10:45:26 lemdit.com lemmy_server[27525]: 4: lemmy_server::root_span_builder::HTTP request Jun 25 10:45:26 lemdit.com lemmy_server[27525]: with http.method=GET http.scheme="https" http.host=lemdit.com http.target=/api/v3/resolve_object otel.kind="server" request_id=81e7a3b2-1760-4689-95d6-c7784713b7aa http.status_code=400 otel.status_code="OK" Jun 25 10:45:26 lemdit.com lemmy_server[27525]: at src/root_span_builder.rs:16 Jun 25 10:45:26 lemdit.com lemmy_server[27525]: 5: tokio::task::runtime.spawn Jun 25 10:45:26 lemdit.com lemmy_server[27525]: with kind=local task.name= task.id=191923 loc.file="/home/delendum/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.1.1/src/service.rs" loc.line=74 loc.col=17 Jun 25 10:45:26 lemdit.com lemmy_server[27525]: at /home/delendum/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.2/src/util/trace.rs:16 Jun 25 10:45:26 lemdit.com lemmy_server[27525]: 6: tokio::task::runtime.spawn Jun 25 10:45:26 lemdit.com lemmy_server[27525]: with kind=local task.name= task.id=30 loc.file="/home/delendum/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.1.1/src/worker.rs" loc.line=457 loc.col=25 Jun 25 10:45:26 lemdit.com lemmy_server[27525]: at /home/delendum/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.2/src/util/trace.rs:16 Jun 25 10:45:26 lemdit.com lemmy_server[27525]: 7: tokio::task::runtime.spawn Jun 25 10:45:26 lemdit.com lemmy_server[27525]: with kind=local task.name= task.id=28 loc.file="/home/delendum/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.1.1/src/worker.rs" loc.line=431 loc.col=21 Jun 25 10:45:26 lemdit.com lemmy_server[27525]: at /home/delendum/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.2/src/util/trace.rs:16 Jun 25 10:45:26 lemdit.com lemmy_server[27525]: 8: tokio::task::runtime.spawn Jun 25 10:45:26 lemdit.com lemmy_server[27525]: with kind=local task.name= task.id=27 loc.file="/home/delendum/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-rt-2.8.0/src/arbiter.rs" loc.line=316 loc.col=25 Jun 25 10:45:26 lemdit.com lemmy_server[27525]: at /home/delendum/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.2/src/util/trace.rs:16 Jun 25 10:45:26 lemdit.com lemmy_server[27525]: 9: tokio::task::runtime.spawn Jun 25 10:45:26 lemdit.com lemmy_server[27525]: with kind=block_on task.name= task.id=26 loc.file="/home/delendum/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-rt-2.8.0/src/arbiter.rs" loc.line=144 loc.col=24 Jun 25 10:45:26 lemdit.com lemmy_server[27525]: at /home/delendum/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.2/src/util/trace.rs:16 Jun 25 10:45:26 lemdit.com lemmy_server[27525]: LemmyError { message: Some("couldnt_find_object"), inner: duplicate key value violates unique constraint "community_language_community_id_language_id_key", context: SpanTrace [{ target: "lemmy_apub::objects::community", name: "from_json", file: "crates/apub/src/objects/community.rs", line: 126 }, { target: "lemmy_apub::fetcher::search", name: "from_json", file: "crates/apub/src/fetcher/search.rs", line: 137 }, { target: "lemmy_apub::fetcher::search", name: "search_query_to_object_id", file: "crates/apub/src/fetcher/search.rs", line: 19 }, { target: "lemmy_apub::api::resolve_object", name: "perform", fields: "\u{1b}[3mself\u{1b}[0m\u{1b}[2m=\u{1b}[0mResolveObject { q: "https://rblind.com/c/main", auth: Sensitive }", file: "crates/apub/src/api/resolve_object.rs", line: 21 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "\u{1b}[3mhttp.method\u{1b}[0m\u{1b}[2m=\u{1b}[0mGET \u{1b}[3mhttp.scheme\u{1b}[0m\u{1b}[2m=\u{1b}[0m"https" \u{1b}[3mhttp.host\u{1b}[0m\u{1b}[2m=\u{1b}[0mlemdit.com \u{1b}[3mhttp.target\u{1b}[0m\u{1b}[2m=\u{1b}[0m/api/v3/resolve_object \u{1b}[3motel.kind\u{1b}[0m\u{1b}[2m=\u{1b}[0m"server" \u{1b}[3mrequest_id\u{1b}[0m\u{1b}[2m=\u{1b}[0m81e7a3b2-1760-4689-95d6-c7784713b7aa \u{1b}[3mhttp.status_code\u{1b}[0m\u{1b}[2m=\u{1b}[0m400 \u{1b}[3motel.status_code\u{1b}[0m\u{1b}[2m=\u{1b}[0m"OK"", file: "src/root_span_builder.rs", line: 16 }, { target: "tokio::task", name: "runtime.spawn", fields: "kind=local task.name= task.id=191923 loc.file="/home/delendum/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.1.1/src/service.rs" loc.line=74 loc.col=17", file: "/home/delendum/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.2/src/util/trace.rs", line: 16 }, { target: "tokio::task", name: "runtime.spawn", fields: "kind=local task.name= task.id=30 loc.file="/home/delendum/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.1.1/src/worker.rs" loc.line=457 loc.col=25", file: "/home/delendum/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.2/src/util/trace.rs", line: 16 }, { target: "tokio::task", name: "runtime.spawn", fields: "kind=local task.name= task.id=28 loc.file="/home/delendum/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.1.1/src/worker.rs" loc.line=431 loc.col=21", file: "/home/delendum/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.2/src/util/trace.rs", line: 16 }, { target: "tokio::task", name: "runtime.spawn", fields: "kind=local task.name= task.id=27 loc.file="/home/delendum/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-rt-2.8.0/src/arbiter.rs" loc.line=316 loc.col=25", file: "/home/delendum/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.2/src/util/trace.rs", line: 16 }, { target: "tokio::task", name: "runtime.spawn", fields: "kind=block_on task.name= task.id=26 loc.file="/home/delendum/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-rt-2.8.0/src/arbiter.rs" loc.line=144 loc.col=24", file: "/home/delendum/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.2/src/util/trace.rs", line: 16 }] } Jun 25 10:45:26 lemdit.com lemmy_server[27525]: 2023-06-24T22:45:26.921711Z INFO actix_web::middleware::logger: 192.168.1.56 'GET /api/v3/resolve_object?q=https%3A%2F%2Frblind.com%2Fc%2Fmain&auth=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOjEsImlzcyI6ImxlbWRpdC5jb20iLCJpYXQiOjE2ODc2NDUxMjJ9.NUn48sBqxRp7dJqiMPM-Q-l3LjpUsHhsqotF88rxY30 HTTP/1.1' 400 31 'https://lemdit.com/search?q=https%3A%2F%2Frblind.com%2Fc%2Fmain&type=All&listingType=All&page=1&sort=TopAll' 'Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0' 1.834552

Version

BE: 0.18.0

Lemmy Instance URL

https://lemdit.com

delendum avatar Jun 24 '23 22:06 delendum

See also Issue #3296

RocketDerp avatar Jun 24 '23 23:06 RocketDerp

I'm seeing the same error thrown when ingesting a large amount of data via ActivityPub (no user interaction at all), for a post from a new community. The db insert of the community fails with

2023-06-29T16:17:16.296280Z  WARN HTTP request{http.method=POST http.scheme="http" 
http.host=lemmy.localhost:8536 http.target=/inbox otel.kind="server" 
request_id=5cdf0ff1-054b-47ba-8193-53ecca1be384}:receive:community: 
lemmy_apub::protocol::objects::page: could not dereference community 
http://reddit.com.localhost:5313/r/nba: duplicate key value violates
 unique constraint "community_language_community_id_language_id_key"
   0: lemmy_apub::objects::community::from_json
             at crates/apub/src/objects/community.rs:126
   1: lemmy_apub::activity_lists::community
             at crates/apub/src/activity_lists.rs:113
   2: lemmy_apub::activities::community::announce::receive
             at crates/apub/src/activities/community/announce.rs:46
   3: lemmy_server::root_span_builder::HTTP request
           with http.method=POST http.scheme="http" http.host=lemmy.localhost:8536 http.target=/inbox otel.kind="server" request_id=5cdf0ff1-054b-47ba-8193-53ecca1be384
             at src/root_span_builder.rs:16

and then

2023-06-29T16:17:16.629811Z  WARN Error encountered while processing the incoming HTTP request: lemmy_server::root_span_builder: No community found in cc

because the dereference fails.

(the first error is usually just silently ignored, i added code to make it visible)

phiresky avatar Jun 29 '23 16:06 phiresky

(the first error is usually just silently ignored, i added code to make it visible)

Please share!

RocketDerp avatar Jun 29 '23 16:06 RocketDerp

this is the patch:

diff --git a/crates/apub/src/protocol/objects/page.rs b/crates/apub/src/protocol/objects/page.rs
index ad89cf35b..6da9a42ec 100644
--- a/crates/apub/src/protocol/objects/page.rs
+++ b/crates/apub/src/protocol/objects/page.rs
@@ -204,11 +204,14 @@ impl InCommunity for Page {
         loop {
           if let Some(cid) = iter.next() {
             let cid = ObjectId::from(cid.clone());
-            if let Ok(c) = cid.dereference(context).await {
-              break c;
-            }
+            match cid.dereference(context).await {
+              Ok(c) => {break c;}
+              Err(e) => {
+                tracing::warn!("could not dereference community {cid}: {e}");
+              }
+            } 
           } else {
-            return Err(LemmyError::from_message("No community found in cc"));
+            return Err(LemmyError::from_message(&format!("No community found in cc. {:#?}", self)));
           }
         }
       }

But ignoring parts of those errors is intentional because the "cc" and "to" properties also contain the https://www.w3.org/ns/activitystreams#Public url which is not a community.

phiresky avatar Jun 29 '23 17:06 phiresky