[Bug] FeedFetcher-Google is requesting the same endpoint too frequently

[unixfox]

Describe the bug For some unknown reason, FeedFetcher-Google, the agent for pubsub is requesting the /feed/webhook too frequently. It will try to do the same POST request many times in a row:

::ffff:74.125.215.141 - - [13/Feb/2023:10:54:45 +0000] "POST /feed/webhook/v1:1676144239:6e776313:ea085ac8a352b07541a1d7b1e0a28a54ec3af34e HTTP/1.1" 200 20 "-" "FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)" - [connection keep-alive]\x0A[x-hub-signature sha1=9a8536e4b9affc8d5d966d44fe9dc16915b06f2f]\x0A[accept */*]\x0A[cache-control no-cache,max-age=0]\x0A[accept-encoding gzip, deflate, br]\x0A[pragma no-cache]\x0A[host yewtu.be]\x0A[content-length 932]\x0A[user-agent FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)]\x0A[link <https://www.youtube.com/xml/feeds/videos.xml?channel_id=UC2ped54WsnVpSuKUw7HikOg>; rel=self, <http://pubsubhubbub.appspot.com/>; rel=hub]\x0A[from googlebot(at)googlebot.com]\x0A[content-type application/atom+xml]\x0A
::ffff:74.125.215.141 - - [13/Feb/2023:10:54:47 +0000] "POST /feed/webhook/v1:1676144239:6e776313:ea085ac8a352b07541a1d7b1e0a28a54ec3af34e HTTP/1.1" 200 20 "-" "FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)" - [connection keep-alive]\x0A[x-hub-signature sha1=192495bf932ee54352aff0c3a0cc052e936f0809]\x0A[accept */*]\x0A[accept-encoding gzip, deflate, br]\x0A[content-type application/atom+xml]\x0A[content-length 923]\x0A[user-agent FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)]\x0A[from googlebot(at)googlebot.com]\x0A[host yewtu.be]\x0A[cache-control no-cache,max-age=0]\x0A[link <https://www.youtube.com/xml/feeds/videos.xml?channel_id=UC2ped54WsnVpSuKUw7HikOg>; rel=self, <http://pubsubhubbub.appspot.com/>; rel=hub]\x0A[pragma no-cache]\x0A
::ffff:74.125.215.145 - - [13/Feb/2023:10:54:49 +0000] "POST /feed/webhook/v1:1676144239:6e776313:ea085ac8a352b07541a1d7b1e0a28a54ec3af34e HTTP/1.1" 200 20 "-" "FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)" - [connection keep-alive]\x0A[x-hub-signature sha1=24f96908148a4d5096ccf285a81ba4c489600600]\x0A[accept */*]\x0A[accept-encoding gzip, deflate, br]\x0A[content-type application/atom+xml]\x0A[content-length 426]\x0A[user-agent FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)]\x0A[from googlebot(at)googlebot.com]\x0A[host yewtu.be]\x0A[cache-control no-cache,max-age=0]\x0A[link <https://www.youtube.com/xml/feeds/videos.xml?channel_id=UC2ped54WsnVpSuKUw7HikOg>; rel=self, <http://pubsubhubbub.appspot.com/>; rel=hub]\x0A[pragma no-cache]\x0A
::ffff:74.125.215.141 - - [13/Feb/2023:10:54:52 +0000] "POST /feed/webhook/v1:1676144239:6e776313:ea085ac8a352b07541a1d7b1e0a28a54ec3af34e HTTP/1.1" 200 20 "-" "FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)" - [connection keep-alive]\x0A[accept */*]\x0A[link <https://www.youtube.com/xml/feeds/videos.xml?channel_id=UC2ped54WsnVpSuKUw7HikOg>; rel=self, <http://pubsubhubbub.appspot.com/>; rel=hub]\x0A[accept-encoding gzip, deflate, br]\x0A[content-type application/atom+xml]\x0A[cache-control no-cache,max-age=0]\x0A[pragma no-cache]\x0A[user-agent FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)]\x0A[from googlebot(at)googlebot.com]\x0A[content-length 954]\x0A[host yewtu.be]\x0A[x-hub-signature sha1=0adcf41066b4fe05da0c888a6b85e2527c1e9897]\x0A
::ffff:74.125.215.145 - - [13/Feb/2023:10:54:54 +0000] "POST /feed/webhook/v1:1676144239:6e776313:ea085ac8a352b07541a1d7b1e0a28a54ec3af34e HTTP/1.1" 200 20 "-" "FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)" - [connection keep-alive]\x0A[accept */*]\x0A[link <https://www.youtube.com/xml/feeds/videos.xml?channel_id=UC2ped54WsnVpSuKUw7HikOg>; rel=self, <http://pubsubhubbub.appspot.com/>; rel=hub]\x0A[accept-encoding gzip, deflate, br]\x0A[content-type application/atom+xml]\x0A[cache-control no-cache,max-age=0]\x0A[pragma no-cache]\x0A[user-agent FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)]\x0A[from googlebot(at)googlebot.com]\x0A[content-length 967]\x0A[host yewtu.be]\x0A[x-hub-signature sha1=e238c1ebd3840d95ccc5bd0b0870a7f094728880]\x0A
::ffff:74.125.215.145 - - [13/Feb/2023:10:55:01 +0000] "POST /feed/webhook/v1:1676144239:6e776313:ea085ac8a352b07541a1d7b1e0a28a54ec3af34e HTTP/1.1" 200 20 "-" "FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)" - [connection keep-alive]\x0A[accept */*]\x0A[link <https://www.youtube.com/xml/feeds/videos.xml?channel_id=UC2ped54WsnVpSuKUw7HikOg>; rel=self, <http://pubsubhubbub.appspot.com/>; rel=hub]\x0A[user-agent FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)]\x0A[x-hub-signature sha1=91029035905bca631b7bec595f1295160fa92d88]\x0A[cache-control no-cache,max-age=0]\x0A[pragma no-cache]\x0A[accept-encoding gzip, deflate, br]\x0A[from googlebot(at)googlebot.com]\x0A[content-length 968]\x0A[host yewtu.be]\x0A[content-type application/atom+xml]\x0A
::ffff:74.125.215.145 - - [13/Feb/2023:10:55:03 +0000] "POST /feed/webhook/v1:1676144239:6e776313:ea085ac8a352b07541a1d7b1e0a28a54ec3af34e HTTP/1.1" 200 20 "-" "FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)" - [connection keep-alive]\x0A[accept */*]\x0A[link <https://www.youtube.com/xml/feeds/videos.xml?channel_id=UC2ped54WsnVpSuKUw7HikOg>; rel=self, <http://pubsubhubbub.appspot.com/>; rel=hub]\x0A[user-agent FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)]\x0A[x-hub-signature sha1=fb0d2e434e479da3086cfc54706141fbbf864161]\x0A[cache-control no-cache,max-age=0]\x0A[pragma no-cache]\x0A[accept-encoding gzip, deflate, br]\x0A[from googlebot(at)googlebot.com]\x0A[content-length 968]\x0A[host yewtu.be]\x0A[content-type application/atom+xml]\x0A
::ffff:74.125.215.145 - - [13/Feb/2023:10:55:15 +0000] "POST /feed/webhook/v1:1676144239:6e776313:ea085ac8a352b07541a1d7b1e0a28a54ec3af34e HTTP/1.1" 200 20 "-" "FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)" - [connection keep-alive]\x0A[link <https://www.youtube.com/xml/feeds/videos.xml?channel_id=UC2ped54WsnVpSuKUw7HikOg>; rel=self, <http://pubsubhubbub.appspot.com/>; rel=hub]\x0A[from googlebot(at)googlebot.com]\x0A[user-agent FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)]\x0A[content-type application/atom+xml]\x0A[host yewtu.be]\x0A[content-length 930]\x0A[accept-encoding gzip, deflate, br]\x0A[accept */*]\x0A[pragma no-cache]\x0A[x-hub-signature sha1=47b0d82d2003b3180412eaad0f5126875688069d]\x0A[cache-control no-cache,max-age=0]\x0A
::ffff:74.125.215.145 - - [13/Feb/2023:10:55:17 +0000] "POST /feed/webhook/v1:1676144239:6e776313:ea085ac8a352b07541a1d7b1e0a28a54ec3af34e HTTP/1.1" 200 20 "-" "FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)" - [content-type application/atom+xml]\x0A[x-hub-signature sha1=3c7c970366bb78f772175f004ee002fa78ce19fe]\x0A[user-agent FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)]\x0A[accept */*]\x0A[cache-control no-cache,max-age=0]\x0A[host yewtu.be]\x0A[content-length 961]\x0A[pragma no-cache]\x0A[accept-encoding gzip, deflate, br]\x0A[from googlebot(at)googlebot.com]\x0A[link <https://www.youtube.com/xml/feeds/videos.xml?channel_id=UC2ped54WsnVpSuKUw7HikOg>; rel=self, <http://pubsubhubbub.appspot.com/>; rel=hub]\x0A[connection keep-alive]\x0A
::ffff:74.125.215.145 - - [13/Feb/2023:10:55:38 +0000] "POST /feed/webhook/v1:1676144239:6e776313:ea085ac8a352b07541a1d7b1e0a28a54ec3af34e HTTP/1.1" 200 20 "-" "FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)" - [connection keep-alive]\x0A[from googlebot(at)googlebot.com]\x0A[link <https://www.youtube.com/xml/feeds/videos.xml?channel_id=UC2ped54WsnVpSuKUw7HikOg>; rel=self, <http://pubsubhubbub.appspot.com/>; rel=hub]\x0A[x-hub-signature sha1=9d9a9237e82f4280f336fd6541b7962db3fc3564]\x0A[content-length 983]\x0A[cache-control no-cache,max-age=0]\x0A[user-agent FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)]\x0A[pragma no-cache]\x0A[accept */*]\x0A[accept-encoding gzip, deflate, br]\x0A[host yewtu.be]\x0A[content-type application/atom+xml]\x0A
::ffff:74.125.215.145 - - [13/Feb/2023:10:55:43 +0000] "POST /feed/webhook/v1:1676144239:6e776313:ea085ac8a352b07541a1d7b1e0a28a54ec3af34e HTTP/1.1" 200 20 "-" "FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)" - [connection keep-alive]\x0A[from googlebot(at)googlebot.com]\x0A[link <https://www.youtube.com/xml/feeds/videos.xml?channel_id=UC2ped54WsnVpSuKUw7HikOg>; rel=self, <http://pubsubhubbub.appspot.com/>; rel=hub]\x0A[content-type application/atom+xml]\x0A[host yewtu.be]\x0A[x-hub-signature sha1=595f8c62715e1cbd465370d00fb2c7f6e92b9848]\x0A[content-length 951]\x0A[cache-control no-cache,max-age=0]\x0A[user-agent FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)]\x0A[pragma no-cache]\x0A[accept-encoding gzip, deflate, br]\x0A[accept */*]\x0A
::ffff:74.125.215.141 - - [13/Feb/2023:10:55:43 +0000] "POST /feed/webhook/v1:1676144239:6e776313:ea085ac8a352b07541a1d7b1e0a28a54ec3af34e HTTP/1.1" 200 20 "-" "FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)" - [connection keep-alive]\x0A[from googlebot(at)googlebot.com]\x0A[link <https://www.youtube.com/xml/feeds/videos.xml?channel_id=UC2ped54WsnVpSuKUw7HikOg>; rel=self, <http://pubsubhubbub.appspot.com/>; rel=hub]\x0A[content-type application/atom+xml]\x0A[host yewtu.be]\x0A[x-hub-signature sha1=dbaa063fd09d101ddaebf76b9d89011a9ac1e7c2]\x0A[content-length 996]\x0A[cache-control no-cache,max-age=0]\x0A[user-agent FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)]\x0A[pragma no-cache]\x0A[accept-encoding gzip, deflate, br]\x0A[accept */*]\x0A
::ffff:74.125.215.141 - - [13/Feb/2023:10:55:44 +0000] "POST /feed/webhook/v1:1676144239:6e776313:ea085ac8a352b07541a1d7b1e0a28a54ec3af34e HTTP/1.1" 200 20 "-" "FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)" - [connection keep-alive]\x0A[from googlebot(at)googlebot.com]\x0A[link <https://www.youtube.com/xml/feeds/videos.xml?channel_id=UC2ped54WsnVpSuKUw7HikOg>; rel=self, <http://pubsubhubbub.appspot.com/>; rel=hub]\x0A[content-type application/atom+xml]\x0A[host yewtu.be]\x0A[x-hub-signature sha1=ede9cf6874c534dc9bdd1c1aa94082eb76c39d1f]\x0A[content-length 989]\x0A[cache-control no-cache,max-age=0]\x0A[user-agent FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)]\x0A[pragma no-cache]\x0A[accept-encoding gzip, deflate, br]\x0A[accept */*]\x0A
::ffff:74.125.215.145 - - [13/Feb/2023:10:55:45 +0000] "POST /feed/webhook/v1:1676144239:6e776313:ea085ac8a352b07541a1d7b1e0a28a54ec3af34e HTTP/1.1" 200 20 "-" "FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)" - [connection keep-alive]\x0A[from googlebot(at)googlebot.com]\x0A[link <https://www.youtube.com/xml/feeds/videos.xml?channel_id=UC2ped54WsnVpSuKUw7HikOg>; rel=self, <http://pubsubhubbub.appspot.com/>; rel=hub]\x0A[content-type application/atom+xml]\x0A[host yewtu.be]\x0A[x-hub-signature sha1=96c40ebe7d113941f797304b732eff947e639106]\x0A[content-length 979]\x0A[cache-control no-cache,max-age=0]\x0A[user-agent FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)]\x0A[pragma no-cache]\x0A[accept-encoding gzip, deflate, br]\x0A[accept */*]\x0A
::ffff:74.125.215.141 - - [13/Feb/2023:10:55:48 +0000] "POST /feed/webhook/v1:1676144239:6e776313:ea085ac8a352b07541a1d7b1e0a28a54ec3af34e HTTP/1.1" 200 20 "-" "FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)" - [content-type application/atom+xml]\x0A[x-hub-signature sha1=f2e7cbd681d1feb1b085b850c71ecb99814dfe92]\x0A[accept */*]\x0A[host yewtu.be]\x0A[content-length 996]\x0A[connection keep-alive]\x0A[user-agent FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)]\x0A[from googlebot(at)googlebot.com]\x0A[cache-control no-cache,max-age=0]\x0A[accept-encoding gzip, deflate, br]\x0A[link <https://www.youtube.com/xml/feeds/videos.xml?channel_id=UC2ped54WsnVpSuKUw7HikOg>; rel=self, <http://pubsubhubbub.appspot.com/>; rel=hub]\x0A[pragma no-cache]\x0A

It's on my instance yewtu.be

I'm not sure why that is happening but it seems to create a lot of load due to that.

Steps to Reproduce N/A

Logs

Screenshots

Additional context

[unixfox]

@SamantazFox any ideas for this bug?

[SamantazFox]

I'm looking at the spec (Pubsubhubbub 0.4 specification) and the code right now, as it's something I not familiar with.

[SamantazFox]

Hmmm, I don't understand why it'd do that.

• It's unlikely to be caused by a feed that has been subscribed to multiple times, as there is a condition in the code to prevent that (though, it could happen if the DB isn't properly updating) and the protocol is designed to prevent malicious actors from forging requests in your name.
• That's definitely not due to the pubsubhubbub server retrying (as per section 7) because the payloads would be the same size.

Can you log the content of some of those requests?

[github-actions[bot]]

This issue has been automatically marked as stale and will be closed in 30 days because it has not had recent activity and is much likely outdated. If you think this issue is still relevant and applicable, you just have to post a comment and it will be unmarked.

[unixfox]

This is probably caused by too many invidious processes requesting the pubsub endpoint.

I have pointed out how to disable the concerning jobs here: https://github.com/iv-org/documentation/commit/2814c5ef472fefbb043436df808c25d3339d6a7c