wordpress-activitypub icon indicating copy to clipboard operation
wordpress-activitypub copied to clipboard

Intermittent Downtime After Posting

Open DeadSuperHero opened this issue 7 months ago • 7 comments

Quick summary

We have about 2,000 or so ActivityPub followers on our site at We Distribute. Whenever we post an article, the site briefly goes down for a little while. I suspect this is happening because the server needs to dispatch posts to 2,000 people?

Steps to reproduce

  1. Migrate followers from a decent-sized Fediverse account to WordPress.
  2. Make sure Enable Mastodon Apps and Friends are also installed.
  3. Post an article. Alternatively, post a podcast episode with Seriously Simple Podcasting.
  4. Watch as your site briefly goes down.

Some things have helped mitigate various problems with ActivityPub and WordPress, such as using Surge, Redis, and Jetpack. Curiously, the site still goes down pretty frequently, which is not ideal.

What's weird is that most parts of the server are still fine. Nginx, Redis, PHP-FPM, and various applications all seem to run just fine. It's not clear why this happens, just that the WordPress part of the site becomes unreachable until about 20 minutes pass.

Are there best practices for scaling up a WordPress installation that heavily makes use of ActivityPub?

Site owner impact

More than 60% of the total website/platform users

Severity

Critical

What other impact(s) does this issue have?

No response

If a workaround is available, please outline it here.

No response

DeadSuperHero avatar Jun 15 '25 00:06 DeadSuperHero

For reference https://social.wedistribute.org/objects/1af46a7c-b798-4026-95be-7175f49b7acb

pfefferle avatar Jun 16 '25 13:06 pfefferle

Let's first check if WordPress is properly installed. Do you have the possibility to run the WP CLI in your WordPress root folder? If so, can you try to run wp core is-installed and wp core verify-checksums? (I had similar issues after a an auto-update went wrong, that's why we should rule that out)

Do you use Surge already?

Can you send us the output of the "Site Health - Info": /wp-admin/site-health.php?tab=debug (simply use the "Copy site info to clipboard" and paste it into a comment here).

pfefferle avatar Jun 16 '25 16:06 pfefferle

I've run the two WP-CLI tasks. First command produced no output, second command ran without issues.

I am using Surge already.

Here's my Site Health output:

### wp-core ###

version: 6.8.1
site_language: en_US
user_language: en_US
timezone: America/Phoenix
permalink: /%year%/%monthnum%/%postname%/
https_status: true
multisite: false
user_registration: 0
blog_public: 1
default_comment_status: open
environment_type: production
user_count: 22
dotorg_communication: true

### wp-paths-sizes ###

wordpress_path: /var/www/wordpress
wordpress_size: 1.99 GB (2134925464 bytes)
uploads_path: /var/www/wordpress/wp-content/uploads
uploads_size: 1.90 GB (2041193081 bytes)
themes_path: /var/www/wordpress/wp-content/themes
themes_size: 34.97 MB (36664890 bytes)
plugins_path: /var/www/wordpress/wp-content/plugins
plugins_size: 273.20 MB (286466997 bytes)
fonts_path: /var/www/wordpress/wp-content/uploads/fonts
fonts_size: directory not found
database_size: 301.27 MB (315899904 bytes)
total_size: 4.48 GB (4815150336 bytes)

### wp-dropins (2) ###

advanced-cache.php: true
object-cache.php: true

### wp-active-theme ###

name: Jannah Child (jannah-child-theme)
version: 1.0.2
author: TieLabs
author_website: https://tielabs.com
parent_theme: Jannah (jannah)
theme_features: core-block-patterns, widgets-block-editor, post-formats, automatic-feed-links, title-tag, html5, post-thumbnails, Arqam_Lite, wp-block-styles, align-wide, editor-style, menus, widgets
theme_path: /var/www/wordpress/wp-content/themes/jannah-child-theme
auto_update: Disabled

### wp-parent-theme ###

name: Jannah (jannah)
version: 7.0.6
author: TieLabs
author_website: https://tielabs.com/
theme_path: /var/www/wordpress/wp-content/themes/jannah
auto_update: Disabled

### wp-themes-inactive (2) ###

Twenty Twenty-Five: version: 1.2, author: the WordPress team, Auto-updates disabled
Twenty Twenty-Four: version: 1.3, author: the WordPress team, Auto-updates disabled

### wp-plugins-active (55) ###

ActivityPub: version: 6.0.2, author: Matthias Pfefferle & Automattic, Auto-updates enabled
Add Fediverse Icons to Jetpack: version: 0.5.0, author: Jan Boddez, Auto-updates enabled
Advanced Custom Fields: Color Swatches: version: 1.0.7, author: Nick Ford, Auto-updates enabled
Advanced Custom Fields PRO: version: 6.4.2, author: WP Engine, Auto-updates disabled
Antispam Bee: version: 2.11.7, author: pluginkollektiv, Auto-updates disabled
AudioIgniter: version: 2.0.1, author: The CSSIgniter Team, Auto-updates disabled
Cal.com: version: 1.0.0, author: Cal.com, Inc & Shycoder, Auto-updates enabled
Chart Block: version: 1.1.6, author: bPlugins, Auto-updates enabled
Co-Authors Plus: version: 3.6.6, author: Mohammad Jangda, Daniel Bachhuber, Automattic, Auto-updates enabled
Easy Charts: version: 1.2.4, author: Kiran Potphode, Auto-updates enabled
Easy Liveblogs: version: 2.3.7, author: Jeffrey van Rossum, Auto-updates enabled
Enable Mastodon Apps: author: (undefined), version: 1.4.6, Auto-updates enabled
Event Bridge for ActivityPub: version: 1.1.0, author: André Menrath, Auto-updates enabled
Fediverse Embeds: version: 1.5.5, author: stefanbohacek, Auto-updates enabled
Force Regenerate Thumbnails: version: 2.2.2, author: Exactly WWW, Auto-updates disabled
Friends: author: (undefined), version: 3.6.0, Auto-updates enabled
Friends Mastodon-like Interface: author: (undefined), version: 0.3, Auto-updates disabled
Friends Post Collection: author: (undefined), version: 1.2.4, Auto-updates disabled
host-meta: version: 1.3.2, author: Matthias Pfefferle, Auto-updates enabled
IndexNow: version: 1.0.3, author: Microsoft Bing, Auto-updates enabled
Jetpack: version: 14.7, author: Automattic, Auto-updates enabled
Jetpack Boost: version: 4.0.0, author: Automattic - Jetpack Site Speed team, Auto-updates enabled
Jetpack Protect: version: 4.1.0, author: Automattic - Jetpack Security team, Auto-updates disabled
Jetpack Search: version: 4.1.0, author: Automattic - Jetpack Search team (latest version: 5.0.0), Auto-updates disabled
Jetpack VaultPress Backup: version: 3.2, author: Automattic - Jetpack Backup team (latest version: 3.3), Auto-updates disabled
Masonry Image Gallery Block: version: 2.2.3, author: Zakaria Binsaifullah, Auto-updates enabled
Media Library Organizer: version: 1.6.5, author: Optimole, Auto-updates disabled
MSN Partner Hub: version: 2.8, author: Microsoft, Auto-updates enabled
NodeInfo: version: 2.3.1, author: Matthias Pfefferle, Auto-updates enabled
Nostr Verify: version: 1.2.0, author: Jeremy Herve, Auto-updates enabled
OG — Better Share on Social Media: version: 3.3.7, author: Marcin Pietrzak, Auto-updates enabled
OpenID Connect Generic: version: 3.10.0, author: daggerhart, Auto-updates disabled
Prismatic: version: 3.5, author: Jeff Starr, Auto-updates enabled
Public Post Preview: version: 3.0.1, author: Dominik Schilling, Auto-updates enabled
Redis Object Cache: version: 2.5.4, author: Till Krüss, Auto-updates disabled
Seriously Simple Podcasting: version: 3.10.3, author: Castos, Auto-updates enabled
Seriously Simple Speakers: version: 1.2.0, author: Castos, Auto-updates enabled
Seriously Simple Stats: version: 1.7.0, author: Castos, Auto-updates enabled
Seriously Simple Transcripts: version: 1.2.0, author: Castos, Auto-updates enabled
ShareOpenly: version: 1.1, author: David Artiss, Auto-updates enabled
Simple Mastodon Verification: version: 2.0.3, author: Jay McKinnon, Auto-updates enabled
SimpleTOC - Table of Contents Block: version: 6.7.3, author: Marc Tönsing, Auto-updates enabled
SMTP Mailer: version: 1.1.20, author: naa986, Auto-updates disabled
Social Post Embed: version: 2.0.1, author: David Artiss, Auto-updates disabled
Support For Icomoon with Advanced Custom Fields: version: 4.0.14, author: ViiVue, Auto-updates enabled
Surge: version: 1.1.0, author: Konstantin Kovshenin, Auto-updates disabled
The Events Calendar: version: 6.13.2.1, author: The Events Calendar, Auto-updates enabled
User Role Editor: version: 4.64.5, author: Vladimir Garagulya, Auto-updates enabled
Visual Link Preview: version: 2.2.7, author: Bootstrapped Ventures, Auto-updates enabled
WebFinger: version: 3.2.7, author: Matthias Pfefferle, Auto-updates enabled
Webmention: version: 5.4.1, author: Matthias Pfefferle, Auto-updates enabled
Web Stories: version: 1.42.0, author: Google, Auto-updates disabled
WordPress Sentry: version: 8.4.0, author: Alex Bouma, Auto-updates enabled
WP All Export: version: 1.4.11, author: Soflyy, Auto-updates disabled
Yoast SEO: version: 25.3, author: Team Yoast, Auto-updates enabled

### wp-plugins-inactive (5) ###

Contact Form 7: version: 6.0.6, author: Takayuki Miyoshi, Auto-updates disabled
Database Collation Fix: version: 1.2.10, author: Dave Jesch, Auto-updates disabled
FediPress: version: 1.0, author: Lívia Gouvêa, Auto-updates disabled
One User Avatar: version: 2.5.0, author: One Designs, Auto-updates disabled
Post My CF7 Form: version: 6.1.0, author: Aurovrata V., Auto-updates disabled

### wp-media ###

image_editor: WP_Image_Editor_Imagick
imagick_module_version: 1691
imagemagick_version: ImageMagick 6.9.11-60 Q16 x86_64 2021-01-25 https://imagemagick.org
imagick_version: 3.7.0
file_uploads: 1
post_max_size: 2000M
upload_max_filesize: 2000M
max_effective_size: 2 GB
max_file_uploads: 20
imagick_limits: 
	imagick::RESOURCETYPE_AREA: 122 MB
	imagick::RESOURCETYPE_DISK: 1073741824
	imagick::RESOURCETYPE_FILE: 768
	imagick::RESOURCETYPE_MAP: 512 MB
	imagick::RESOURCETYPE_MEMORY: 256 MB
	imagick::RESOURCETYPE_THREAD: 1
	imagick::RESOURCETYPE_TIME: 9.2233720368548E+18
imagemagick_file_formats: 3FR, 3G2, 3GP, AAI, AI, APNG, ART, ARW, AVI, AVIF, AVS, BGR, BGRA, BGRO, BIE, BMP, BMP2, BMP3, BRF, CAL, CALS, CANVAS, CAPTION, CIN, CIP, CLIP, CMYK, CMYKA, CR2, CR3, CRW, CUR, CUT, DATA, DCM, DCR, DCX, DDS, DFONT, DNG, DPX, DXT1, DXT5, EPDF, EPI, EPS, EPS2, EPS3, EPSF, EPSI, EPT, EPT2, EPT3, ERF, FAX, FILE, FITS, FRACTAL, FTP, FTS, G3, G4, GIF, GIF87, GRADIENT, GRAY, GRAYA, GROUP4, H, HALD, HDR, HEIC, HISTOGRAM, HRZ, HTM, HTML, HTTP, HTTPS, ICB, ICO, ICON, IIQ, INFO, INLINE, IPL, ISOBRL, ISOBRL6, J2C, J2K, JBG, JBIG, JNG, JNX, JP2, JPC, JPE, JPEG, JPG, JPM, JPS, JPT, JSON, K25, KDC, LABEL, M2V, M4V, MAC, MAGICK, MAP, MASK, MAT, MATTE, MEF, MIFF, MKV, MNG, MONO, MOV, MP4, MPC, MPG, MRW, MSL, MTV, MVG, NEF, NRW, NULL, ORF, OTB, OTF, PAL, PALM, PAM, PATTERN, PBM, PCD, PCDS, PCL, PCT, PCX, PDB, PDF, PDFA, PEF, PES, PFA, PFB, PFM, PGM, PGX, PICON, PICT, PIX, PJPEG, PLASMA, PNG, PNG00, PNG24, PNG32, PNG48, PNG64, PNG8, PNM, POCKETMOD, PPM, PREVIEW, PS, PS2, PS3, PSB, PSD, PTIF, PWP, RADIAL-GRADIENT, RAF, RAS, RAW, RGB, RGBA, RGBO, RGF, RLA, RLE, RMF, RW2, SCR, SCT, SFW, SGI, SHTML, SIX, SIXEL, SPARSE-COLOR, SR2, SRF, STEGANO, SUN, TEXT, TGA, THUMBNAIL, TIFF, TIFF64, TILE, TIM, TTC, TTF, TXT, UBRL, UBRL6, UIL, UYVY, VDA, VICAR, VID, VIDEO, VIFF, VIPS, VST, WBMP, WEBM, WEBP, WMV, WPG, X, X3F, XBM, XC, XCF, XPM, XPS, XV, XWD, YCbCr, YCbCrA, YUV
gd_version: 2.3.3
gd_formats: GIF, JPEG, PNG, WebP, BMP, AVIF, XPM
ghostscript_version: 9.55.0

### the-events-calendar ###


### wp-server ###

server_architecture: Linux 5.15.0-134-generic x86_64
httpd_software: nginx/1.18.0
php_version: 8.3.11 64bit
php_sapi: fpm-fcgi
max_input_variables: 1000
time_limit: 30
memory_limit: 256M
max_input_time: 60
upload_max_filesize: 2000M
php_post_max_size: 2000M
curl_version: 7.81.0 OpenSSL/3.0.2
suhosin: false
imagick_availability: true
pretty_permalinks: true
static_robotstxt_file: false
current: 2025-06-16T18:13:41+00:00
utc-time: Monday, 16-Jun-25 18:13:41 UTC
server-time: 2025-06-16T11:13:36-07:00

### wp-database ###

extension: mysqli
server_version: 10.6.22-MariaDB-0ubuntu0.22.04.1
client_version: mysqlnd 8.3.11
max_allowed_packet: 16777216
max_connections: 151

### wp-constants ###

WP_HOME: undefined
WP_SITEURL: undefined
WP_CONTENT_DIR: /var/www/wordpress/wp-content
WP_PLUGIN_DIR: /var/www/wordpress/wp-content/plugins
WP_MEMORY_LIMIT: 256M
WP_MAX_MEMORY_LIMIT: 256M
WP_DEBUG: false
WP_DEBUG_DISPLAY: true
WP_DEBUG_LOG: false
SCRIPT_DEBUG: false
WP_CACHE: true
CONCATENATE_SCRIPTS: undefined
COMPRESS_SCRIPTS: undefined
COMPRESS_CSS: undefined
WP_ENVIRONMENT_TYPE: undefined
WP_DEVELOPMENT_MODE: undefined
DB_CHARSET: utf8mb4
DB_COLLATE: undefined

### wp-filesystem ###

wordpress: writable
wp-content: writable
uploads: writable
plugins: writable
themes: writable
fonts: does not exist

### acf ###

version: 6.4.2
plugin_type: PRO
update_source: ACF Direct
activated: true
activated_url: https://wedistribute.org
license_type: Personal
license_status: active
subscription_expires: 1757187420
ui_field_groups: 6
php_field_groups: 0
json_field_groups: 0
rest_field_groups: 4
post_types_enabled: true
ui_post_types: 40
json_post_types: 0
ui_taxonomies: 24
json_taxonomies: 0
ui_options_pages_enabled: true
ui_options_pages: 0
json_options_pages: 0
php_options_pages: 0
rest_api_format: light
registered_acf_blocks: 0
blocks_per_api_version: 
blocks_per_acf_block_version: 
blocks_using_post_meta: 0
preload_blocks: true
admin_ui_enabled: true
field_type-modal_enabled: true
field_settings_tabs_enabled: false
shortcode_enabled: true
registered_acf_forms: 0
json_save_paths: 1
json_load_paths: 1

### jetpack ###

site_id: 235772966
ssl_cert: No
time_diff: undefined
version_option: 14.7:1749100484
old_version: 14.6:1746594825
public: Public
master_user: #25798931 deadsuperhero
is_offline_mode: off
is_offline_mode_constant: off
current_user: #25798931 deadsuperhero
tokens_set: Blog User
blog_token: 0CFuolFF$#DcxM30)(5I7wTy*YG)luZz
user_token: 8DaJc4i*)$Gi#oZgRXE%!8yX$Z^o3Gpb
version: 14.7
jp_plugin_dir: /var/www/wordpress/wp-content/plugins/jetpack/
plan: free
protect_header: {"trusted_header":"REMOTE_ADDR","segments":1,"reverse":false}
full_sync: {"start_action_processed":false,"cancelled_action_processed":true,"started":"Wed, 05 Mar 2025 21:35:18 +0000","finished":"Wed, 05 Mar 2025 21:35:28 +0000","progress":{"options":{"finished":true},"functions":{"finished":true},"constants":{"finished":true},"users":{"total":1,"sent":1,"finished":true,"last_sent":"25798937"},"network_options":{"finished":true}},"config":{"options":true,"functions":true,"constants":true,"users":[25798937],"network_options":true},"context":null}
sync_size: undefined
sync_lag: 0 seconds
full_sync_size: undefined
full_sync_lag: 0 seconds
idc_urls: {"home":"https:\/\/wedistribute.org","siteurl":"https:\/\/wedistribute.org","WP_HOME":"","WP_SITEURL":""}
idc_error_option: false
idc_optin: true
cxn_tests: All Pass.

### friends ###

version: 3.6.0
mbstring: Yes
roles: pending_friend_request, friend_request, subscription
main_user: #0 
parsers: SimplePie, Microformats, JSON Feed, ActivityPub

### event_bridge_for_activitypub ###

active_event_plugins: undefined

### web_stories ###

web_stories_version: 1.42.0
web_stories_db_version: 3.0.16
web_stories_amp_version: 2.5.6-alpha
web_stories_cdn_url: https://wp.stories.google/static/21
web_stories_dev_mode: false
web_stories_theme_support: Disabled
web_stories_enabled_experiments: No experiments enabled
web_stories_libxml_version: 2.9.14

### activitypub ###

actor_mode: blog
object_type: wordpress-post-format
post_template: [ap_excerpt]

[ap_permalink type="html"]
authorized_fetch: 0
vary_header: 1
shared_inbox: 0
ACTIVITYPUB_PLUGIN_VERSION: 6.0.2
ACTIVITYPUB_PLUGIN_DIR: /var/www/wordpress/wp-content/plugins/activitypub/
ACTIVITYPUB_PLUGIN_BASENAME: activitypub/activitypub.php
ACTIVITYPUB_PLUGIN_FILE: /var/www/wordpress/wp-content/plugins/activitypub/activitypub.php
ACTIVITYPUB_PLUGIN_URL: https://wedistribute.org/wp-content/plugins/activitypub/
ACTIVITYPUB_REST_NAMESPACE: activitypub/1.0
ACTIVITYPUB_EXCERPT_LENGTH: 400
ACTIVITYPUB_NOTE_LENGTH: 400
ACTIVITYPUB_MAX_IMAGE_ATTACHMENTS: 4
ACTIVITYPUB_HASHTAGS_REGEXP: (?:(?<=\s)|(?<=<p>)|(?<=<br>)|^)#([A-Za-z0-9_]+)(?:(?=\s|[[:punct:]]|$))
ACTIVITYPUB_USERNAME_REGEXP: (?:([A-Za-z0-9\._-]+)@((?:[A-Za-z0-9_-]+\.)+[A-Za-z]+))
ACTIVITYPUB_URL_REGEXP: (https?:|www\.)\S+[\w\/]
ACTIVITYPUB_CUSTOM_POST_CONTENT: [ap_title type="html"]

[ap_content]

[ap_hashtags]
ACTIVITYPUB_DISABLE_REWRITES: false
ACTIVITYPUB_DISABLE_INCOMING_INTERACTIONS: false
ACTIVITYPUB_DISABLE_OUTGOING_INTERACTIONS: false
ACTIVITYPUB_DEFAULT_OBJECT_TYPE: wordpress-post-format
ACTIVITYPUB_OUTBOX_PROCESSING_BATCH_SIZE: 100
ACTIVITYPUB_MASTODON_HTML_SANITIZER: Array
(
    [p] => Array
        (
        )

    [span] => Array
        (
            [class] => 1
        )

    [br] => Array
        (
        )

    [a] => Array
        (
            [href] => 1
            [rel] => 1
            [class] => 1
        )

    [del] => Array
        (
        )

    [pre] => Array
        (
        )

    [code] => Array
        (
        )

    [em] => Array
        (
        )

    [strong] => Array
        (
        )

    [b] => Array
        (
        )

    [i] => Array
        (
        )

    [u] => Array
        (
        )

    [ul] => Array
        (
        )

    [ol] => Array
        (
            [start] => 1
            [reversed] => 1
        )

    [li] => Array
        (
            [value] => 1
        )

    [blockquote] => Array
        (
        )

    [h1] => Array
        (
        )

    [h2] => Array
        (
        )

    [h3] => Array
        (
        )

    [h4] => Array
        (
        )

)

ACTIVITYPUB_DATE_TIME_RFC3339: Y-m-d\TH:i:s\Z
ACTIVITYPUB_ACTOR_MODE: actor
ACTIVITYPUB_BLOG_MODE: blog
ACTIVITYPUB_ACTOR_AND_BLOG_MODE: actor_blog
ACTIVITYPUB_CONTENT_VISIBILITY_PUBLIC: undefined
ACTIVITYPUB_CONTENT_VISIBILITY_QUIET_PUBLIC: quiet_public
ACTIVITYPUB_CONTENT_VISIBILITY_PRIVATE: private
ACTIVITYPUB_CONTENT_VISIBILITY_LOCAL: local

DeadSuperHero avatar Jun 16 '25 18:06 DeadSuperHero

Have you configured surge by hand or was it done by the plugin?

pfefferle avatar Jun 17 '25 13:06 pfefferle

@pfefferle I just used the defaults provided by the plugin. Should I configure by hand?

DeadSuperHero avatar Jun 17 '25 22:06 DeadSuperHero

@DeadSuperHero that is perfect! I wanted to get sure that the configuration was not accidentally added twice which could also cause issues.

pfefferle avatar Jun 18 '25 08:06 pfefferle

@DeadSuperHero have you had the chance to test the behaviour with a "normal" blogpost without an audio file?

Another solution might be to use a more compressed/smaller audio file for federation or to use a CDN for it? (but to go that path, it should be verified that this is the issue)

pfefferle avatar Jun 18 '25 08:06 pfefferle

@DeadSuperHero Have you had a chance yet to try out the suggestions above^?

obenland avatar Jun 26 '25 16:06 obenland

@DeadSuperHero Please feel free to re-open with any new insights. Closing this out for now.

obenland avatar Jul 02 '25 19:07 obenland

Apologies for the delay, just following up.

have you had the chance to test the behaviour with a "normal" blogpost without an audio file?

I just attempted to make a short microblog post through Enable Mastodon Apps. A few moments after posting, now I'm getting 504 Gateway Time-out.

So, I don't think it's explicitly tied to audio file sizes. This seems to be a problem with dispatching to an audience of 2,000+ followers. Are there any logs I should provide that might be useful for analysis?

DeadSuperHero avatar Jul 10 '25 18:07 DeadSuperHero

Added some additional context. Our server is using Nginx, which is somewhat non-standard compared to WordPress installs using Apache. Not really sure if it makes a big difference or not.

DeadSuperHero avatar Jul 10 '25 18:07 DeadSuperHero

Okay, digging into some of my error logs for PHP and FPM, before digging into Nginx.

WARNING: [pool www] server reached pm.max_children setting (5), consider raising it

I raised php-fpm's config and restarted the service. Will keep an eye out to see if the problem persists at all.

DeadSuperHero avatar Jul 10 '25 18:07 DeadSuperHero