Automatic removal of toots does not work

[GatoOscuro]

Steps to reproduce the problem

Since the automatic toot deletion option is available, I have it set to a deletion every 1 week to preserve the principle of ephemerality within the platform. In addition, to further preserve privacy.

Expected behaviour

That publications will be removed after a week or at least two, even three.

Actual behaviour

He has not deleted anything for a month. I have tested both the current version of Mastodon social and mstdn.social with no effective results.

Detailed description

Since a month ago my posts that had been deleting themselves after a week due to "automatic toots deletion" settings have now stopped doing so, and I can't make any changes or get any information about the error.

Specifications

Mastodon v4.0.2 LibreWolf 109.0.1-2 Instance: mastodon.social / /mstdn.social/

[ClearlyClaire]

Hi! The auto-deletion feature throttles deletions to avoid putting a strain on the network, and avoids running when the server is busy to avoid putting strain on the particular server.

However, how the process determines when a server is “busy” was engineered and tested based on a small server, and it is very likely that they are not tuned for servers the scale of mastodon.social or mstdn.social, causing the deletion to run very rarely. If that is the issue, it may be addressed by #23320.

It is also possible there is an unrelated bug. That would be difficult to track down without involvement of those servers' admins though.

[GatoOscuro]

Hi! The auto-deletion feature throttles deletions to avoid putting a strain on the network, and avoids running when the server is busy to avoid putting strain on the particular server.

However, how the process determines when a server is “busy” was engineered and tested based on a small server, and it is very likely that they are not tuned for servers the scale of mastodon.social or mstdn.social, causing the deletion to run very rarely. If that is the issue, it may be addressed by #23320.

It is also possible there is an unrelated bug. That would be difficult to track down without involvement of those servers' admins though.

Thank you for your prompt reply, as far as I can see they have not agreed that such a feature meets its ultimate goal in the different scenarios, they focused only on its early stages.

Regards.

[biglifedecision]

I run a small instance (20-30 active users), and as far as I can tell Automated Post Deletion has never worked. From what I can understand, there have been improvements in how it works that should help larger servers in 4.1.2, but we are a small server, and so I doubt even if we were running a version closer to the most current version of Mastodon the problem would be fixed.

I would be happy to try troubleshooting to see if I can help track down if it is a bug. Admittedly we are running Hometown (v4.0.2+hometown-1.1.1), but I think any troubleshooting tips would still be helpful for anyone else looking at this.

How can I tell if any Automated Post Deletion has ever happened, or if it didn't, why? What tweaks could be done to try and trigger it to happen?

[ClearlyClaire]

The first thing to look at would be the size and latency of your sidekiq queues, including the “retry” queue. If those are too high, the automated post deletion feature will not run.

There is also a bug causing deletions to not spread out across accounts as deleted but instead will not delete any post of the users who enabled the feature last before it has finished deleting posts from the older accounts. This has been fixed in the development version, but not in any release yet.

There have also been multiple changes after 4.1.2 that make the process more aggressive and more efficient.

[biglifedecision]

@ClearlyClaire thank you for the reply.

our sidekiq queues always quiet and sleepy and empty. it's not a terribly busy server (i don't think).

i tried following some of your troubleshooting advice here: https://github.com/mastodon/mastodon/issues/20327

I am wondering if there is a way to check on the ruby console across all accounts which have enabled the feature which is the next post that should be deleted. then I could see if anything is getting deleted at all. or any other query i can do just to see if post deletion is happening, ever.

and since this is a big of general troubleshooting, can system memory ever affect whether post deletion will happen, either on our version, or in the newer version? i do know that are server runs a little hot usually at 66% memory usage.

[biglifedecision]

I have looked somewhat at the pull request for this feature, and also the changelog for the most recent update. i'm pretty confident that our server shouldn't be limited by any of those... limits that constrain it. the push and default queues are almost always empty, latency 0.

so i'm really looking at how i could troubleshoot this! could be a bug, but because i don't think i'm hitting any of the limits, i doubt that is the issue, so i think it could be something else. it seems that most users on our instance report that post deletion is not working.

[ClearlyClaire]

In a Rails console (RAILS_ENV=production bundle exec rails c), you can try the following:

AccountStatusesCleanupPolicy.where(enabled: true).find_each(order: :asc) do |policy|
  puts "policy #{policy.id}: last_inspected=#{policy.last_inspected}; next_status=#{policy.statuses_to_delete(1, nil, policy.last_inspected).pick(:id)}\n"
end

[biglifedecision]

Thank you! I tried poking around some more, at there do seem to be 14 users who have automated post deletion active, but we are still not seeing any post deletion as far as I can tell. My best guess is the server is running a little too low on memory and something is getting hung up when accounts_statuses_cleanup_scheduler runs? or it's related to the bug you mentioned:

There is also a bug causing deletions to not spread out across accounts as deleted but instead will not delete any post of the users who enabled the feature last before it has finished deleting posts from the older accounts. This has been fixed in the development version, but not in any release yet.

so maybe we just have to wait and hope that update helps, or try upgrading the server to more memory.

[ClearlyClaire]

It's unlikely that memory limitations would affect the account cleanup process, I think other things would break first.

What was the output of the command I wrote above? It should shed some light on whether the process is running at all, and if so, if it's stuck on one particular account.

[biglifedecision]

re: memory we didn't think so either, but noticing how bad the memory situation was it seemed worthwhile to upgrade and see if it changes anything. so far (overnight) doubling the memory doesn't seem to have helped with this issue.

opening a rails console with: RAILS_ENV=production bundle exec rails c

and running:

AccountStatusesCleanupPolicy.where(enabled: true).find_each(order: :asc) do |policy|
  puts "policy #{policy.id}: last_inspected=#{policy.last_inspected}; next_status=#{policy.statuses_to_delete(1, nil, policy.last_inspected).pick(:id)}\n"
end

output:

policy 1: last_inspected=; next_status=109553895141818451
policy 2: last_inspected=; next_status=
policy 3: last_inspected=; next_status=109632677260017881
policy 4: last_inspected=; next_status=
policy 5: last_inspected=; next_status=
policy 6: last_inspected=; next_status=109288274549095706
policy 7: last_inspected=; next_status=109287856419433476
policy 8: last_inspected=; next_status=109778986028996307
policy 9: last_inspected=; next_status=109553778427198481
policy 10: last_inspected=; next_status=107089259105843161
policy 11: last_inspected=; next_status=109514086074749171
policy 12: last_inspected=; next_status=
policy 13: last_inspected=; next_status=109259187044856049
policy 14: last_inspected=; next_status=108190363053718013

I think there was a nil line at the end, sorry didn't capture that. some DB queries for account_status_cleanup_policies i did confirmed we have 14 policies. happy to provide some other output from the rails console/DB queries! thank you for taking a look!

[ClearlyClaire]

It seems like it is not running at all, though I'm not sure why that is.