New post notifications sent to subscribers for categories they aren't subscribed to when using Newsletter Categories

[mgozdis]

Impacted plugin

Jetpack

Quick summary

When using Newsletter Categories and selecting a single category for users to subscribe to, subscribers still receive new post notifications for any new post created on the site, regardless of the category. It's unknown if this affects only when setting a single category or not, but only a single category was tested as that is what user had setup in this interaction: 7686351-zd-a8c

A workaround is to specifically set the newsletter setting on a post to Post only on categories you don't want emails sent out for, but this is tedious and an extra step.

Steps to reproduce

1. Enable Newsletter Categories under Settings > Newsletters and select a single category
2. Add a Subscribe Block somewhere on the site.
3. View the site and submit your email address in the form to subscribe and confirm subscription.
4. View subscription and see you are subscribed to the category
5. Create a new post that is NOT in the newsletter category and publish it
6. Check your email and see you received a new post notification

A clear and concise description of what you expected to happen.

Subscribers should only receive emails for new post notifications in categories they are subscribed to as the feature states: Newsletter categories allow visitors to subscribe only to specific topics. When enabled, only posts published under the categories selected below will be emailed to your subscribers.

What actually happened

Users receive emails for new post notifications regardless of the category they are subscribed to.

Impact

Most (> 50%)

Available workarounds?

Yes, easy to implement

Platform (Simple and/or Atomic)

Atomic

Logs or notes

Tested and reproducible on an AT test site only. Simple sites not tested yet.

[github-actions[bot]]

Support References

This comment is automatically generated. Please do not edit it.

• [ ] 7686351-zen
• [ ] 7686536-zen
• [ ] 8639985-zen
• [ ] 9059308-zen

[TimBroddin]

Hey @mgozdis

Can you offer some clarification on step 3?

View the site and submit your email address in the form to subscribe and confirm subscription.

Could you let me know if you selected the single category in this step? Or did you leave the default "All categories" checked? You will receive new post emails for any category in the latter case. The newsletter category feature does not limit the sending of emails to only the categories selected. It will still email new posts outside the selected newsletter categories to subscribers who have not opted into any newsletter category. We should look into rewording the documentation is this causes any confusion.

If you did select the single category, this is a bug, and we'd need to further look into it.

[mgozdis]

@TimBroddin With a single category set for newsletter subscriptions, it never presents a modal to the user to select the categories to subscribe to but appears to only subscribe to that category. Subscriptions were showing like this with only the single category showing as being subscribed to, but emails were being received for all post notifications still:

[TimBroddin]

This is strange. Even with 1 newsletter category enabled, the modal should show:

I took a look at the site, but I couldn't find a subscribe block there. Is the customer ok with us creating a test page and putting one there?

--

Regardless, I think there's a disconnect between what the customer expects this feature to do and what it actually does. I see they created a Featured Blog category, and I can imagine they expect only posts in this category to go out. However, that's not how the newsletter category feature works. By default (when selecting All Content), all posts, regardless of category, will be emailed.

[mgozdis]

I tested on an AT test site the other day and could reproduce. The modal never showed but subscribed me to only that category while still sending notifications for all. It should be fine to create a test page on their site.

[TimBroddin]

I just tested locally on my JT site and on an AT site and in both cases, I was presented with the modal:

I feel like the copy that's being shown on an individual subscriber's detail could use some clarification, stating that they'll receive emails from the selected newsletter categories and all content published outside of the site owner's selected newsletter categories. But I agree that it's very confusing.

I'm going to create a page on the site of the customer now, and see if I can replicate why it doesn't show the modal.

[simison]

@crisbusquets on copy suggestion above.

[TimBroddin]

@mgozdis The Jetpack SSO module is disabled on their site, so I can't really do much with SU. I'm a bit hesitant to enable it, as it could break some of their plugins.

Would you mind asking them to put the subscribe block on a page?

I tested the URL of the modal flow for their site, and when called it showed me the option to select the Newsletter Category:

[renata-franco]

I got another report here: 7686536-zen.

[mgozdis]

@TimBroddin They have a subscribe block on the left side of this page: https://nashvillesevereweather.com/who-we-are/

I tested again now on their site and my test site and could no longer reproduce the modal not displaying. It's now displaying, but Jetpack has also updated since the report.

On my test site, I tested subscribing only to the single category, leaving "All Content" unchecked and still receiving post notifications for all new posts, regardless of the category.

The copy in the settings when enabling Newsletter categories states subscribers will only receive posts published under the categories:

[TimBroddin]

On their site the modal now also displays for me. I wonder what caused it not to display before 🤔

On my test site, I tested subscribing only to the single category, leaving "All Content" unchecked and still receiving post notifications for all new posts, regardless of the category.

That's actually the intended behavior. There's discussion underway to change the inner workings of newsletter categories here, which would make it behave like expected: p1707294603033769-slack-C02TCEHP3HA

[crisbusquets]

stating that they'll receive emails from the selected newsletter categories and all content published outside of the site owner's selected newsletter categories

@crisbusquets on copy suggestion above.

@simison @TimBroddin what about something more generic? Like Interests, that doesn't convey that they're only subscribed to this. In the future, we could use this information to add filters to the Subscribers Page.

We can also add a tooltip to the right of Interestto explain what it is:

• This subscriber gets emails from this category and everything else not labeled as a newsletter category.

[mgozdis]

Just noting that another potential workaround for users with AT sites that want to restrict new post notifications to only their Newsletter categories is to add a code snippet like the following. Can this hook be implemented when using Newsletter categories by default? Seems like that would be an easy fix for making things work the way we state it works in the settings and docs.

function jetpackme_exclude_all_posts_subscriptions_except( $categories ) {
    $categories = array( 'cat1', 'cat2', 'cat3');
    return $categories;
}
add_filter( 'jetpack_subscriptions_exclude_all_categories_except', 'jetpackme_exclude_all_posts_subscriptions_except' );

[lsl]

Reprioritizing normal as there doesn't seem to be many user reports and this doesn't seem to have a large impact on ux - confusing maybe, but not particularly broken.

[mrfoxtalbot]

We have a new report of this in 8639985-zd-a8c

I did some detailed testing and this affects simple sites, AT sites and self-hosted sites.

I am unsure about the correct priority or impact here but this is certainly a bug. Moreover, it is confusing for HEs (that means a lot more time wasted trying to figure out what's causing it) Example in p1724394063884449-slack-C03TY6J1A

[crisbusquets]

cc @Automattic/zap ☝️

[heavyweight]

I can no longer reproduce this issue. If I have "Travel" checked for my subscription then I will receive email from posts in the travel category.

Otherwise I don't get any email. Can anyone try to reproduce this?

[mrfoxtalbot]

Thank you @heavyweight. Is that simple or atomic?

[heavyweight]

I was testing on atomic

[obenland]

I was able to reproduce this issue. @mashikag Could someone from Apex take a look at this?

[mashikag]

I was able to reproduce this issue. @mashikag Could someone from Apex take a look at this?

Yes, someone from @Automattic/apex will take it up as soon as they're available. We’ll add an additional comment here to confirm once it’s been picked up.

[mashikag]

Just an update that no one on the team had a chance to look at the issue yet, but I am planning to look into the issue tomorrow.

[crisbusquets]

Just an update that no one on the team had a chance to look at the issue yet, but I am planning to look into the issue tomorrow.

Thanks for the update, @mashikag! Let me know if you need me to review anything :)

[mashikag]

Due to unforeseen capacity constraints within the team, I wasn’t able to get to the issue today. Please bear with us as we work through current tasks. We will get back to this one as soon as possible. 🙇 cc @Automattic/apex

[ivan-ottinger]

Hey folks 👋🏼🙂 I have looked into this issue today and these are my findings (tested with Simple and Atomic).

Test one: following the steps from the issue description

Site's Newsletter category settings

Subscriber's Newsletter category settings

Post published in a "non-apple" category that hasn't been selected in the site's Newsletter category settings (e.g. orange)

Result: → Email is sent to the subscriber.

Basically, when a post is published with a category that hasn't been selected in the site's Newsletter category settings, it will go to every subscriber.

This is expected behavior. More context can be found in @TimBroddin's message above: https://github.com/Automattic/jetpack/issues/35431#issuecomment-1932078470).

---

Test two: experimenting with two different categories

Site's Newsletter category settings

Subscriber's Newsletter category settings:

Scenario A

Post published in the apple category

Result: → Email is sent to the subscriber.

Scenario B

Post published in the orange category

Result: → Email is not sent to the subscriber.

---

When reviewing the last user's report (8639985-zd-a8c) and their site, I believe there's a confusion in how the Newsletter categories feature works (similarly to what was discussed above: https://github.com/Automattic/jetpack/issues/35431#issuecomment-1932078470).

The user in the last ticket says the following:

I've setup Jetpack newsletter on only send emails for a selected category "Articles." But when I make a new post, each one is still set to email subscribers. I need to be able to post in the other categories without emailing people.

This tells me that they expect the newsletters will be sent only for categories selected in the site's Newsletter settings.

But this is not how the feature works. The newsletters are sent to all subscribers regardless of their Newsletter settings - if that specific post category hasn't been selected in the site's Newsletter settings (in other words, we are hitting the Test one case explained above).

If the user doesn't want certain posts to be sent out as a newsletter, they may use the following option when publishing their post instead:

---

Summary

There doesn’t appear to be a bug at the moment. The main issue seems to be a mismatch between user expectations and how the feature functions.

To improve user experience and limit confusion, we could do the following:

• short-term: consider copy changes like @crisbusquets suggested above (https://github.com/Automattic/jetpack/issues/35431#issuecomment-1933621446).
• long-term: explore changing how Newsletter categories work internally: p1707294603033769-slack-C02TCEHP3HA

What do you think?

Of course, if you found some bug I haven't observed, please do share.

[alinclamba]

The user who reported this initially 9059308-zd-a8c asking for an update. I am going to explain them the misunderstanding as described by @ivan-ottinger above, but I also added a line in our docs here:

Emails sent for non-selected categories: posts published in categories not explicitly excluded in the site's Newsletter settings will still be emailed to all subscribers.

This should help, but if you have any suggestions, please let me know.

[ivan-ottinger]

That sounds good to me. 👍🏼 Thanks, Alin!

[bizanimesh]

Sorry, I mistakenly closed this - the primary issue looks different. Reopened it!

[ilonagl]

Should this still be open? Or is it resolved already?

[ivan-ottinger]

Should this still be open? Or is it resolved already?

Hey @ilonagl 👋🏼 🙂

I wouldn't consider the issue outlined in this thread as a bug, but if there's a bandwidth, we could consider one of these approaches to improve user experience:

To improve user experience and limit confusion, we could do the following:

• short-term: consider copy changes like @crisbusquets suggested above (New post notifications sent to subscribers for categories they aren't subscribed to when using Newsletter Categories #35431 (comment)).
• long-term: explore changing how Newsletter categories work internally: p1707294603033769-slack-C02TCEHP3HA