Email notification upon replies

[TWiStErRob]

Usually commenting solutions allow for users to be notified via email about replies to their comments, or in case of a flat commenting stream when anyone posts a new comment. Is it possible to do this with staticman?

[eduardoboucas]

This is currently in development and will be released soon!

[illus0r]

@eduardoboucas looking forward to it! I've just discovered, I've received a first comment on my blog. This comment is useful, but I'm not able to say “Thank you” to it's author.

[eduardoboucas]

The feature is pretty much ready. I could use with some help with testing. Would any of you be interested in being the guinea pig? 😄

[illus0r]

@eduardoboucas ready!

[eduardoboucas]

Okay, here's how to get notifications working.

Emails are sent using Mailgun. There is an account associated with the public instance of Staticman, but it comes with a limit of 10,000 emails a month. This limit shouldn't be an issue in the immediate future, but users are encouraged to use their own Mailgun accounts instead. They are free (for the first 10k emails), but you do need a custom domain to use them with. I can possibly provide *.staticman.net subdomains.

To enable notifications, you need a notifications block in your site config and you must be using the config file format introduced in v2 (i.e. having a staticman.yml file, instead of using Jekyll's _config.yml). See the sample config file for more info.

If you're using your own Mailgun account, you'll need to add a Mailgun API key and domain to the config, which need to be encrypted. To encrypt those, you can use the following endpoint: https://api.staticman.net/v2/encrypt/{TEXT TO BE ENCRYPTED}.

Next, your entries will need to specify that a user wishes to subscribe to a specific entry. This is an example:

<form method="POST" action="https://api.staticman.net/v2/entry/joebloggs/my-site/master/comments">
 <input type="hidden" name="options[origin]" value="http://mysite.com/post1.html">
 <input type="hidden" name="options[parent]" value="867bd1e0-921c-11e6-930f-79eeedf443ea">
 
 <input type="text" name="fields[name]" placeholder="Title">
 <input type="text" name="fields[email]" placeholder="Email">
 <textarea name="fields[body]"></textarea>
 <input type="checkbox" name="options[subscribe]" value="email">
 
 <button type="submit">Go!</button>
</form>

In the from above, you can see the following:

• options[origin] contains the full URL of the entry. This will be included in the email sent to subscribers, allowing them to open the post directly.
• options[parent] is a unique identifier to the entry the user is subscribing to. This could be the title of a post (although it needs to be URL-friendly, so this needs to be a slug) or the _id field (which is now automatically added to every entry) if you want to subscribe to another comment.
• options[subscribe] contains the name of the field corresponding to the email address of the subscriber.

Finally, and to make sure we don't send emails asking users to click on links that have been spoofed, you need to add an allowedOrigins field (see docs) to your site config, where you'll specify which domains are valid origins. If you add only mysite.com, then an entry where options[origin] is http://myothersite.com/post1.html will be rejected.

I'm sure there are still a few rough edges, so if any of you can try implementing this and share your feedback that'd be very useful.

Thanks!

[dancwilliams]

Works great! Thanks for your work on this!

[zongren]

Is this working on public staticman instance? I tried it a couple of times and failed. Here is my staticman.yml:

allowedFields: ["name", "email", "url", "message", "origin", "parent", "subscribe"]
requiredFields: ["name", "message"]
format: "json"
generatedFields:
  date:
    type: "date"
    options:
      format: "iso8601"
moderation: true
branch: "master"
path: _data/{options.slug}
filename: comment-{@timestamp}
transforms:
  email: md5
name: "宗仁的博客"
allowedOrigins: ["localhost", "zongren.me"]
notifications: 
  enabled: true
  apiKey: "qPB1uWby7FS6T0wgLqUfjcQkkVtLkcergXOqiEoZyTo5yqMmGU/cuzOD825KOZkvbE7m0mOYo2LKPj82v+BcQDxxcIULev8lwpQ1KZJwjv6Ei3f1HbFyIq5N2Ehmya3PyPGga3IaedFVTPFrue67DQ2W5+tu8xJX1S2PahUEgAA="
  domain: "FkFQQFNFX96xVIOYvgtB8IxeBP60lZa/sUhHv0Y3KWV3EdRjMLED0zZr6nGGC5opytzczKvQtR2Y6YJvKQ2ltOBV1aFuAvsN/HPnOZ4e5JMBI+BjGWWlaqUsKp/mNGq/q9oWDk3FT8tdfw7UBqa8lC99eYa+QMXj/k+gpPx5ki0="

and comment form:

<form class="comment-form" method="POST" action="https://api.staticman.net/v2/entry/<%- theme.staticman %>">
    <input type="hidden" name="options[origin]" value="<%- config.url %><%- config.root %><%- item.path %>">
    <input type="hidden" name="options[parent]" value="<%- item.slug %>">
    <input type="hidden" name="options[redirect]" value="<%- config.url %><%- config.root %><%- item.path %>">
    <input type="hidden" name="options[slug]" value="<%- item.slug %>">
    <span class="comment-form__input-wrapper">
      <input autocomplete="off" spellcheck="false" class="comment-form__input" name="fields[name]" type="text" placeholder="请输入姓名"/>
    </span>
    <span class="comment-form__input-wrapper">
      <input autocomplete="off" spellcheck="false" class="comment-form__input" name="fields[email]" type="email" placeholder="请输入邮箱（可选）"/>
    </span>
    <span class="comment-form__input-wrapper">
      <input autocomplete="off" spellcheck="false" class="comment-form__input" name="fields[url]" type="url" placeholder="请输入你的网站（可选）"/>
    </span>
    <span class="comment-form__input-wrapper">
    <textarea class="comment-form__input comment-form__input--textarea" name="fields[message]" placeholder="请输入评论内容（支持Markdown）"></textarea>
    </span>
    <input type="checkbox" name="options[subscribe]" value="email" id="subscribe" /><label for="subscribe">订阅此文章的评论</label>
    <button class="comment-form__input comment-form__input--button">提交评论</button>
  </form>

And there are no logs on my mailgun dashboard.

[eduardoboucas]

@zongren link to the repo, please?

[zongren]

@eduardoboucas the repository is at gitlab

[zongren]

And I used encrypted "zongren.me" as notifications.domain.

[wes-brooks]

Hi Eduardo! Any chance you could check the logs of the staticman service to help me understand why my attempts to get reply notifications aren't working? I think I've configured staticman properly and have a working mailgun account, but I must be doing something wrong.

The repo is wrbrooks.github.io. Thanks in advance! -W

[eduardoboucas]

@wrbrooks I don't see any errors in the logs. Can you link to the page you're using to submit entries?

[justinrummel]

Hello @eduardoboucas I'm having issues as well w/ Staticman triggering emails with Mailgun. I'm able to validate with Mailgun's example curl statement but cannot trigger anything from my primary test page: https://www.justinrummel.com/macworld-2010-pictures/

(API Key removed)

justinrummel@Rummel-MBPr ~> curl -s --user 'api:key-12341234123412341234123412341234' \
                                https://api.mailgun.net/v3/mg.justinrummel.com/messages \
                                -F from='Excited User <[email protected]>' \
                                -F [email protected] \
                                -F subject='Hello' \
                                -F text='Testing some Mailgun awesomness!'
{
  "id": "<[email protected]>",
  "message": "Queued. Thank you."
justinrummel@Rummel-MBPr ~>

[eduardoboucas]

@justinrummel Staticman uses the Mailing Lists functionality from Mailgun, where each thread/post is created as a mailing list. If you go to your Mailgun account and navigate to Mailing Lists, do you see any lists created? If so, do you see the email addresses corresponding to the users that have subscribed to that thread?

[justinrummel]

I see one mailing list that I think was created from my API tests, but not from the site.

[chuckmasterson]

I've been planning how to integrate email notifications onto my blog. I have my comment fields defined somewhat idiosyncratically, so the author's email is authoremail instead of just email. I suppose that probably wouldn't work, since I haven't found where to set the name of the field that Staticman looks for... do you know of a way to make it work, or will I just need to sed all my authoremails into emails?

Also, I saw in your reply to #72 (which brilliantly solved a problem I was having, thanks) that the notifications feature isn't considered fully stable yet. Do you have a sense of how likely it is that an existing staticman notification configuration would continue working as the project continues toward stability?

[eduardoboucas]

@chuckmasterson You can specify the name of the field that contains the email address. It's the value of options[subscribe]. In your case, that would be:

<input type="hidden" name="options[subscribe]" value="authoremail">

Regarding the stability of the email notification system, I have to admit I've been struggling to find enough time to dedicate to it. In any case, I think the existing configuration format is flexible enough to accommodate any changes in the near future, so I don't foresee any breaking changes.

[chuckmasterson]

Completely understand about having time. If I knew the first thing about Node.js I'd try contributing, but alas.

I tried setting up with the advice you gave and got as far as getting some mailing lists created. But nothing triggered sending to these lists. I'll look at it again soon... if you find the time to read this and reply, a couple questions that might help me out are:

• How does Staticman create the "Alias Address" in Mailgun for each mailing list? Mine looks like a random series of hex digits, but it seems like it should correspond to the _id of some post, or at least something that I see in the YAML of one of the comments.
• How does Staticman determine what (if any) Alias Address to send notification emails to? I'm not quite wrapping my head around how this is achieved with just the options[parent] field. Does it look at all the files in the post's comment directory and see if any have an _id that matches options[parent]? If so, how does it work if you give it a slug as option[parent], since you aren't telling it whether it's identifying the page or the parent's _id? And how does it make the leap from there to knowing where to send the email?

[chuckmasterson]

After writing that, I took a crack at reading the code anyhow, and I think I may have understood enough to make a suggestion that would get us partway to a solution, though not all the way. (I would write this as a pull request but my understanding of Node.js is so weak that I would almost certainly break more than I'd fix.)

Suggestion:

Use a different variable besides options.parent to control subscriptions, perhaps a new one named options.subscribeTo.

Reasoning:

Though I haven't found anywhere you gave explicit directions on how to use options.parent, @mmistakes is already using it in threaded commenting, and a few other people have copied his strategy (I'm one). This strategy relies on options.parent being written into the YAML comment file as _parent. In this strategy _parent (options.parent) is the _id of a child comment's parent.

From what I understand of the Staticman code, Staticman expects options.parent to be some identifier of a post that the user is currently subscribing to. This conflicts with the other usage. If a user is writing a top-level comment on a post, then that comment's _parent will be null, and hence even if the user clicks the "subscribe me" checkbox, no mailing list is created because this condition doesn't get satisfied:

if (subscriptions && options.parent && options.subscribe && this.fields[options.subscribe]) {
  subscriptions.set(options.parent, this.fields[options.subscribe]).catch(err => {
    console.log(err.stack || err)
  })
}

If I follow the logic aright, the only subscriptions that would work with Staticman as it stands are where a user makes a child comment and clicks "subscribe me" - in which case they would get subscribed to all replies to their comment's parent. No one can subscribe to replies to their own comment, only to replies to some top-level comment that already exists.

Although my and mmistakes' use of options.parent here is apparently nonstandard, I think there's reason to favor it over what's currently in the code:

• It's already employed in several sites.
• The name parent describes a child comment's parent more intuitively than it describes "what comment the user is currently subscribing to".
• I don't think you'd be breaking any currently functioning sites. It seems like all the Jekyll pioneers here who have tried implementing Mailgun notifications have gotten stuck at about the same stage I have, and the only one I'm aware of who apparently has it working is mmistakes. (I've left a comment on his site to try to see if notification works in the strange way I've predicted, but it's in moderation right now.) The sites of @justinrummel and @zongren currently have no "subscribe" options and @wrbrooks 's does but I can't test it because no comments are going through at all for me (hey wrbrooks, did you know that?).

I'm not sure what format would make sense for the hypothetical options.subscribeTo as I still don't fully understand how Staticman currently interprets options.parent.

I'm still confused on one point: you mentioned in #76 that options fields aren't written to the comment file, but options.parent sure is getting written. I'm happy with that, but it is also a bit unusual; maybe it would make sense for it to go into the documentation eventually that "options fields generally don't get written to the entry, with the exception of options.parent, which is prefixed with an underscore (_parent) to denote the difference."

I'm happy to be a guinea pig on this feature as I'd definitely like to have it, but I'm also going to be on extended travels with little to no coding starting April 1, so I hope we can all get it figured out by then.

And thanks for Staticman! I hope this helped and wasn't too full of misunderstandings of how the code works.

[mmistakes]

@chuckmasterson I just merged in your comment. My understanding is you should get a notification email once a new comment is merged in. I'm going to do that now and make it off the main thread (not a child of your comment to see if your hypothesis is true).

My site takes about 20 minutes to build as it crunches through a bazillion images, so give it that long to see if anything shows up.

[eduardoboucas]

Thanks for the suggestions, @chuckmasterson. Let me try to answer some of the questions.

How does Staticman create the "Alias Address" in Mailgun for each mailing list? Mine looks like a random series of hex digits, but it seems like it should correspond to the _id of some post, or at least something that I see in the YAML of one of the comments.

Each mailing list has a corresponding email address. This email address is a MD5 hash of the GitHub username, repository and entry id concatenated together. This happens here.

How does Staticman determine what (if any) Alias Address to send notification emails to?

Whenever we process an entry, we take its id, generate the hash described above and check if there's any mailing list with a matching address. If there is, we fire notifications to all the addresses in it. Otherwise, there's nothing to do.

From what I understand of the Staticman code, Staticman expects options.parent to be some identifier of a post that the user is currently subscribing to. This conflicts with the other usage.

My thinking was that options[parent] describes the "thread" that you want to subscribe to. If you're writing a top-level comment as you mentioned, this parent would be an identifier of the post you're commenting on (it could be a title, slug, or anything else that uniquely identifies the entry). If you're commenting on an existing comment, then the parent would be the id of the comment.

Theoretically, it could go on how many levels necessary, but as @mmistakes will be able to tell you, it's probably not a good idea to nest further than 2 levels because otherwise the site generation process becomes too complex.

So the general idea is: options[parent] can be whatever — a post title, slug or a Staticman entry id. All it does is create a mailing list with subscribers for that particular parent, which we use whenever we want to process that entry.

Does this help?

[eduardoboucas]

And sorry if this has caused confusion and made you have to guess what Staticman does. My intention was to document this whenever I had the chance to make it fully stable, but I understand that people want to use this as soon as it's available, so from now on I'll make sure to document things as they get deployed to the live API.

[eduardoboucas]

Also:

I'm still confused on one point: you mentioned in #76 that options fields aren't written to the comment file, but options.parent sure is getting written. I'm happy with that, but it is also a bit unusual; maybe it would make sense for it to go into the documentation eventually that "options fields generally don't get written to the entry, with the exception of options.parent, which is prefixed with an underscore (_parent) to denote the difference."

This is very true. This should be in the documentation.

[chuckmasterson]

Ahhh, it's becoming clearer. @eduardoboucas, thanks for all the answers.

@mmistakes, your comment has been processed but I didn't get a notification. I think I was right, and if I'm right about being right, we're both stuck until either Eduardo makes the change I suggested or we change our commenting mechanism so it uses options[parent] how Eduardo intended.

In your comment you said that "my understanding is options[parent] are options[origin] are two different things, the later used to track which thread you’re currently subscribed to." I don't think options[origin] is used for this at all, and in fact I haven't been paying any attention to that field. It appears to me that it's only used for (1) preventing spoofing attacks and (2) displaying a link in the processed email.

Sounds like Eduardo is saying that options[parent] is indeed what's used to keep track of the thread (Mailgun mailing list). In SubscriptionsManager.js this happens:

const compoundId = md5(`${this.parameters.username}-${this.parameters.repository}-${entryId}`)

...which would boil down to, for instance, md5("chuckmasterson-chuckmasterson.github.io-[somePostIdentifier]"). That "somePostIdentifier" looks like it's supplied by options[parent].

If not for the condition check (does options.parent exist?) that I quoted earlier, my top-level comment on MadeMistakes would've resulted in a mailing list created with a name of whatever results from md5'ing "mmistakes-made-mistakes-jekyll-", since the entryId (options[parent], blank for a top-level comment) would have been null. Since there is that check, I'm guessing if you looked in your Mailgun mailing lists you'd find that no list with my address in it has been created (it's a mail.chuckmasterson.com address).

So... if I'm right, which of us should change?

[mmistakes]

That makes sense @chuckmasterson. Because I'm using options[parent] for nested comments it will only fire reply emails on child comments. The main thread sends null for that variable so no email notifications.

I'll probably change the language on my site so it's clear that you're only subscribing to "replies made on your comment" and not to the entire thread. Which to be honest is probably fine. I'm less concerned about email notifications than I am with stopping the barrage of comment spam and resulting Staticman PR's I delete daily :wink:

Also I'm not using my own MailGun account. That's probably why I'm one of the few who have email notifications working. I'm just piggy backing on the public instance of Staticman since I never had much luck configuring it.

[zburgermeiszter]

@mmistakes you might want to have a look at the brand new reCAPTCHA feature to stop the spam.

[eduardoboucas]

If I understand correctly, the only thing missing in @mmistakes's implementation is to send a post identifier on options[parent] for the main thread.

@mmistakes I think I already asked you this, but have you tried a honey pot field? I found it to be quite effective in stoping spam on my own site. If that hasn't worked, feel free to open a separate issue so we can find a solution. I'm also more concerned with that than with email notifications.

[mmistakes]

@zburgermeiszter Gave it a go this morning but trying to POST to dev.staticman.net was giving me errors. I had the same problem when I switched to the v2 API and had to jump through some hoops to remove/add Staticman as a collaborator.

[zburgermeiszter]

@mmistakes Can you please report this the issue #20, with your staticman.yml please, to see what could be wrong?

[chuckmasterson]

@mmistakes People would be subscribing to "other replies made to the comment you're replying to", if that's alright with you.

@eduardoboucas That would work for Michael except that in his comments Liquid (which I got familiar with while I was adapting it), top-level comments are identified as such by how they're missing a _parent field. I suppose we could check whether _parent equals the post slug instead of whether it's null. That seems a little fragile though.