Website down due to "Invalid private key" (Capturing error for worker during execution of job: email-analytics-fetch-latest)

[jaller94]

Issue Summary

I had upgraded to 5.0.0 and then noticed that my website was down, even when ghost-cli reported the page as being running.

In journalctl I found the issue. See the logs below. In the logs I replaced the actual page name with foobar.

I fixed the issue by going to Mailgun, creating a new Sending Key and replacing the existing one in config.production.json.

I'm reporting this because I didn't expect an invalid Mailgun key to silently break the website.

Steps to Reproduce

I didn't test if this is reproducible. I don't have a non-production environment.

Ghost Version

5.0.0 and 5.0.1

Node.js Version

v16.15.0

How did you install Ghost?

Ubuntu 20.04

Database type

MySQL 8

Browser & OS version

No response

Relevant log / error output

Started Ghost systemd service for blog: foobar-de.
Love open source? We’re hiring Node.js Engineers to work on Ghost full-time.
https://careers.ghost.org/product-engineer-node-js
- Inspecting operating system
[2022-05-25 17:27:24] INFO Ghost is running in production...
[2022-05-25 17:27:24] INFO Your site is now available on https://foobar.de/
[2022-05-25 17:27:24] INFO Ctrl+C to shut down
[2022-05-25 17:27:24] INFO Ghost server started in 0.741s
[2022-05-25 17:27:24] INFO Bootstrap client was closed.
[2022-05-25 17:27:24] INFO Database is in a ready state.
[2022-05-25 17:27:24] INFO Ghost database ready in 1.043s
[2022-05-25 17:27:25] WARN Theme translations file locales/de.json not found.
[2022-05-25 17:27:25] WARN Theme translations falling back to locales/en.json.
[2022-05-25 17:27:26] INFO Stripe not configured - skipping migrations
[2022-05-25 17:27:27] INFO Ghost booted in 3.417s
[2022-05-25 17:27:27] INFO Adding offloaded job to the queue
[2022-05-25 17:27:27] INFO Scheduling job email-analytics-fetch-latest at 53 2/5 * * * *. Next run on: Wed May 25 2022 17:27:53 GMT+0000 (Coordinat>
[2022-05-25 17:27:27] INFO Adding offloaded job to the queue
[2022-05-25 17:27:27] INFO Scheduling job update-check at 6 13 9 * * *. Next run on: Thu May 26 2022 09:13:06 GMT+0000 (Coordinated Universal Time)
[2022-05-25 17:27:27] INFO Ghost URL Service Ready in 3.532s
[2022-05-25 17:27:27] INFO Bootstrap client was closed.
[2022-05-25 17:27:28] INFO "GET /ghost/api/admin/authentication/setup/" 200 723ms
[2022-05-25 17:27:53] INFO Worker for job "email-analytics-fetch-latest" online
[2022-05-25 17:27:54] INFO Capturing error for worker during execution of job: email-analytics-fetch-latest
[2022-05-25 17:27:54] ERROR Invalid private key

Invalid private key
----------------------------------------
Error: Invalid private key
    at IncomingMessage.<anonymous> (/var/www/foobar-ghost-prod/versions/5.0.1/node_modules/mailgun-js/lib/request.js:327:17)
    at IncomingMessage.emit (node:events:539:35)
    at endReadableNT (node:internal/streams/readable:1345:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)

[2022-05-25 17:27:54] INFO Capturing error for worker during execution of job: email-analytics-fetch-latest
[2022-05-25 17:27:54] ERROR Worker for job "email-analytics-fetch-latest" exited with code 1

Worker for job "email-analytics-fetch-latest" exited with code 1
----------------------------------------
Error: Worker for job "email-analytics-fetch-latest" exited with code 1
    at Worker.<anonymous> (/var/www/foobar-ghost-prod/versions/5.0.1/node_modules/bree/lib/index.js:419:40)
    at Worker.emit (node:events:527:28)
    at Worker.[kOnExit] (node:internal/worker:278:10)
    at Worker.<computed>.onexit (node:internal/worker:198:20)

Code of Conduct

• [X] I agree to be friendly and polite to people in this repository

[ErisDS]

Hey @jaller94, thanks for the report. We’ll look into improving the error handling in the job - it’s not expected that an error here is able to take the site offline.

Purely from curiosity, do you have any idea why the key became invalid?

[kevinansfield]

It sounds like you've configured Mailgun with a sending key rather than a private key. Sending keys won't work with the analytics service because they only allow you to send mail, the analytics service needs to be able to fetch the event logs which requires a private key.

I fixed the issue by going to Mailgun, creating a new Sending Key and replacing the existing one in config.production.json.

The analytics service doesn't use anything from `config.production.json. It sounds like maybe there was a different problem that was keeping your site offline?

[jaller94]

Every couple of months I have issues with the configuration of Mailgun in Ghost – where it seems to stop working all of a sudden. I likely changed the configuration, but I don't have a history of when which config was deployed.

[jaller94]

A private key? The Ghost documentation says to get an SMTP password from the Domain Settings. https://ghost.org/docs/config/#create-a-mailgun-account

EDIT: Ah, I though this was about the config.production.json. I now found the field for a Mailgun Private API key at https://jenniferpfalzgraf.de/ghost/#/settings/newsletters

[github-actions[bot]]

This issue is currently awaiting triage from @ErisDS. We're having a busy time right now, but we'll update this issue ASAP. If you have any more information to help us triage faster please leave us some comments. Thank you for understanding 🙂

[ErisDS]

Have you managed to get to the bottom of this?

[github-actions[bot]]

Note from our bot: The needs info label has been added to this issue. Updating your original issue with more details is great, but won't notify us, so please make sure you leave a comment so that we can see when you've updated us.

[github-actions[bot]]

Hey @jaller94 👋

Our team needed some more info to get to the bottom of this, however we've not heard back from you. We're going to close this for now, but let us know if you manage to dig up some more info and we'll reopen.

[jaller94]

I have not been able to reproduce the issue.