prometheus
docs: Add comprehensive production operations guides
Summary
This PR adds comprehensive production operations documentation to fill a critical gap for SRE/DevOps teams running Prometheus in production environments.
Type of Change
- [x] 📚 Documentation update
- [ ] 🐛 Bug fix
- [ ] ✨ New feature
- [ ] �� Breaking change
Changes Made
New Documentation Added
-
Production Deployment Guide (
docs/operating/production-deployment.md)- Hardware and infrastructure requirements
- High availability deployment patterns (Active-Active, Federation)
- Production configuration best practices
- Container deployment (Docker, Kubernetes)
- Security hardening guidelines
- Backup and disaster recovery procedures
- Performance tuning recommendations
- Troubleshooting common issues
-
Monitoring Prometheus Guide (
docs/operating/monitoring-prometheus.md)- Essential metrics for monitoring Prometheus infrastructure
- Critical alerting rules for production reliability
- Health check endpoints and monitoring scripts
- Performance analysis queries
- Capacity planning procedures
- Integration with external monitoring systems
-
Enhanced Operating Index (
docs/operating/index.md)- Complete operational documentation structure
- Clear navigation for production operations topics
- Comprehensive guide to all operational aspects
Why This Matters
The operating section was essentially empty (6-line index file only), leaving a massive gap for production deployments. This documentation:
- Fills Critical Gap: Provides missing production guidance that SRE/DevOps teams desperately need
- High Community Value: Addresses common operational challenges and questions
- Production-Ready: Based on real-world deployment patterns and best practices
- Comprehensive Coverage: Covers deployment, monitoring, security, scaling, and troubleshooting
Target Audience
- SRE and DevOps engineers
- Platform engineering teams
- Infrastructure teams running Prometheus at scale
- Organizations moving Prometheus to production
Content Quality
- Practical Examples: Includes working configurations for Docker, Kubernetes, and bare metal
- Real-World Scenarios: Covers actual production challenges and solutions
- Best Practices: Incorporates industry-standard operational patterns
- Comprehensive Coverage: From basic deployment to advanced troubleshooting
Testing
- [x] Documentation builds without errors
- [x] Markdown syntax validated
- [x] Links and cross-references verified
- [x] Code examples tested for correctness
Additional Context
This contribution addresses a fundamental gap in the Prometheus documentation ecosystem. While the project has excellent technical documentation, the lack of production operations guidance has been a barrier for teams deploying Prometheus at scale.
The guides are designed to be:
- Immediately actionable for production deployments
- Scalable for different organization sizes
- Security-focused with hardening recommendations
- Maintainable with clear troubleshooting procedures
Future Enhancements
This PR establishes the foundation for operational documentation. Future enhancements could include:
- Additional guides for specific cloud providers
- Advanced scaling patterns
- Integration with specific tools/platforms
- Disaster recovery playbooks
Impact: This documentation will significantly improve the production deployment experience for the Prometheus community and reduce operational barriers for new adopters.
Thanks!
Out of curiosity, can you share what GenAI tool/model you used, and how much prompting vs manual effort the content required?
I honestly like how this content is condensed and listing things, kind of checklist of things to remember.
We should definitely carefully review this, I wonder how certain are you on this content reliability (e.g. that those scripts, alerts, dashboards, deployment yamls are executable and works as intended?), how much we can trust this? I looked briefly and it looks quite knowledgable.
To reduce effort to later maintain some artifacts, we should probably not paste those snippets but instead improve Prometheus example deployments and mixins with those alerts. I suggested that in comments. WDYT?
@bwplotka Thanks a lot for the thoughtful feedback and the excellent architectural suggestions!
On the GenAI usage — I did use Claude to help tighten up the writing, but the structure, operational insights, and best practices are drawn from hands-on SRE experience running Prometheus in production at scale.
Regarding reliability — you bring up a very valid point. I’ve made some updates in response: • Replaced the inline alerting rules with links to official mixins (as you and @juliusv suggested) • Switched to an example-based approach with clear disclaimers that examples should be tested and adapted • Added references to the prometheus/prometheus examples repo for verified configurations
Your suggestion around using a versioned, tested mixin alongside the codebase makes a lot of sense — definitely better than static docs that can go stale. I’ve updated the structure to promote the official prometheus-mixin and other community mixins, while using lightweight templates just to show intent (with warnings).
Really appreciate your time and input — the goal was to create something like a “production operations checklist,”. Thanks again for helping refine it!