cloudquery
cloudquery copied to clipboard
cloudquery
Who's using CloudQuery in production/dev ? :)
Hey everyone!
I created this GitHub issue so people can share with each other how/where they use CQ. This would be super helpful for new and current users to explore new workflows.
Thanks!
P.S - would love also to do a deeper dive on our blog as a guest post if anyone has a more complex use-case.
Edit from the future: Cloudquery changed their licensing and costs, I am no longer a Cloudquery user.
My Cloudquery deployment is currently at proof of concept stage, with a plan to move it to production in the next ~90 days. We are scanning 5 AWS accounts and one Datadog account currently with plans to add more. It is replacing a series of custom developed APIs that were doing fetching from the AWS API + storing resources in Redis.
The proof of concept is deployed using Aurora Serverless v1 Postgres for the databases and fetches in our k8s cluster but we are not using the helm chart. Instead, we use our own internal deployment tools and docker image, but that is just for "compliance" reasons rather than features missing from the helm chart. For production, we'll be using Aurora Serverless V2 and EKS with Fargate for fetching. One feature we've added around Cloudquery in our images is emitting metrics and events to Datadog for our own monitoring - maybe this could be exported via Prometheus metrics in the future?
One potentially interesting part of my deployment is that I am using Postgrest to supply a RESTful api on top of our Cloudquery database. Its able to offer a swagger.json file that I can use to automatically generate clients. I also experimented with a Graphql interface, but my users preferred the RESTful implementation.
My primary users are my fellow Infrastructure Engineers and our Security and Compliance teams. We are planning to build APIs on top of Cloudquery to provide information to our platform users in the future. I want to get my Security teams using cloudquery policies in conjunction with their existing tooling to help with Compliance scans.
We use CloudQuery in a production environment, deploying everything via terraform infrastructure as code to Google Cloud Platform.
Our deployment resides within a GCP environment and leverages GKE and Docker for computing. We utilize Cloud SQL for the PostgreSQL database piece and Grafana for the Data Analysis/Visualization.
Our deployment fetches resources from our multi-cloud environments nightly via a cron job and runs benchmark policy checks every so often. Policy results get stored in a Google Storage Bucket as objects that are synced to BigQuery dataset/tables and linked to Google Sheets for ease of reporting to key stakeholders and management.
Currently, we are fetching resources from 80+ AWS accounts and over 1000+ GCP projects, with plans to grow.
Our primary use case for CloudQuery is for assurance monitoring, compliance, and a backup CSPM. Primary consumers of CloudQuery data are the Security Operations Center, Cloud Security, and Data Governance teams.
Happy to offer a deeper dive into our deployment architecture if it will be beneficial to others interested in our use case.
Good luck CloudQuery'ing! I am looking forward to seeing the growth of the tool and the community!
@MichaelTay0 We are also looking into CloudQuery for GCP compliance, would love to get an architecture overview
We're starting to look into cloudquery. We think it has real potential for tracking and enforcing policies and migrations across a sprawling AWS estate. We've been playing around with it locally and are planning to deploy into EKS with an aurora serverless postgres db soon.
Our initial use cases will be for running policies against AWS resources, but I'm also excited to see plugins for Cloudflare, Github & Terraform, all of which we use extensively.
Longer term I'd really love to find a way to connect CQ with our Backstage deployment, it seems that could be done either by writing a Backstage plugin that sits calls into the CQ db, or as a CQ destination plugin that could write entities via the Backstage API.
@andrewthetechie I'd be interested to see what you've done for Datadog monitoring. We use DD as our primary monitoring tool for our EKS clusters, so I imagine we'll want to do something similar.
@andrewthetechie, are you able to generate swagger files for the resources defined in the AWS SDKs?
I do not understand how is it open source when everything is paid. It allows only 10M rows per month free. https://hub.cloudquery.io/plugins/source/cloudquery/gcp/v12.3.0/docs
@khanakia We have decided to move away from offering everything for free as a way to keep this project sustainable. All the reasons are explained in our founder's blog post. Nevertheless, the SDKs are staying open source so you can develop your own plugins and use CloudQuery for free this way. We will continue investing in the SDKs to help building a variety of source and destination plugins by different creators that may also choose to keep the plugins open source.
We are no longer using Cloudquery due to the pricing changes.
Please remove my customer story from cloudquery.io and remove Autodesk from your page.
I am also annoyed that you commercialized the open-source contributions I made to your aws plugin. Your MPL license allows it, but it still feels like a slimy thing to do to someone.
@andrewthetechie I understand your frustration, and I am sorry you feel that way. I have opened a PR on our internal repo for cloudquery.io to remove the references.
We are no longer using Cloudquery due to the pricing changes.
Please remove my customer story from cloudquery.io and remove Autodesk from your page.
I am also annoyed that you commercialized the open-source contributions I made to your aws plugin. Your MPL license allows it, but it still feels like a slimy thing to do to someone.
Im sorry for your experience. We are removing you story. As we mentioned in the blog Unfortunately we couldn't make cloudquery work as an open source / free product and make it a sustainable business. All the old code is still available in history for people to fork and maintain their own code. reference to AutoDesk is also removed. Feel free to ping me if you need any additional assistance.
Closing this here and moving case studies: https://docs.cloudquery.io/case-studies Thanks everyone for sharing their case study!