cloudflare-gcp

cloudflare-gcp copied to clipboard

Currently the Cloud Function that is created writes the logs directly to BigQuery. This is very useful for being able to analyse the logs and visualise them using Data Studio.

One enhancement that could be made to the function would be to provide the ability (e.g. via an environment variable for example) to push the logs to Stackdriver as well, using the Stackdriver NodeJS SDK. This will provide the ability to search for logs in Stackdriver directly, which can be more cost-effective, especially when the log size increases (resulting in BigQuery analysis cost increase). By also setting the httpRequest field, the logs can show up in a more useful manner, e.g.:

Please let me know of your thoughts on this, and maybe we can help with the implementation.

@gtseres-workable thanks for this suggestion. If this can meaningfully reduce costs, it should be implemented no question. Can you provide any estimate on the cost difference? I think the cost of a BigQuery search vs. the cost of querying those same logs in Stackdriver over a 48 hour time period would provide a solid starting point

Taking costs out of the equation this would allow the use of strackdriver metrics to allow Cloudflare to be monitored for problems and have all logs in a single location.

@shagamemnon for example, we have 1Tb logs in BQ, I did couple requests to BQ with 8Tb data read, and price for that is 35$

The major challenge here: memory and time limits for Cloud Functions. In order to push every log into stackdriver, this function would need to decompress the batches of log files (e.g. gunzip), and then buffer each file into memory, or read them as a stream. Right now, this task is managed by BigQuery - which has built-in utilities to ingest the .gz files and transform them into rows.

I would certainly love to add this functionality. But right now, I don't see a path to doing so.