infra Migrate fastly log processor to pluto

This makes the service that ingest raw fastly logs and generates summaries (see https://github.com/NixOS/infra/tree/master/metrics/fastly) to pluto. Currently it's running on one of my home machines.

This service runs once a week and kicks off some AWS Athena queries. It expects the credentials for the fastly-log-processor AWS user in /home/nix-metrics/.aws/config. That user has access to the nixos-metrics, nixos-athena and fastly logs buckets.

Feb 22 '24 20:02 edolstra

This seems to have stopped working:

$ AWS_PROFILE=fastly-log-processor ./ingest-raw-logs.sh 2025-02-24 2025-03-03

An error occurred (InvalidRequestException) when calling the StartQueryExecution operation: Unable to verify/create output bucket nixos-athena

May 17 '25 22:05 edolstra

We'll need to create an IAM user with a policy that has sufficient access. I can help with that. I dont currently see a user named fastly-log-processor defined in this PR.

Note that it's not enough to have just a bucket policy. The IAM user policy must also allow bucket access. It worked for the eelco.dolstra user as that user has admin policy and the Union of the user policy and the bucket policy allows access

We could maybe also consider moving this to a small EC2 instance as then we can use an IAM role instead of user. Which means we don't have to think about credentials rotation. Though not super string opinion in that. It's just nice not to have to think about credentials.

May 18 '25 08:05 arianvp