Add nf-co2footprint plugin for emissions tracking

[edmundmiller]

Summary

Enables CO2 footprint tracking during nf-test runs and uploads reports to S3 for analysis.

• Adds [email protected] plugin to test configuration
• Uploads all CO2 trace files to S3 after each test shard completes
• Organizes reports by date/branch/profile/shard for easy querying with DuckDB

Changes

Configuration

• tests/config/nf-test.config: Added nf-co2footprint plugin

CI/CD

• .github/workflows/nf-test.yml: Added S3 upload step for CO2 reports
  • Finds all co2footprint_trace*.txt files per shard
  • Uploads to s3://nf-core-resource-optimization/co2reports/modules/YYYY-MM-DD/branch-name/profile/shard/
  • Runs even if tests fail (uses if: always())

S3 Report Structure

Reports are organized for analysis:

s3://nf-core-resource-optimization/co2reports/modules/
  └── 2025-10-29/
      └── co2-footprint-tracking/
          └── docker/
              └── 1/
                  ├── co2footprint_trace.txt
                  └── co2footprint_trace_*.txt

Report Format

Each trace file contains TSV data with:

• task_id, status, name
• energy_consumption, CO2e, CO2e_market, carbon_intensity
• %cpu, memory, realtime, cpus
• powerdraw_cpu, cpu_model

Testing

• [ ] Verify plugin loads successfully
• [ ] Confirm CO2 reports are generated during test runs
• [ ] Check S3 uploads complete successfully
• [ ] Validate report structure for DuckDB compatibility

Refs: https://github.com/nf-core/modules/issues/9291

🤖 Generated with Claude Code

Co-Authored-By: Claude [email protected]

[mashehu]

I don't understand the use case for this. Is it to measure the general footprint of nf-core? We don't run modules test with any real data so we don't see any real estimates what's ach module actually needs. Will it not be a lot of small files, which probably have a larger footprint than what they actually measure on the long run?

[edmundmiller]

I don't understand the use case for this. Is it to measure the general footprint of nf-core?

Exactly!

This bucket will also house the reports for Megatests, and pipeline tests(which definitely add up)

Will it not be a lot of small files, which probably have a larger footprint than what they actually measure on the long run?

Maybe, but I don't think we'll know until we try. I don't know what the carbon footprint per line is but it's just a line per process.

https://github.com/nf-core/modules/actions/metrics/usage?dateRangeType=DATE_RANGE_TYPE_PREVIOUS_MONTH

Last month we had: 77,538 Total minutes across all workflows in this organization for last month 29,699 Total job runs

[mashehu]

But we are saving all the different entries without any clear use case. Not worth to have all the infrastructure for that imo.

[mashehu]

I am already saving and displaying all megatest footprints for the website https://github.com/nf-core/website/pull/3536

[mashehu]

actions minutes are not a good metric here imo, because we waste a lot of time on setup steps, which will not be covered by the plugin. Sorry for all the separate commentd, am on my phone while bringing a child to bed.