[receiver/githubactionsreceiver] feat: add new github actions receiver

[krzko]

Description:

Added a new receiver, githubactionsreceiver, to process GitHub Actions webhook events, transforming them into trace telemetry. This receiver handles workflow_job and workflow_run event payloads, converting each GitHub Action workflow or job, along with its steps, into trace spans.

This enables observation of workflow execution times, success, and failure rates. Additionally, it supports payload validation through a configurable secret to ensure data integrity.

Link to tracking Issue: 27460

Testing:

• Added unit tests for the githubactionsreceiver to validate the reception and processing of GitHub Actions webhook payloads into trace spans.
• Performed end-to-end testing with real webhook payloads to verify the trace outputs against expected telemetry for different GitHub Actions workflows.

Documentation:

• Included a detailed README.md within the githubactionsreceiver directory, explaining the purpose, configuration, and usage of the receiver.
• Provided example configurations for setting up the receiver.
• Documented the trace structure for observed GitHub Actions workflows and jobs, including the conversion logic for workflow steps into deterministic spans.

[linux-foundation-easycla[bot]]

The committers listed above are authorized under a signed CLA.

• :white_check_mark: login: krzko / name: Kristof Kowalski (db2db79edace4b54273fd7aef1af5a0ebfe1fc07, f691f0111a771f48e1645809c12b1a85c317834f, e2b46687abf992ec1cf5105a3554bfbd0fd4362a, a09296e89f27a46fb1f9ee15e5e7ec61250bd11d, 97ab9dad200372c4e28781b3241f144e4e64bd52, f2f5558de5d378cd01f1f9977d8d98728f024934, 5ca1a294c2db087946989b33adadcefda8232df7, ad0d4c299aac158aae9ede84892cc059d5324471, c97219ad9aecf5d289e8251f32d7adcb2a0fcc20, ecb66d9571bd1f417fa9a42441f16fbbac10beae, 631a48965e917c527d22ee3ed911091d4ae918cd, 8650ba1e4e705efcadc6806c5b11405c1cc404c3, e536601804a3e1eaadfaa6363fd99941dd02ac2e, 824a3d7167b8c9e9f4fa2cfbc87abbfa52eb0863, 248a8a85b5b0c4a590527a481a8bb384a4c0e9ab, 40684582c443b6cdb2c23cad8a6044612e680693, 54d014a576a3715c80f3c1c9b2149aa9e2bb55e0, 0acbb4ef66788a15b2cf116329c0eb2d9623a053, c72b7589395baa6e38f9c9b202bd79c938bc8bbd, 1022aeea0fd8b103e8e626fca740e0b232618dc6, 44b508cf0ab4206fa99a0cd32f95fc055c09c9fc, 00369ceb0df4bcc9aa97b1ac08a35ad567c2f8a5, 23f78433eaea7bdf99fdb0ee97ca08a9b725feeb, 8c95b0f9357f4346a91f916a5fd23e3a9e9992f6, 71ad0e7e701511051d84735937b7de50d4e526b5, ce12d83452c4ab865bb9142cee3721e6c3b8d598, 8a931db13267215ad29aec0cec042622a7e40d1f, d68c5735b5aaa8b87388d80876db76f90975fc8b, 551772b7c36755f71b6e93f058e07af57cf525e3, e3f1d915dd9b40f08355c59f0ce619bfad0e3447, dbbe9a1307b26c60c566c72730d14c1fb312e342, 1de564ba061c8566aca346a32f063565d38310cc, 39c3f5d512a1aebac6d70c811d0ad71dc5a77b87, 68931bdab17e9f73aaed67c1c5b34308ae51b17c, 444a675b8b7523b01dfdbbc089ed5f131bb175a4, 08b73cab507ee569674fd4c5fa375fb44bbb28bd, 7f9584b6a91a2fcffd1bf01df5381b3c29404841, 6db046ada5c5648d11ae48a1dd516d9818320984, 88a020baf96d2997d8da75460222b6252891aa56, 06f8bd565becea90ca75a6ac91ec2fd2daa09c1b, 28e2ca8551e9a0d7842e1164279abbc7a9c80ad1, 5c33dfbb5c356ec3da80abb4f4a24370b4e012c6, f612ec5b60a6330cc7a926493f563e64c1c02a10, 033cb55a8c6cc9780ee8cef69b9b292c747c1aa0, cf1eb0c0cbab556dfaccfb76f3006f20e266c19b, 64531bc50bb3c2c90f4fced93b26f4bb4c6108e2, 47b0900f4e922a6feacc9be4bb39ba15dfc9d150, 80d820f9e7c36448d6566c27c3012017a8859d88, 9ab19e6fccdb8244ef5c6a5b2d36ad6ef2ede3c2, b894d07c49c99daae6deea22bbd91eb3923322ff, f0fc1aaf52bf131cdd863434dafddc4479efbd40, f0110d5097a63b381078cbc417b66c3ab3874fa7, 4fd12c88b4409c8772f8bfe0b3743bd3faa6b1f1, 834a30d4a2b4c0047493e5448a14457f415bd406, 8c8d9c86b9a3b05f9988f1c70ce9b68578845afd, 5cedb3ced6002b934517d7b0fd0bf18ea8f2826a, 448b6a77c16ccec7f47d5fd78947a5d76c08a2a3, 7f5f37e39e67838a4b46d73246f1209e619867ed, 3f773b0ea21022b8596d54d3abfc90c30a08fe7a, 5339be78cb857568a2cec21a8b11e0cb7aad02d5, eb756ebfa0ea1094ebe57b52538e9d13da9a8834, f37256caadf5d9683a7e0e86b8a498bab472755c, efc8630355e05d76a076577cd1db85fc6420c427, 7e4494526f4944b1df6544da7be1d6dcd9fb4886, 2473d5963eeea0e23bb58dcfc560335c185e3e58, e36202666e9ee9bb96d770031883408b9b083f88, bc8191c40a9c52a10a20d316b3175b78b1d91424, c1fa24b7863cee79dd5ce4e53f888a13a3ed1d48, 67f1e79bf6930106fca3a43d507ffa9ed1aa030b, bf9c5c361a14ac21af2b4eff2da2c568c6ffc929, 0a465dffd223761d2ba4a6360044b7c05e06c249, c9878ff8b37231602557aeed3181b9784e81bfcc, b45f1739c29039a0314b9c97c3ccc578562df36f, f40fcd96108cd0e2cafd2a2c5b2a3de26adb7ce6, 90fadaea4a34a5c7ca69978812165613a8f0b9e8, 32747d0aa87d98e951e80c69df1c96adcc7f1a32, b8fd3f5606423e39d284093af2b65228db331c94, 76ea086078c1fd904a9c5b874960cf2f588cf48c, e8a759d173d24f37b918b7a6d86135a06861eef9, 82f62cefc29f6c161cdd71e00279cd4c2f72e78b, 404996b89684494d6d28effc795976b6a4869468, fb2dfc87f033107414d51bc7f6f14d49bf4a11c0, 19d28a340293e84381c169636e0ec44ce62e5a64, edc0f22d7deb6563a236daef5558f1e971a81c5d, 9c951a335eb74b537f8b180843d59b5f794d1c3f, fe37dea25f1fa6a7f6cd1f4552cee277c1f1eb9d, 24870c82e9624adfd9dc4b262c0f6c5a2b1d1af1, 172130b988d86d6a740fdca4a232bab979cfb110, 166091e9ca39eca2a3c419db47ea3adab0921828, 6f609a3b24faab8030698484dd2052264716b82c, d508604cd7786c287034d6ec0b46e9fcd032489c, 0d8e929b8a18b22e4255085bc527d06341201167, cd88f47634f52620f3ed1a156b786ee1c864b1f9, c8561ffb25896af71eae32dcb99665a97242b8e8, fe2ed063856ccfeaa3aa35136f63584092f50ca5, 683199e4af64f8616a9a965c0ad542af7f50d621, 850f575bb3593ea95def67e6bf13119d94e82f60, 74ccdf56bf2f9d82f50aecb67bdf4a6d86de15fd, b698228c35769a159ca66ddfe1fed7864d10b54d)

[atoulme]

Please secure a sponsor for this new receiver. Please see https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#adding-new-components and follow the steps outlined.

[krzko]

Thanks @atoulme. Working through getting a sponsor for this and seems we might be getting closer to one now.

[krzko]

To aid in testing of the component, you can find an ocb image via the below link:

https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/27460#issuecomment-1996422702

I've added a public image of this component via this repo https://github.com/krzko/otelcol-distributions, built using ocb.

Here's the direct link to the GHCR image - https://github.com/krzko/otelcol-distributions/pkgs/container/otelcol-distributions%2Fgithubactions

[github-actions[bot]]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[robwhitby]

@krzko thanks for working on this. I tried it out with ghcr.io/krzko/otelcol-distributions/githubactions:0.96.5 but I get this error:

json: cannot unmarshal object into Go struct field Repository.repository.license of type string

error    [email protected]/trace_processing.go:30    Failed to unmarshal job event    {"kind": "receiver", "name": "githubactions", "data_type": "traces", "error": "json: cannot unmarshal object into Go struct field Repository.repository.license of type string"}                                                                                                                            
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/githubactionsreceiver.(*jsonTracesUnmarshaler).UnmarshalTraces                                               
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/[email protected]/trace_processing.go:30

in the payload, repository.licence is an object:

"license": {
    "key": "other",
    "name": "Other",
    "spdx_id": "NOASSERTION",
    "url": null,
    "node_id": "MDc6TGljZW5zZTX="
}

whereas your model expects a string: https://github.com/krzko/opentelemetry-collector-contrib/blob/feat-add-githubactionseventreceiver/receiver/githubactionsreceiver/types.go#L270

Not sure why you don't import go-github? https://github.com/google/go-github/blob/master/github/actions_workflow_jobs.go

Thanks again!

[krzko]

Hey @robwhitby, thanks for the heads up. Seems you're using GHE, which I don't have access to test with.

When the original implementation was done, the go-github package didnt support all the types I needed, so opted to derive them from the webhook payloads.

I did the migration to go-github today, based off @Elfo404 initial effort, so it should support more of the webhook event types.

Just pushed v0.98.0, give that a try:

docker pull ghcr.io/krzko/otelcol-distributions/githubactions:0.98.0

[robwhitby]

Hi again @krzko

Thanks for updating the types, it's working now! (yep we're on GHE)

I'm now having trouble generating the correct span id in a workflow step. Firstly, it seems the docs don't match the code:

https://github.com/krzko/opentelemetry-collector-contrib/tree/feat-add-githubactionseventreceiver/receiver/githubactionsreceiver#generating-ids-in-bash

generate_span_id() {
  job_id=$1          
  run_attempt=$2
  step_name=$3
  step_number=$4 # optional
  input="${job_id}${run_attempt}${step_name}${step_number}"
  echo -n "$input" | openssl dgst -sha256 | sed 's/^.* //'
}

is different to the code: https://github.com/krzko/opentelemetry-collector-contrib/blob/feat-add-githubactionseventreceiver/receiver/githubactionsreceiver/trace_event_handling.go#L262

Even making the changes I cannot generate the same span id for a step from within a workflow - this is what i have after modifying the functions in the docs:

generate_trace_id() {
  run_id="${GITHUB_RUN_ID}"
  run_attempt="${GITHUB_RUN_ATTEMPT}"
  echo -n "${run_id}${run_attempt}t" | openssl dgst -sha256 | sed 's/^.* //' | cut -c-32
}

generate_span_id() {
  run_id="${GITHUB_RUN_ID}"
  run_attempt="${GITHUB_RUN_ATTEMPT}"
  job_name="${GITHUB_JOB}"
  step_name="build"          # have to hardcode this?
  step_number="4"            # how to know this?
  input="${run_id}${run_attempt}${job_name}${step_name}${step_number}"
  echo -n "$input" | openssl dgst -sha256 | sed 's/^.* //' | cut -c-16
}

Even hardcoding step_name and step_number I don't get the correct id. Have you got this working yourself? Maybe you could share an example of getting the right values from the github runner environment.

Thanks very much!

(also maybe this isn't the right place to report issues? let me know!)

[krzko]

Hey @robwhitby, with the step span you are trying to link, is the workflow type by any chance a resuable, composite type or a matrix strategy?

[robwhitby]

Hey @robwhitby, with the step span you are trying to link, is the workflow type by any chance a resuable, composite type or a matrix strategy?

it's just a normal workflow. I have made some more progress - this function correctly generates the span id for a step named build:

generate_span_id() {
  run_id="${GITHUB_RUN_ID}"
  run_attempt="${GITHUB_RUN_ATTEMPT}"
  job_name="${GITHUB_JOB}"
  step_name="build"
  step_number=""  
  input="${run_id}${run_attempt}${job_name}${step_name}${step_number}"
  echo -n "$input" | openssl dgst -sha256 | sed 's/^.* //' | cut -c17-32
}

the key is it should be chars 17-32! So now I just need to figure out where to get step_name and step_number from. Any ideas? Or maybe those variables could be replaced with ones that are present in the github env or context?

[krzko]

Hey @robwhitby, apologies, the spanID generation of the job span is incorrect. The method used, was taking in the jobID and not the runID. Will push a new collector image soon on top of updating the docs with the shell func to derive the IDs.

On the step details, sadly GitHub doesn't provide a way to get the step names during runtime, not via vars or context. They are include the step names and numbers, in the job payload when accessing via the API, but to derive that within the current step, is a bit of a challenge.

On top of this, the way GitHub uses JOB_NAME envs, is different to what the API returns. The API version is usually interpolated from a reusable or composite workflow, of via a matrix strategy. Rather confusing to say the least.

To compliment this component I also created https://github.com/krzko/run-with-telemetry, which will allow you to emit telemetry form with a run command. Just need to push the new collector image and it'll align again, thanks to the snafu.

This action will emit telemetry under the step span when run in a simple workflow, but when run in the a reusable, composite or matrix strategy, then it'll be found to the job span. In most o11y backends, it'll still be showing under the step span, due to the timestamp values.

There also https://github.com/krzko/setup-telemetry which generates a bunch of useful outputs such as traceparent, amongst others. Also https://github.com/krzko/export-job-telemetry if you just want to export the job telemetry with specific attributes for analysis, all to compliment the IDs that this component emits.

Just published docker pull ghcr.io/krzko/otelcol-distributions/githubactions:0.99.1 with the fix for the erroneous job span IDs, and also pushed an update to the readme about the correct format to generate those deterministic IDs.

[ritesh-singh]

Hey @krzko

I was trying to hook it with GHE private repo, by following below steps.

extensions:
  health_check:

receivers:
  githubactions:
    # secret: my-secret
    #custom_service_name: github-android-test // default org-repo-name
    # service_name_prefix: foo- // ignored if custom_service_name set
    # service_name_suffix: -bar // ignored if custom_service_name set
    cors:
      allowed_origins: ["*"]
      allowed_headers: ["*"]

    endpoint: 127.0.0.1:8080
    path: /webhook

processors:

exporters:
  debug:
    verbosity: detailed
  otlp:
    endpoint: 127.0.0.1:4317

service:
  pipelines:
    traces:
      receivers: [githubactions]
      processors: []
      exporters: [debug,otlp]

  extensions: [health_check]

  telemetry:
    logs:
      level: debug

And then running the docker container using below command

docker run -v $(pwd)/config.yaml:/etc/githubactions/config.yaml ghcr.io/krzko/otelcol-distributions/githubactions:0.99.1

After running a workflow, i am not able to see receiver logs on CLI. Very likely, i am missing something here. I can confirm, that webhook smee client is forwarded to localhost:8080/webhook and the delivery is successful when a workflow is initiated.

[krzko]

Hey @krzko

I was trying to hook it with GHE private repo, by following below steps.

Answered in https://cloud-native.slack.com/archives/C0598R66XAP/p1717915879464499

[github-actions[bot]]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[github-actions[bot]]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[adrielp]

@krzko - I think this is a bit much for one pr to grok. My guess (correct me) is that this was the original pr opened prior to finding a sponsor. Now that you have a sponsor, maybe this pr can be closed and a new pr opened with the initial skeleton of the receiver.

[github-actions[bot]]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[github-actions[bot]]

Closed as inactive. Feel free to reopen if this PR is still being worked on.