taskcluster
taskcluster copied to clipboard
taskcluster
CI at Scale
I’ve been noticing this for some time (months), but it’s difficult to isolate or pin down. I’ve seen it when loading individual tasks, task graphs, and even when loading URLs...
A concrete example: ``` [taskcluster 2024-12-19T00:33:49.820Z] Worker Type (translations-1/b-linux-v100-gpu-d2g-4) settings: [taskcluster 2024-12-19T00:33:49.820Z] { [taskcluster 2024-12-19T00:33:49.820Z] "config": { [taskcluster 2024-12-19T00:33:49.820Z] "deploymentId": "" [taskcluster 2024-12-19T00:33:49.820Z] }, [taskcluster 2024-12-19T00:33:49.820Z] "generic-worker": { [taskcluster 2024-12-19T00:33:49.820Z]...
https://community-tc.services.mozilla.com/worker-manager/proj-git-cinnabar%2Fwindows/workers This morning after worker images upgrade no workers were being created. They stayed in requested state for a long period of time. By looking in DB directly it looked...
[This task](https://stage.taskcluster.nonprod.cloudops.mozgcp.net/tasks/Ujo9jB_GS52cICfFAVTL1Q/definition) was successfully submitted and was scheduled to run. Notable details of its definition include: ``` created: '2024-12-12T19:27:20.756Z' deadline: '2024-12-13T19:27:20.756Z' expires: '2025-03-12T19:27:20.756Z' payload: artifacts: public/build: path: /builds/worker/artifacts type: directory...
We're encountering test failures as we transition to macOS 14 because the `worker-runner` binary is not signed. Details: [https://bugzilla.mozilla.org/show_bug.cgi?id=1932140](https://bugzilla.mozilla.org/show_bug.cgi?id=1932140) In summary, we can use the following command as a workaround:...
In GCP we seem to have intermittent failures for snd-aloop feature. See e.g. [task exception](https://community-tc.services.mozilla.com/tasks/IJ_1SRUrRm6GDi4EtyNzMQ/runs/0/logs/public/logs/live.log#L133) vs [successful task](https://community-tc.services.mozilla.com/tasks/XTnA1JesRYeMOowJx8FwSg/runs/0/logs/public/logs/live.log#L132). Currently we test worker capabilities after a loopback audio task has been...
**Describe the bug** Worker `3297482374640595201` of https://firefox-ci-tc.services.mozilla.com executed [this task](https://firefox-ci-tc.services.mozilla.com/tasks/XgFUn2bOTyOQqayM96duiQ) but its task list [shows no tasks](https://firefox-ci-tc.services.mozilla.com/provisioners/gecko-t/worker-types/t-linux-xlarge-noscratch-gcp/workers/us-central1-a/3297482374640595201?sortBy=started&sortDirection=desc). Bastien Orivel highlighted the server responds to the request about worker information with...
Challenges: * taskcluster-proxy isolation between tasks * separate worker configs / logs / .... * shared caches? flock protection / ? * protect against reboots .... ....
A common thing for me to do when a task is pending for a period of time is to check out the count of pending tasks for that worker type....
Currently provisioner fetches all workers, then iterates and updates counts. https://github.com/taskcluster/taskcluster/blob/main/services/worker-manager/src/provisioner.js#L114-L120 It doesn't do anything else with workers besides updating existing/requested/stopping capacity. Having thousands of running workers and doing provision...