CI Minutes, issue 1: Is your CI slow, or just waiting?
A pull request shows one number: how long CI took. That number hides two different problems. Either the jobs take long to run, or they wait long to start. The fixes are different, so it's worth knowing which one you have.
Split the two
Every GitHub Actions job records three timestamps: created_at,
started_at and completed_at.
- Queue time =
started_at−created_at. Time spent waiting for a runner. - Run time =
completed_at−started_at. Time spent doing the work.
Pull a week of jobs from the API (GET /repos/{owner}/{repo}/actions/runs,
then the jobs for each run) and compute both per job.
Read the numbers
- Compare p50 with p95. If p50 queue time is low but p95 is high, most jobs start fast and a few wait a long time: usually peak-hour capacity.
- Split by runner label. One label queueing while others don't points at that pool, not at CI as a whole.
- Split by hour. Queues that grow at the same hours every day are a capacity problem, not a code problem.
- Count re-runs (
run_attemptabove 1). Flaky jobs double both numbers.
What to do with it
- Mostly run time: look at dependency caching, test sharding and job splitting before buying bigger machines.
- Mostly queue time: add capacity at peak hours, give short jobs (lint, unit tests) their own runners, or move to runners that scale per job.
From Runzivo
We're building managed Linux x64 runners for GitHub Actions, with each job in a fresh, single-use microVM. Sign up at https://app.runzivo.dev
Next issue: the hidden costs of self-hosted runners, and how to put a number on them.
Don't miss what's next. Subscribe to Runzivo:
Continue the conversation: