Where our CI time goes: 7,963 jobs on our own runners

Our own CI runs on Runzivo. We use the same Insights view available to every workspace under CI & Runners to see where those jobs spend their time. This is a look at our workspace, with its particular mix of checks, releases and smoke tests. It is a record of our work, rather than a prediction of what another team’s jobs will do.

What we measured

We pulled the summary, heatmap and slow-jobs JSON on November 11, 2026 at 18:48 UTC. The selected window was 30 days, but the first jobs in it are from October 19. The data therefore covers October 19 to November 11, 2026, a span of 24 days, with the last day still in progress.

There were 7,963 jobs. The job conclusions were 7,017 succeeded, 427 failed, 462 cancelled, 55 other and 2 pending. The busiest day was November 8, with 820 jobs. These totals describe the snapshot when we read it, including jobs whose conclusions were still pending.

Waiting versus running

A typical job waited about a minute to start and ran for about a minute and a quarter. Those are separate measurements. Queue wait measures the interval from queued to started; duration measures the running job. Keeping them separate helps us decide which part of the experience needs a closer look.

We do not add the two medians to estimate a total. They summarize different sets of measured intervals, with different sample counts. Neither value describes every job, and adding them would not give the median time from queued to finished.

When jobs wait

The heatmap groups jobs by weekday and hour in UTC. Looking only at cells with at least 20 started jobs, the longest median waits were Tuesday at 15:00, at 4:56; Tuesday at 07:00, at 3:07; and Tuesday at 14:00, at 2:28. The shortest median waits among those cells were about 0:24 to 0:31.

That puts the longest median waits in these Tuesday cells. It does not explain the cause. Each cell combines the same weekday and hour across the window, so it also does not describe a single Tuesday. The minimum sample threshold keeps a cell with only a handful of jobs out of this comparison.

Where minutes go

Recorded time tells a different part of the story from job counts. Here is its share by runner label. These percentages are rounded; each of the two smallest shares is less than 1%.

This is where recorded runner time went in our workspace. It does not tell us how busy the processors were or which size a job should use. A share of minutes is a useful starting point for reviewing the mix of jobs assigned to each size.

Slowest workflows

Insights ranks slow workflows by p90 duration. Here are the first eight rows, with repository and workflow names unchanged. The median is p50. Nine in ten measured jobs finished within the p90 duration. The longest column records the maximum instead.

The CI workflow in Runzivo/rzv-core makes the distinction clear. Across 4,573 runs, its median was 1:14 and its p90 was 4:25. Its longest job lasted 15:55:31. That maximum is a single outlier, not a description of the usual run. This summary does not tell us why it lasted that long, and we will not guess.

What Insights does not tell you

Insights does not record why a job waited. Queue wait can include time behind the workspace concurrency limit. When waits are high, the recommendation in Insights suggests asking support to review capacity and concurrency, and it does not name a cause. A high wait is a reason to inspect the jobs and the workspace settings, rather than evidence for a particular cause.

Check your own

Open CI & Runners, then Insights in your workspace. Recorded minutes are visible to people with billing access. Start with the summary, check the heatmap for patterns, then read the slow-workflow rows with their sample counts. The Insights guide covers the view and CLI. These commands request a 30-day window; check when your own recorded jobs begin before describing the period.

rzv insights summary --days 30 --json
rzv insights heatmap --days 30 --json
rzv insights slow-jobs --days 30 --json