tale as old as time
Performance at scale matters. Cause it costs money.
Recently I’ve been working on a large overhaul of a main CI/CD pipeline for a big Rails monolith. We’ve all seen this story play out.
The app gets bootstrapped and some default simple test / build steps are implemented in CI-runner-du-jour, and for a good couple years it works completely fine.
Then something big happens - whether it is a round of funding that brings in twenty more engineers, you score a large client that introduces additional challenges in your codebase, or you just finally got to up the test coverage from 50 to 90%.
Whatever this is, it sneaks up on you and before you know it, like the proverbial frog in the pot, you’ve slowly boiled yourself. The pipeline that took minutes, then half an hour, then an hour, starts to spin out of control.
The tests aren’t parallelized, docker images builds are getting longer, additional integrations and checks get introduced.
So now we’re here.
how did we tackle this
We took a trifold approach. I’ve described it in other blogposts so here I’ll just link them.
- we optimized the docker builds
- we paralelized the rails test suite
- we made a lot of ci/cd tweaks, although this post doesn’t go that deep. Something to write about some other day.
One thing worth mentioning is using merge queues. If you’re interested in what they are, I suggest you do your research. If your company has tons of developers and your CI/CD costs are spiraling out of control, you may want to employ merge queue tooling. It didn’t work for us at the time and the juice wasn’t worth the squeeze, so we dropped it.
simple math
All in all, the total time saving was x_minutes_saved minutes per run, and with n_of_engineers engineers pushing code let’s say daily_pushes times a day each, that makes it x_minutes_saved * cost_per_minute * n_of_engineers * daily_pushes.
Now, what are you waiting for? Go make those improvements, and write a report just like this blogpost.
addendum
If you’re curious about really quantifying your costs, I recommend this excellent blogpost on Quantifying the Costs of Builds that I refer to often when trying to make a point. It’s primary school math but not done on apples and oranges, but dollars. You’re an engineer. Make a point with numbers. And then send that to your CFO.