motivation

I’ve recently worked on a large project where a lot of time and space (and subsequently money) was spent on Docker hub. This slowed down our CI/CD, development speed, and generally costed us. We figured that if we make the images smaller and leaner, we can win a few things at the same time.

gains

• spend less money on CI
• spend less time waiting on builds during local testing
• spend less money on storage in Docker hub

how did we do it

Through many small alterations. Just as with optimizing any system, there rarely is one large thing to change that has big impact on a performace of it all, but a lot of interdependent tweaks have helped us to:

• reduce the image size by almost 40%
• reduce the build times for most jobs by 80% !

Below is a list of things we’ve done to achieve that.

listicle

(number seven will surprise you!)

• Optimizing docker image build times

  • move to a small and efficient base image. Alpine linux is my go to.
  • review images provided by Docker or other trusted orgs (example) - they’re more likely to be well-maintained and optimized

• Leverage caching

  • get familiar with how docker layers cache within images and ensure that the instructions that change the image often are further towards the end of your dockerfile. Official documentation can be found here

• Use multi-stage builds

  • reduce size of the final image
  • use one stage to build your app and another to package it into a smaller image for runtime
  • this approach keeps only the necessary artifacts and deps in the final image, reducing its size and build time

• Minimize the number of layers

  • each instruction creates a new layer in the image. Use && in your shell scripts to combine instructions into single invocations which reduces the number of layers (for example instead of installing packages one by one, use a single RUN to install them all)

• Optimize package managers

  • using apt or yum with default settings can lead to extra packages being installed that have nothing to do with your application. Be judicious and remove unnecessary dependenceis using flags like --no-install-recommends or --no-install-suggests. (refer to your package manager docs for the actual flags)
  • clear package manager caches within the same RUN instruction to reduce the image size

• use .dockerignore effectively

  • this one you probably are well aware of, but if you are lagging on removing extra files from your image build, maybe you will be pleasantly surprised. Remember that disk reads are super expensive and should be avoided at all costs when tuning performance of your builds

• cache external dependencies

  • if your app relies on external dependencies like libs or data files, consider caching them separately and only update when needed

• multi-image layers

  • if you build a lot of projects and they share base images, consider reviewing them to see if there are opportunities for using a common base image - remember that a linear growth of the number of projects can still lead to exponential growth of build time if left unchecked.

• use docker buildkit to monitor the builds and seek insights.

  • Set up a simple script that emails you a report at the end of the image build. If you’re really feeling testy, make it calculate the dollar amount. Nothing beats seeing a dollar amount. There’s something psychologically more significant about seeing costs go up (see fallacy of gaining $0.05 vs losing $0.05) than them going down.

• some CI/CD systems and Docker registries offer build caching services

  • utilize them to share build caches. it’s agains their business model (as they want you to spend more money) but they still offer it to you. Don’t get fooled.

• upgrade docker

  • this one is pretty obvious. Literally just upgrade your docker version. Performance improvements happen all the time. Keep your tools up to date. Always.

epilogue

That’s all, folks. There you go. Now run with it, do your job, save some time an money for everyone and be the hero the world deserves.