Case study
The Dockerfile Nobody Had Touched in Two Years
A client's Docker build took 18 minutes. Developers had stopped building locally and were pushing to CI for every change. The fix took an afternoon, but the real cost was the two years of compounded slowness nobody measured.
The tech lead's exact words were: "Our developers say everything is slow, but I don't know what 'everything' means." This was three days into an engagement with a 40-person SaaS company. The team had grown from 12 to 40 engineers in about 18 months, and somewhere in that growth, iteration speed had fallen off a cliff.
I started where I usually start: I asked a backend developer to walk me through making a small change and getting it running locally.
She changed a single line in a controller. Then she typed docker compose up --build, leaned back, and said "this is the part where I go get coffee."
The build took 18 minutes and 42 seconds. I timed it.
The Dockerfile
I pulled up the Dockerfile for the main API service. It was 67 lines long and had a last-modified date of March 2024. The git blame confirmed it — one engineer had written it during the initial containerization effort, and it hadn't been meaningfully changed since.
Here's what it looked like (simplified, but the structure is accurate):
FROM ubuntu:22.04
RUN apt-get update && apt-get install -y \
curl wget git build-essential python3 \
imagemagick ffmpeg ghostscript \
postgresql-client redis-tools \
vim nano htop strace \
# ... 34 more packages
COPY . /app
WORKDIR /app
RUN npm install
RUN npm run build
EXPOSE 3000
CMD ["node", "dist/server.js"]If you've spent time optimizing Docker builds, you're probably already wincing. But when I showed this to the team, nobody saw the problem. It worked. It had always worked. Slow, but functional.
What was actually happening
The issues stacked on top of each other in a way that turned minutes into a quarter-hour.
Base image. ubuntu:22.04 is about 77MB compressed, but the real cost isn't the download — it's the apt-get install that follows. The team was installing 47 packages. Some were genuine dependencies. imagemagick and ghostscript were needed for PDF processing. postgresql-client was used in a health check script. The rest — vim, nano, htop, strace, wget — were debugging tools someone had added during an incident in 2024 and never removed.
No .dockerignore. The COPY . /app was copying everything. The .git directory (680MB of history), node_modules (if they existed locally), test fixtures, documentation, IDE configs. On one developer's machine, this step alone took 3 minutes because they had a large dataset in a tmp/ folder they used for local testing.
Layer ordering. This is the one that hurts the most. COPY . /app appears before npm install. Docker's layer cache works bottom-up: if a layer changes, every layer after it is invalidated. So every time any file changed — a one-line fix to a controller, a README update, a new test — Docker threw away the cached npm install and ran it fresh. For this project, npm install took about 6 minutes.
Production image bloat. There was no multi-stage build. Dev dependencies, test frameworks, TypeScript source, build tooling — all of it ended up in the production image. The final image was 2.8GB.
Note
The fix
The whole thing took an afternoon. Here's what the Dockerfile looked like after:
FROM node:20-slim AS builder
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --ignore-scripts
COPY tsconfig.json ./
COPY src/ ./src/
RUN npm run build
FROM node:20-slim
RUN apt-get update && apt-get install -y --no-install-recommends \
imagemagick ghostscript postgresql-client \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev --ignore-scripts
COPY --from=builder /app/dist ./dist
EXPOSE 3000
CMD ["node", "dist/server.js"]And the .dockerignore:
.git
node_modules
dist
*.md
.env*
tmp/
test/
coverage/
.vscode/
.idea/
The result: 1 minute 53 seconds for a clean build. About 12 seconds for a rebuild after a source code change (because npm ci is cached). The production image went from 2.8GB to 340MB.
The cost nobody measured
Here's what bothered me more than the 18-minute build. The team had adapted to it. Developers pushed to CI to test changes because building locally was too slow. This meant the CI queue — which was sized for pull request builds — was handling three to four times the expected load. Average queue wait time was 11 minutes. So a developer changing one line waited 18 minutes locally or 11 minutes in CI plus the time to check the result.
Some developers had started skipping the build entirely, testing against the shared staging environment by deploying their branch. Staging was broken about 40% of the time I was there.
Nobody connected these problems. The slow Docker build was just "how things are." The CI queue was "something we should throw more runners at." Staging being broken was "a discipline problem." But they were all the same problem, compounding.
A rough estimate: 35 backend developers, averaging maybe 8 builds per day, each spending 10 extra minutes waiting compared to a reasonable build time. That's about 47 hours of developer time per day, burned on waiting. Across a month, it's something like 940 hours. I don't love converting developer hours to dollars — it's never that clean — but even at a conservative rate, the company was losing more per month in idle time than it would cost to hire a senior DevOps engineer for a year.
Why it stayed broken
Two years is a long time for something this fixable to stay broken. But I see it constantly. Build infrastructure lives in a weird dead zone: it's not a feature (so product doesn't prioritize it), it's not a production incident (so SRE doesn't own it), and it works (so nobody files a bug). It degrades gradually as the codebase grows — a new dependency here, a larger test fixture there — and each individual change is too small to notice.
The engineer who wrote the original Dockerfile had left the company a year ago. Nobody else felt ownership over it. When I asked the team who was responsible for build performance, I got six different answers, all of which were some variation of "not me, but probably DevOps." The DevOps team said their job was CI/CD pipelines and infrastructure, not individual service builds.
The ownership vacuum is the real bug. The Dockerfile is just where it manifested.
What I'd check in yours
If you haven't looked at your Dockerfiles recently, run docker images and check the sizes. Then time a rebuild after changing a single source file. If it's reinstalling dependencies on every change, your layer order is wrong. If your production image is over a gigabyte, you probably have dev dependencies or build tools you don't need at runtime.
These are 30-minute fixes that pay dividends on every build, for every developer, for the remaining life of the project. The only reason they stay broken is that nobody looks.