The Day GitHub Actions Went Down and Nobody Could Deploy

GitHub Actions was out for nearly eleven hours on August 6. On the client project I was embedded in, it took about forty minutes for us to realize we had no Plan B. We couldn't deploy, couldn't run tests, couldn't merge with confidence. A ten-hour lesson in single points of failure.

I was mid-standup on a Wednesday afternoon when the first Slack message came in. "Heads up, CI is red on my PR." Then two more within a minute. Then six. By 4 PM, our team channel was a wall of red X screenshots.

GitHub Actions had gone down. Not a brief hiccup — a nearly eleven-hour outage that started around 15:22 UTC on August 6. Workflows were failing, webhook triggers were throttled down to roughly 15% capacity, and some events were dropped entirely. If you pushed code during that window, your CI run may never have started at all.

I was embedded in a mid-size e-commerce company at the time, helping them rework their deployment pipeline. Ironic timing.

The first forty minutes

The team's initial reaction was to wait. "GitHub will fix it, probably back in twenty minutes." Fair assumption — most outages are brief. We kept working on feature branches, writing code we couldn't merge.

After forty minutes, the mood shifted. A production bug had been found in the payment flow. Not catastrophic, but losing a small percentage of transactions every hour. The kind of thing you'd normally hotfix and deploy in fifteen minutes.

Except we couldn't deploy.

Our entire deployment process — linting, unit tests, integration tests, container builds, image pushes, Kubernetes rollouts — ran through GitHub Actions. Every single step. There was no manual deployment script. No local build-and-push process. No break-glass procedure documented anywhere.

The senior engineer who'd set up the pipeline two years ago had left the company. Nobody else had ever deployed without CI.

What we tried

Someone suggested pushing directly to the container registry. Reasonable idea, except the Dockerfiles referenced build arguments that were injected by the Actions workflow, and nobody could find the values. They were stored in GitHub's encrypted secrets. We could see the secret names in the workflow YAML, but not the actual values.

Another engineer tried to reconstruct the build locally. She got most of it working after about an hour, but the integration tests needed a database seeded with specific fixtures that the CI job created from a script that downloaded test data from — you guessed it — a GitHub-hosted artifact.

We ended up deploying the hotfix by manually running the build steps we could replicate, skipping the integration tests entirely, and pushing the container image to the registry by hand. Then someone ran kubectl set image against production. It worked. The bug was fixed. But the deploy had no audit trail, no test verification, and it took three engineers two hours to pull off what the pipeline normally did in eight minutes.

The uncomfortable post-outage conversation

GitHub Actions came back around 2 AM UTC. By Thursday morning, everything was green again. You'd think the story ends there.

But the retro was uncomfortable. I asked the team a simple question: "If GitHub Actions disappeared permanently tomorrow, how long would it take you to deploy?"

The honest answer was: days. Maybe a week to reconstruct the full pipeline somewhere else. The workflow files referenced custom actions, marketplace actions pinned to specific versions, matrix builds across three Node versions, caching steps that relied on GitHub's built-in cache. It was a deeply coupled system masquerading as "just YAML."

This wasn't a poorly run team. Their pipeline was actually well-designed — good test coverage, proper staging gates, efficient caching. They'd done the hard work of building reliable CI. What they hadn't done was build reliable CI that could survive its own platform going away.

The break-glass runbook

After the retro, we spent two days building what I now recommend to every client: a break-glass deployment runbook. Not a replacement pipeline — just the minimal steps to get code into production when your primary CI is unavailable.

Ours looked roughly like this:

# 1. Run the critical test suite locally
npm run test:unit
npm run test:integration -- --db-url=$LOCAL_DB_URL
 
# 2. Build the container image
docker build -t registry.example.com/app:hotfix-$(date +%s) .
 
# 3. Push to the container registry
docker push registry.example.com/app:hotfix-$(date +%s)
 
# 4. Deploy to staging, verify, then production
kubectl set image deployment/app \
  app=registry.example.com/app:hotfix-$(date +%s) \
  --namespace=staging

We stored the required secrets in the company's vault (not only in GitHub). We documented which tests could be skipped in an emergency and which absolutely could not. We tested the runbook by actually deploying through it once, on a quiet Friday afternoon.

The whole thing fit on two pages. It wasn't elegant. It didn't run matrix builds or upload coverage reports. But it could get a critical fix into production in under twenty minutes, even if GitHub was a smoking crater.

The broader pattern

This isn't really about GitHub. Actions is a solid product and this was their first outage of this severity in a long time. The pattern I keep seeing is teams that build sophisticated automation on a single platform and never ask, "What if this platform is unavailable for twelve hours?"

It happens with CI providers. It happens with cloud regions. It happens with Kubernetes clusters that nobody has ever failed over. The sophistication of the happy path becomes the fragility of the unhappy path.

I'm not arguing for multi-cloud CI or running your own Jenkins server as a backup. That's overkill for most teams. But a tested, documented, manual deployment procedure that two or three people on the team can execute from memory? That's a few hours of work that buys you something no amount of pipeline optimization can: the ability to ship when everything else is on fire.

How many steps does it take for your team to deploy without CI? If you don't know the answer, you might want to find out on a quiet Friday — not during the next outage.