The CSV Export That Took Down the API

A customer clicked "Export to CSV" on 380,000 records. The server loaded them all into memory, OOMed, and took the API offline for every tenant. The fix was straightforward. The real question is why nobody caught it sooner.

The feature request was five words: "Add a CSV export button." The kind of ticket that gets estimated at half a day, merged by lunch, and forgotten by the next standup.

The client was a SaaS platform for logistics companies — tracking shipments, invoices, delivery confirmations. About 200 tenants, most with a few thousand records. The export endpoint was simple: query the shipments table, map each row to a CSV line, return the whole thing as a response body. It shipped in March. Nobody thought about it again until July.

What happened in July

A new customer onboarded. They were the client's biggest account yet — a freight company running 380,000 shipment records through the platform. On their second day, someone in their ops team clicked "Export All" on the shipments page.

The endpoint did exactly what it was told to do. It ran a SELECT * with no limit, loaded 380,000 rows into a JavaScript array, mapped each one through a formatting function that joined 34 columns with commas, concatenated the result into a single string, and sent it back.

The Node.js process hit its memory ceiling and crashed. Kubernetes restarted it. But the ops person saw a timeout and clicked the button again. The new process spun up, got the same request from the retry queue, and died the same way. Meanwhile, all other API requests for every tenant were failing because the pod kept cycling.

It took 11 minutes before the on-call engineer killed the export requests in the queue and the service stabilized. Eleven minutes of full downtime, across all tenants, because of a CSV button.

The code that caused it

The endpoint looked like this:

async function exportShipments(req: Request, res: Response) {
  const shipments = await db.shipment.findMany({
    where: { tenantId: req.tenantId },
  });
 
  const header = COLUMNS.map((c) => c.label).join(",");
  const rows = shipments.map((s) =>
    COLUMNS.map((c) => escapeCSV(c.format(s))).join(",")
  );
 
  const csv = [header, ...rows].join("\n");
 
  res.setHeader("Content-Type", "text/csv");
  res.setHeader("Content-Disposition", "attachment; filename=shipments.csv");
  res.send(csv);
}

Nothing about this screams "danger." It's readable, it handles escaping, and it works perfectly for 2,000 records. The problem is that it holds the entire dataset in memory three times over: once as the ORM result, once as the mapped array of strings, and once as the concatenated output.

For 380,000 rows with 34 columns, that was roughly 1.8 GB of heap usage in a container with a 512 MB limit.

The fix

We rewrote it to stream. Instead of loading everything into memory, we used a database cursor and piped rows directly to the response:

async function exportShipments(req: Request, res: Response) {
  res.setHeader("Content-Type", "text/csv");
  res.setHeader("Content-Disposition", "attachment; filename=shipments.csv");
  res.write(COLUMNS.map((c) => c.label).join(",") + "\n");
 
  const cursor = db.$queryRawUnsafe(
    `SELECT * FROM "Shipment" WHERE "tenant_id" = $1`,
    req.tenantId
  );
 
  for await (const row of cursor) {
    const line = COLUMNS.map((c) => escapeCSV(c.format(row))).join(",");
    res.write(line + "\n");
  }
 
  res.end();
}

Memory usage dropped from "all of it" to a few megabytes regardless of dataset size. The export for 380,000 records took about 45 seconds, which isn't fast, but nobody's server died.

We also added a request timeout and a rough row-count check that switches to a background job for exports over 50,000 records, sending a download link via email instead.

The part that bugged me

The real issue wasn't the code. It was the assumption baked into how we built it: that the biggest tenant would always look like the current biggest tenant. When the endpoint shipped, the largest customer had 8,000 shipments. Nobody asked "what happens at 100,000?" because 100,000 felt hypothetical.

This is a pattern I keep running into on consulting projects. Teams build features against today's data shape and forget that production datasets don't grow linearly. They grow in steps, usually when a big customer signs up or a migration dumps historical data into the system. The feature that was fine for months suddenly isn't, and it always happens at the worst time — during a demo, a compliance audit, or the first week of your biggest contract.

The fix is boring: before you ship any endpoint that touches a collection, ask what happens when that collection has 10x the data. Not 2x. Not "a lot more." Specifically 10x. If the answer involves loading it all into memory, you have a problem you just haven't met yet.

What we changed after

Beyond fixing the export endpoint, we audited every query in the codebase that didn't have a LIMIT clause. There were fourteen. Most were admin endpoints or internal tools — places where "just get everything" felt safe because the data was small. We added pagination or streaming to all of them.

We also added a memory usage alert at 70% of the container limit. The previous alert was at 90%, which gave us about four seconds of warning before OOM. Not exactly actionable.

The whole fix took a day and a half. The outage took 11 minutes and cost the client a very uncomfortable call with their new biggest customer. It's the kind of bug that makes you feel stupid in retrospect, which is usually a sign that the failure mode should have been more obvious from the start. Maybe it was. Maybe we just didn't want to complicate a half-day ticket.

What's the most "obviously fine" code you've shipped that turned out to have a ceiling you didn't see coming?