Every disaster recovery plan I have ever reviewed has a diagram somewhere in it — primary power, backup generator, fuel reserve, all neatly boxed and arrowed. What almost none of them have is a box for "25 people carrying diesel up 17 flights of stairs in the dark." And yet that is precisely the layer of redundancy that kept a Manhattan data center alive during one of the most destructive storms in New York City's history.

This is a true story I came across while researching case studies for an upcoming book on datacenter reliability — extensively documented at the time, and worth telling properly, because it is the clearest real-world illustration I know of a principle every IT leader claims to believe and almost never actually plans for: your resilience is only as good as the layer underneath the layer you were counting on.

The Night the Basement Went Underwater

Hurricane Sandy made landfall on the U.S. East Coast on October 29, 2012. Its storm surge pushed seawater deep into Lower Manhattan, flooding basement levels throughout the Financial District — including 75 Broad Street, home to a Peer 1 Hosting data center serving roughly 100 companies.

The building's design looked, on paper, exactly like what a disaster recovery audit wants to see: a backup generator, and a 20,000-gallon diesel reserve to keep it running through an extended outage. The flaw was not in the redundancy. It was in the geography. Both the generator's fuel-delivery pumps and the reserve tank itself sat in the basement — the one part of the building Sandy's storm surge was guaranteed to reach first.

As the flooding spread, Con Edison cut electrical power to the area. The basement generator and its fuel infrastructure went down with it. Peer 1's only remaining operable generator was several floors up — reachable only by stairwell, since there was no power left to run the elevators.

A 200-Gallon Tank Against a 40-Gallon-an-Hour Problem

The rooftop-level generator had its own small day tank — roughly 200 gallons — enough to run independently for a few hours. But the data center it was powering drew up to 40 gallons of diesel per hour. Simple arithmetic made the problem obvious immediately: without a way to keep refilling that tank, the facility had a matter of hours, not days, before it went dark — with no elevators, no basement pumps, and a 20,000-gallon reserve sitting useless under several feet of seawater one floor below where it was needed.

There was no engineered solution to this problem still available. The systems designed to solve exactly this scenario were the ones that had just failed.

The Bucket Brigade

What happened next was not in anyone's runbook. Employees from Peer 1, from Squarespace — one of the roughly 100 companies hosted in the facility — and from Fog Creek Software, maker of the project-management tool Trello, formed what they started calling the "bucket brigade."

Diesel fuel was hauled up seventeen flights of stairs in five-gallon buckets, two per person, handed off floor to floor in relay fashion through pitch-dark, diesel-reeking stairwells. Roughly 25 people rotated through the effort continuously. A low-fuel alarm at Peer 1's second-floor office signaled when it was time to restart the relay.

The brigade ran from Tuesday afternoon, October 30, until roughly 1 a.m. Wednesday, October 31 — and then continued through the day that followed. By 2 p.m. Wednesday, Peer 1 estimated it had roughly eight hours of diesel reserve left. Fog Creek seriously considered migrating its 65 servers to another Manhattan facility with power as a last resort. The bucket brigade kept the lights on long enough that the migration was never needed.

Jesse Hertzberg, a senior Squarespace executive at the time, put it simply afterward: the facility had "already made it 28 hours longer than we thought we would."

Why This Story Matters More Than the Redundancy Diagram

It would be easy to read this as an inspiring but irrelevant one-off — a story about heroism, not architecture. That reading misses the actual lesson.

Every layer of Peer 1's disaster recovery plan worked exactly as designed, right up until the layer underneath it failed. The generator worked. The failover logic worked. What nobody had modeled was that the fuel delivery mechanism and the fuel reserve shared a single point of geographic failure with each other — and that the building's own resilience plan had never asked "what happens if the thing that refuels our backup power is in the same room as the thing our backup power exists to survive?"

This is the exact failure pattern this site has written about before in different forms: a single point of trust that nobody stress-tested because it was "reliable." The generator was reliable. The basement fuel system was reliable. Neither assumption survived contact with a storm surge that didn't know it was supposed to respect the boundary between "primary risk" and "backup mitigation."

The bucket brigade did not fix the architecture. It substituted human effort, in real time, for a redundancy layer that had been quietly missing the entire time nobody was testing for a flood.

What This Means for Every IT Leader Planning Around "We Have a Generator"

1. Where does your backup power's fuel actually live — and does it share a failure domain with the thing it's backing up? A generator on an upper floor and a fuel tank in a flood-prone basement are not two independent layers of resilience. They are one layer, wearing two names.

2. What is your actual hours-of-runway number, not your theoretical one? Peer 1 knew its day tank held roughly 200 gallons and burned 40 gallons an hour. That arithmetic — a handful of hours, not days — is what turned a flood into an active, minute-by-minute operational crisis rather than a manageable inconvenience. Most organisations have never done this math for their own facilities.

3. If your engineered failover failed the way this one did, what is your actual human contingency plan — and have you ever rehearsed it? Nobody trained Squarespace and Fog Creek employees to run a bucket brigade. They improvised one, successfully, because the people in the building that night were willing to treat "keep the data center up" as everyone's problem, not just the vendor's. That kind of improvisation is far more likely to work if it has been discussed once, in advance, than if it is invented cold at 2 a.m. during a flood.

The Bottom Line

The engineering lesson from 75 Broad Street is not "buy a bigger generator." It's that resilience plans fail quietly at the seams between systems — the point where "our backup power" meets "the thing that feeds our backup power" — and those seams are exactly where nobody usually looks, because each side of the seam looks fine on its own.

What actually kept a Manhattan data center online through Hurricane Sandy was not a redundant system. It was 25 people who decided that the plan failing was not the same thing as the mission failing, and carried the difference up seventeen flights of stairs, one five-gallon bucket at a time.

Sources: Data Center Knowledge, Computerworld, The Register, HuffPost.