[cfgeeks] So Gil...

[syberghost]

I've never had anything near that bad.  My worst moment that wasn't my
fault was this:

The power goes out.  It's been over half an hour and it's still out.
At the time I was an ops monkey in the monitoring team.  Our Managing
Director calls in and asks for a status update, and I tell him:

"Everything's fine, Chuck.  I just visually verified the generator is
running, and hardware tells me the transfer switch is green, so in a
few minutes when the batteries hit their threshold the switch will
kick in and we'll be on generator."

All the lights go out.  The data center is completely dark, and the
ominous sound of a thousand disks spinning down is all you can hear.

"Uhm, I'm gonna have to call you back."

It was 19 hours after the power came back that the DBAs pronounced the
last Production database fully recovered; we spent the next month
planning, and the next year executing, a move of all Production
servers to Memphis.

Where, a year or so later, we had a big power mishap and it took 38
hours to get the last database happy.  :(