13 December 1996
What a day. I apologize to all our customers and their customers for
the fiasco that has been on-going since 12:30 PM EST on Friday afternoon.
It is now 3 AM on Saturday morning and we are finally back up. In a nutshell,
this is what happened...
An electrical problem caused the T1 circuitry to fail. We immediately
went to the back-up. Then something failed in a repeater that homes all
our datacom lines. So we switched over to the emergency ISDN back-up. Then
the impossible happened. The ISDN failed. What occurred was statistically
impossible...three simultaneous failures that each took down all three
legs of our redundant telco systems.
To BellSouth's credit, they performed far above anything that anyone
would have expected. Engineers were all over the place, replacing every
piece of equipment between us and the POP. The last engineer left our house
shortly after 2 AM on Saturday morning. And the folks at the NOC were amazing.
They kept us appraised every hour as to what exactly was happening for
the 13 1/2 hours we were down. Everyone at BellSouth deserves every single
penny of their salary and they are all invited over on the first warm,
spring day for a cookout - even the folks in Atlanta. Nice job folks...and
a special thanks to James, Joe, Chris, Ron, Dave, and Garfield (the barking
For those customers who received any email during this outage, it will
be delivered within two or three hours. Fortunately, emails do not get
lost; they just bounce around for up to three days until they can be delivered.
We are now stable and properly reconfigured. BellSouth will continue
testing tomorrow. What will happen is that they will get the ISDN fully
operational. We will then go down for about five minutes while we switch
TCP/IP over to the ISDN. Then BellSouth will replace physical copper on
our T1 lines. When they are properly configured, we will go down for about
five minutes again to switch back to T1. At that point, we will be fully
up and running. Of course, that's when the power will fail. :)
Speaking of which...I realized that many of our customers do not know
the level of redundancy we have built into all our processes (including
security measures.) Sometime this up-coming week, I will put together a
web page on what we have done - or better yet - what Mikey has created
and will post the link here. Come check it out.
Again, sorry if the situation today caused anyone any problems. If it
is any consolation, the service we received from BellSouth was so much
better than any other provider could have even come close to that there
is no comparison.
ps. Just to add insult to injury, two other things happened in Raleigh,
North Carolina today that made it memorable in the annals of the computer
world. The State of North Carolina government offices (statewide) went
down for almost three hours because of a computer malfunction. And to top
it all off, the main plant of Cisco Systems, the manufacturer of routers,
had a generator fire that knocked out their power and shut them down for
a considerable length of time. Again, what a day...
October 21, 1996
Took the secure server down for a few minutes in order to send a secure
super-encrypted top secret confidential take-out order to Domino's pizza.
Within 500 milliseconds, I had a large pickle and onion pan pizza, with
extra cheese, pouring out of the T1. Mmmm! Want
Status of Mikey<: Full, but there
are leftovers if anyone's interested...
October 15, 1996
The unthinkable happened today. Our web server came to life and cooked
us all breakfast this morning! We had eggs and chicken and doritos and...
Alright, I'll admit that the food choice was a bit odd, but you've got
to give it credit for trying. Anyway, we were all sitting around enjoying
the day when Valerie noticed that the server--he
asked us to call him George--had unplugged himself from the T1. We ran
into the server room and sure enough, the floor was covered with packets
that had leaked out. What a mess!
As everyone knows, packets will die if they sit out in the open for too
long and sure enough, several of them had expired. We gathered as many
of them as we could and put them in the bit bucket so that we could perform
CPR (computer packet resuscitation).
As a result of today's activities, you may need to come to PagePlop's corporate
headquarters to claim your surviving packets. Funeral services for the
deceased packets will be held at the end of the week.
Status of Mikey: I'm very upset
by the whole situation. I thought George knew better.
October 4, 1996
Ahhhh! We were down for 4 hours lastnite due to a disgruntled UNIX hacker
with a John Deere frontend loader. He took out ten high-speed connections
along the East coast before BBNPlanet special agent Bill Dearing took him
down with a canister of "Hacker-B-Gone" nerve gas. Cisco router
technicians on the scene were further shocked when police, while attempting
to remove the frontend loader, inadvertently rolled over a nearby router!
The damage was extensive, but easily repairable and we were back up within
Status of Mikey: When I noticed
that we were down, I unplugged our T1 and shook it to see if anything was
stuck in there. Then I looked into the T1 and saw this guy on a tractor
on the other end tearing everything up. I stared into the wire a little
too long, however, because when they fixed everything, a high-speed packet
came out a hit me in the eye. I'm okay, but it sure didn't feel good.
September 16, 1996
We were down from 4:00pm to 5:00pm for scheduled maintenance. You know,
the standard stuff like new spark plugs and fluid checks. Our next scheduled
maintenance is either in three months or 30,000 bits.
Status of Mikey: Smoking!!!
September 10, 1996
Hurricane Fran Update: It's over! I thought we were in real trouble
with all that wind and rain, but the T1 never went down and the Macs performed
Duration of power failure: 110 hours
Number of packets lost: 0
Status of Mikey: Busy, removing
wet packets from the T1, drying them off, and carefully re-inserting them
(there's nothing worse than getting a soggy packet).
September 7, 1996
Hurricane Fran Update. The eye of the storm is currently about 40 miles
to the southwest of us. The wind at the airport was just clocked at 79
mph and the rain is coming down in torrents. The power went out a couple
of hours ago, but we're running off the backup generator.
Status of Mikey: Wet. (I'm typing
this on my home computer which is currently submerged beneath 6 feet of
water. Hey, the scuba gear came in handy :-)
September 5, 1996
The National Weather Service has just placed the coast of North Carolina
under a hurricane warning and the entire state under a flash flood warning.
Hurricane Fran is currently a category 3 storm with sustained winds of
115 mph. We may be in for a rough ride...
Status of Mikey: (Mikey could not
be reached for a comment as he is at the store buying bread, water, and
August 27, 1996
Today we proved beyond a shadow of a doubt that there is at least one
moron who works for Apple (and if I can find out who it is...). This morning,
we began experiencing an obscure network problem that manifests itself
as email messages that cannot be delivered :-o
Within 30 seconds, our network operations center had me on the phone and
we began working on the problem. After 1 hour and 20 minutes, we isolated
and corrected the problem, which believe it or not, turned out to be the
result of an omission in the operating instructions of our email server.
Status of Mikey: Very unhappy.
I'm going back to bed now :-(
August 25, 1996
Received and installed our digital ID from Verisign,
giving our secure server the highest level of security currently available.
Just in case you're curious, we were issued a "high assurance"
Class 3 digital ID which is specifically designed to ensure the security
of financial transactions over the Internet.
Status of Mikey: Very happy, and
feeling pretty secure about it :-)
August 15, 1996
Opening Day. We all set up lawn chairs in the server room and watched
the bits go. There were lots of blinking lights. You should have been here;
it was really kewl!
Status of Mikey: Very Happy :-)
August 12, 1996
Test Day. We singled out one test web server and pummeled it with continuous
connections for 18 hours straight. We hit the server especially hard during
the hours of 12:00 noon to 5:00pm:
Total number of hours: 5
Total number of successful connections: 480,000
Total number of unsuccessful connections: 0
Total number of dropped connections: 0
Total number of very happy Mikeys:
Return to the Server Status Menu