10 December 2002
After a six-hour nightmare with the router, we have finally figured out what the problem is. One of the NICs is bad. Mind you, this was the router that was configured by Genuity and shipped after full in-house testing. Yeah right. Unfortunately, the problem was intermittant and it took us so long to find it. On the up side, only the colocation clients were affected and even then it was an up and down thing. They are shipping me a new NIC and it should be here Wednesday. At that time, we will install it and it will work flawlessly or I will kill it.
7 December 2002
8:43 pm Eastern Standard Time.
That's the time the lights went back on in North Carolina. (I've always wanted to twist that phrase.)
Actually, we're pretty lucky that we already have our power back on...no wait. We paid a premium to live in an urban area instead of like some sort of stone-age throwbacks living in the woods. The power never should have gone off to begin with.
I have a need to scream at a politician.
7 December 2002
We are now a bit over 55 hours into the power outage and the generator is holding fine. All telco lines are up and have been so without disruption. Over 75 percent of the region is still without power and full service is not expected to be restored until Wednesday or later.
We will be taking the generator down at around noon today in order to change the oil and filter. The whole process should take about ten minutes, far less than the runtime of the UPS units, so there should be no disruption in service at all. Yeah, right...in your dreams. We are anticipating a complete disaster, but we are making plans to recover in the event something goes wrong so be of good cheer. Mighty Mouse is here to save the day. :)
5 December 2002
At 2330 hours on 4 December, the power went out -- here and all over the city. Transformers are exploding all over the area due to an ice storm that will only get worse before it gets better. We have switched over to generator power and will remain so until the outage is finished. Given the number of explosions being heard combined with the flashing of light all over the horizon, I would anticipate that it may be days before we are back on the grid.
The generator will run for another 90 hours before it is scheduled for an oil change. I can push it to four full days if need be, but much beyond that I will need to shut it down for about 30 minutes and change the oil. (For those who are interested, the power plant is a GM 350 CI engine just like you have in your car except powered by natural gas.)
Hopefully, the power will be back on well before the oil needs to be changed so there will be no disruption at all. If I do need to shut it down it will be in the middle of the night. Even so, I think that I can change the oil before the UPS units give out (they last about a half-hour themselves.)
3 December 2002
Well, this is the eve of the first major ice storm of the season. The National Weather Service, AccuWeather, Weather.com, and WRAL-TV all agree that we are gonna get stomped, beat up, and whooped. (Anyone recognize the musical reference in there?) When all four agree, it's sure to be big.
Anyway, we are expecting some major ice that will probably bring down trees and power lines causing massive power outages lasting for an indeterminate time. Not to worry, though. The 60 KW, natural gas generator with the automatic transfer switch is tested and waiting to go into action within seconds of the power going out. And the gas bill is paid in full. I am happy.
I am not happy that given our generator the entire neighborhood will be coming over to eat, sleep, and get out of the cold. I'm thinking of charging them for shelter, but I have to check with our local ordinances to see if that would constitute the need to be rezoned as a motel. hehehe
I'll keep you posted. In the meantime...keep warm, except for those folks in Florida who I don't want to hear about your 80 degree weather. Keep it to yourself. :)
3 December 2002
One of our email users sent thousands of emails which proceeded to bounce back due to old and bad email addresses. The bounces backed up other incoming email for several hours.
No email was lost, just queued. The email back up has been cleared and email is now functioning normally.
2 December 2002
We had a brief 30 minute telco outage at approximately 7:30 PM due to a PING flood originating from one of our coloc machines. The problem was quickly identified and resolved and telco service was restored.
22 October 2002
The IP switch went very well last evening. We were physically down for less than half an hour as we updated IPs on all machines and switched out the updated DNS information.
But I Can't Get To My Site or Email!
This morning, some clients are unable to reach their web site or email. This is not due to any problem that occurred on our system during the switch. Rather, it is due to the fact that some ISPs ignore the DNS information that we broadcast.
Let me explain...
When you visit your web site or check your email, your ISP checks to see where the domain name is hosted. The registrar (e.g., Network Solutions) shows that PagePlop hosts your domain name. Therefore, the ISP requests information from our DNS servers regarding the location of the web server or email server for your domain name. When this DNS information is provided to your ISP, we also send instructions that the information is valid for only one hour (known as a time to live). That means that for the next hour, the ISP may safely cache (save) the IP number of the web server and email server for your domain name. When that one hour expires, however, the ISP should do another search for updated information.
This is where the problems arises. Some ISPs, including larger ones (notably, Bellsouth, Earthlink and Time Warner Cable), ignore the "time to live" of one hour. Instead, they make their own internal decisions about how long they think the information is valid - usually 24 hours.
Can Anyone Get To My Web Site and Send Me Email?
Yes. In fact, the web servers and email servers are taking a normal amount of traffic. The caching as described above usually affects persons who recently (as of last evening) visited your web site or downloaded email causing the ISP to save incorrect information for up to 24 hours.
What Can I Do?
- One thing you can do is simply wait it out. After approximately 24 hours, everything should work itself out.
- If you are unable to check your email, you may temporarily substitute your POP (pop.domain.com) and SMTP (mail.pageplanet.com) server addresses with the IP number 220.127.116.11. This should enable you to send and receive email. Note: This is a temporary work-around only. You should change your settings back to a POP and SMTP server address within a few days. Not changing your information may cause problems if we need to add email servers in the future.
16 October 2002
One of our customers, who shall remain nameless, replied to an email with almost a thousand blind carbon-copy email addresses. Many of those blind carbon-copied email addresses were not legitimate and bounced back to our email server. The volume of incoming emails to the server caused a back-up starting early this afternoon. No emails have been lost and all are in the process of delivering now.
The customer has been flogged :)
Update: We are currently updating the email server application. The email server will be back online at approximately 7:30 PM Eastern.
1 August 2002
For approximately 2 hours this afternoon the email server was not accepting POP connections because we were tracing down an autoresponder loop. We found the problem at approximately 2:45 and the server is now running smoothly. During this incident, no incoming email was lost or delayed.
22 May 2002
Ok...so what happened...
There were a couple of things going on with the email serveryesterday. First there was the corrupt email box. The owner ofthat box has been killed. Then there was a corrupt email in thequeue trying to deliver to another email box. The sender of thatcorrupt email has been killed. Finally, there was a ton of spamhammering the server the whole time the email machine was backedup because of the corrupt mailbox and email. Unfortunately, thesenders of all the spam have not been killed, but I have doneone better.
We have put in a filter on the email server that rejects emailswhose headers carry certain characteristics of spam email. Thereare a few programs that are used to send bulk email and tendto leave clues of their existence. If those clues show up, theemails they send will be blown into oblivion. That makes me happy.The downside of using said filter is that there may be a smallnumber of clients who sent legitimate emails in bulk that carrythose signatures. Now, for those who ARE using bulk mailers tosend out legitimate emails, well, you really should be usinga listserver for that purpose. I mean, that is what a listserveis made for. On the other hand, if you are using one of thoseprograms and are a legitimate user of our email system then youshould be in the exclusion file anyway and not affected. If youfind that your stuff is bouncing, contact us and we will fixit or convince you to use a listserver.
At 4 am, there are still about 900 emails in the queue stackedup for delivery. Once they are through, I am going to open upthe SMTP incoming for stuff that has been trying to deliver sincethe problem started. I don't know how many thousands of emailsare included in the stack, but they will come hammering in anddeliver rather quickly now that the queue is almost empty.
Again, no mail has been lost -- just delayed. Everything shouldbe back to normal by the time the sun comes up (unless you arein Europe in which case the sun is already up and you reallyshould be sitting in a cafe somewhere taking it easy today.)
21 May 2002
7:00 PM Eastern - Update. We found the need in the haystack! The server is now in the process of delivering about 3,000 messages. As you can imagine that is going to take a while, but all email will be delivered. No emails were lost during the outage. Any email that was delivered to the email server was saved and email email delivered while the email server was down will be systematically re-delivered. The one exception to automatic redelivery of emails is email from MGI forms. If you receive orders, information requests, or other form submissions from your web site via MGI, please check your failed email database for any messages which could not be delivered.
6:00 PM Eastern - Update. We have narrowed down the email delivery problem to a corrupt mailbox. Unfortunately we must identify the one corrupt mailbox out of thousands (i.e., "needle in haystack"). We are in the process of locating that mailbox and now anticipate the email services will be online again by 8 PM Eastern.
3:30 PM Eastern - This afternoon emails began backing up on the email server. No emails were lost, but were simply not delivering to the POP email boxes. We are in the process of re-building the main drive and hope emails will begin delivering again within the next hour.
2 April 2002
This evening, PagePlop moved all servers to a new facility. Everything went very smoothly and services were only down during the actual transport of the equipment from one facility to another. We are proud to say that most clients didn't notice that we'd moved!
20 February 2002
During our maintenance window from 12 midnight to 3 AM Eastern time, service was temporarily interrupted while we performed required maintenance on our router.
We apologize for any inconvenience this may have caused.
16 February 2002
PagePlop experienced intermittent (30 seconds to a few minutes at a time) loss of connectivity during these two days as our router was cycled and upgraded.
Return to the Server Status Menu