Antique Engines and Old Iron
[Home] - [HELP] - [Forums] - [Library] - [Photo Gallery] - [Groups] - [Classified Ads] - [Subscribe] - [Links] - [Books] - [Sponsors] -

Go Back   SmokStak > SmokStak® Support and Information > Web Server News
Forgot Password? Join Us!

Notices

Web Server News Various information and older news about the SmokStak™ web server and its functioning.

Web Server News

System News - Electrical Fire and Explosion


this thread has 2 replies and has been viewed 916 times

Reply
 
Thread Tools Display Modes
  #1  
Old 06-06-2008, 02:51:53 PM
Harry's Avatar
Harry Harry is offline
Administrator
 
Join Date: Oct 2004
Location: Sarasota, Florida USA
Posts: 6,874
Thanks: 590
Thanked 14,305 Times in 2,478 Posts
Images: 316
Default Re: Planet outage, Onan backup

Our server is one of these black boxes.



http://www.oldengine.net/dataphoto.htm

Quote:
Aug 2 2006: The first phase of The Benmar Generator Augmentation Project is complete. This phase of the project added a new 2MW redundant generator, and a 12,000 gallon freestanding fuel tank to Houston's DC1, bringing the total number of generators at DC1 to three. The two generators DC1 had were zoned before the augmentation, but now the three generators have full paralleling gear. We currently have 4.75MW of generator power and 21,000 gallons of diesel fuel on site. At our current usage level this fuel will last for four to five days without refueling.
Here's a look at the power distribution system in one of the seven data centers.

Power distribution, uninterruptible battery supplies to hold power while the generators start up, power transfer switches and generators.

All this and if there's a problem in the conduit from the outside transformer, BANG!
Attached Thumbnails
Click image for larger version

Name:	switch86.jpg
Views:	96
Size:	90.4 KB
ID:	34960   Click image for larger version

Name:	ups92.jpg
Views:	96
Size:	126.3 KB
ID:	34961   Click image for larger version

Name:	cut87.jpg
Views:	87
Size:	88.7 KB
ID:	34962   Click image for larger version

Name:	gen88.jpg
Views:	96
Size:	122.7 KB
ID:	34963  
Reply With Quote
Sponsored Links
  #2  
Old 06-20-2008, 11:17:40 PM
Harry's Avatar
Harry Harry is offline
Administrator
 
Join Date: Oct 2004
Location: Sarasota, Florida USA
Posts: 6,874
Thanks: 590
Thanked 14,305 Times in 2,478 Posts
Images: 316
Default System News - Electrical Fire and Explosion

From datacenter management:

1) "Saturday, May 31, 2008 at approximately 5:45 p.m., a transformer in our H1 data center in Houston caught fire, thus requiring us to take down all generators as instructed by the fire department. All servers are down. We are working with the fire department, with our facilities staff on site, to assess the situation."

2) "We have determined that no servers in the data center have been damaged. Nonetheless, they are down because power is out. Teams across the board are working to take appropriate action."

3) "The explosion and electrical fire damaged, beyond repair, the electrical gear where the utility service enters the building as well as the transfer switch and main distribution panel that feeds the first floor of the data center."

========

Sunday: June 1st, 18 hours into this mess with no sign of recovery, I decided to order a second web server from the Dallas, Texas plant where conditions are fairly stable. See www.old-engine.com to reach this server. The new server was on on hot standby, ready and available for lease, but sold "as is." In other words, there was no choice about memory or hard drives, etc., but it is an active unit located in another datacenter that is unaffected by the fire. It did take another six hours to take control of it and get it ready.

Fortunately, prior to this event, I had been taking weekly backups by transporting our daily backups into an external NAS server account. This NAS backup server was physically relocated to the second data center where I was than able to link into it and transfer backup data into our new server. This brought Smokstak up to the time/date of May 24th. The problem was that if I opened Smokstak to making new posts on this old database, I would have to choose between dumping those interim posts or restore that last week of data when it became available. I decided to temporarily lock the forums into read only mode at that time.

Meanwhile, back at the Houston plant, once an inspection took place and the fire marshall gave the go-ahead, auxiliary power was restored to the second floor network system where our server is located. 24 hours later, power was back on to our number one server, but there was a lot of work to do to get everything working again. With the original server back on and accessible, I extracted a backup of our current database with all posts and photos intact over to the new server.

Older photos from within the Houston data center:
http://www.oldengine.net/dataphoto.htm

========

Later, with power on and apparently stable and network systems working, I decided to switch Enginads and Smokstak back over to the original server where all forums could be open for new posts and with no loss of previous posts or photos. Smokstak.com and Enginads.com and mail.oldengine.net are now open and are currently being run from H1 - Phase II, (the second floor of the Houston plant) which is now running on the plant's original standby generator power system. The power distribution panel for the second floor was unharmed by the fire and the plant auxiliary generator is functional. The server owners on the first floor were not so lucky. I believe at one time, we did in fact have a server on that floor.

Daily backups are being sent by FTP from our server number one to server number two. We can operate all sites from server two if need be, but server one is a much better unit, faster, more memory and it has faster hard drives.

Switching addresses between web servers can be difficult at best. It can take from one hour to one week for the address change to take effect depending upon your internet provider. To explain, DNS is the Yellow Pages of the internet. It takes the web page name you are looking for and converts that to the number of the web server, all in an instant. Unfortunately, these look-up pages take a painfully long time to make their way around the internet.

We do expect a maintenance outage during the weekend of June 14th for cutting power back over to the replacement incoming utility power service. This event will, hopefully, put everything back to normal condition.

=======

Getting information out of a large company is like pulling teeth! It's the ol' CYA approach...

June 2 – 5:35pm CDT - Late last night, I told you we hoped to have power to the 6,000 servers in Phase 2 of our H1 data center by midnight, with all servers up by early morning. I am glad to say we came close, just a few hours after sunrise. At this time, 100% of our servers in Phase 2 have power, and our technicians are working with customers on any remaining server issues. We are confident all remaining issues will be resolved shortly. I also explained the significant challenge we faced in the other phase where the actual explosion occurred. Our team came up with a creative way to restore power quicker than the 4-5 day outage. We decided not to wait for equipment for the electrical room completely, opting instead for a temporary solution to get power to the 3,000 servers. That solution involves using generator power for the next 10 -12 days until all the new equipment arrives to rebuild the electrical room for Phase 1. I explained that we expected to have a temporary solution in place by midnight tonight, with servers powered up tomorrow. The good news is that as you read this letter, the power is restored, and the temporary solution is in effect. Within the next two hours, the remaining 3,000 servers have power. We have overstaffed our data centers again to help during this initial power up.
This now leaves us facing step two of this process, which requires getting all of the equipment delivered and then rebuilding the electrical room to its original standard. To make the cutover to the rebuilt electrical room, the operations group believed it would take a maintenance outage of 24-48 hours. I have good news on that front. It's not perfect, but at present we now believe the maintenance window will be just 4-6 hours. That's still too long, and we will continue this week to find ways to reduce the time. Given that there will be some outage for the cutover, we will execute this step at midnight on a Saturday, either June 7 or June 14. We want to pick the most appropriate time to minimize impact to you.
I must admit that I am amazed. We are almost 18 hours ahead of schedule with this phase, thanks to our great suppliers and of course the great folks working here at The Planet. This could never have happened without the help of both, and I want to thank all of them.
There is still more work to do, but the progress is terrific. We will continue to work any and all customer issues, and we face the challenge of putting the permanent power fix in place for Phase 1. Nonetheless, there is still good news based on what I told you last night.
As each hour passes, we learn more and more. Please give us the time to continue our planning. We will provide you with information as we have it.

=======

(contd.)
Reply With Quote
  #3  
Old 06-20-2008, 11:17:59 PM
Harry's Avatar
Harry Harry is offline
Administrator
 
Join Date: Oct 2004
Location: Sarasota, Florida USA
Posts: 6,874
Thanks: 590
Thanked 14,305 Times in 2,478 Posts
Images: 316
Default Re: System News - Electrical Fire and Explosion

June 11 – 6:00pm CDT
We continue to make great progress getting our H1 data center back to normal operating conditions. In the interest of keeping you up-to-date on where we are, here is the latest information:
  1. Reconstruction of the electrical infrastructure is going well, and we expect cutover to utility power next week. As we finalize our plans, we will alert you and provide more details.
  2. We have brought Phase 1 back on UPS power. This means in the event of a temporary generator failure, the likelihood of an outage is minimized.
  3. We believe there will be no downtime required for the cutover to utility power. In earlier communications, we expected downtime to be somewhere in the range of 4-5 hours. Now that we have UPS backup to support the cutover from the temporary generator, the likelihood of an outage is minimized.
Construction overall is moving along nicely. We hope to be able to put this behind us in the next two weeks.

========

June 16 – 9:25am CDT
Our facilities team is putting the finishing touches on our H1 data center to prepare it for utility power cut-over. Our current schedule anticipates a transfer from generators to utility power on Thursday, June 19. The entire facility is now protected by our permanent UPS system, so we do not expect any downtime in either phase during this process. We will confirm the exact time of the cut-over as soon as it is officially scheduled.

========

Thursday, June 19, 2008, 8:39 AM...
Phase 1 of our H1 data center will be cutover to full utility power late tomorrow morning – Friday, June 20. We will again have our own backup generators up and running. The switch should be seamless to customers.

Late on Friday afternoon, June 20, we plan to cutover Phase 2 of our H1 data center to full utility power. We will first switch to our own generator power, and then perform the final cutover to utility power. At that point, this will be considered “normal” status.

We will send an update when the work has been completed.

========

Friday, June 20, 2008 2:33 PM

H1 Phase 1 has been cut over to "normal" power status: utility power with permanent generator backups. This transfer from temporary generator did not cause any downtime to customers in the data center.

Phase 2 will return to utility power in the next few hours. We will update the status when this change has been completed as well.

=======

Friday, June 20, 2008 4:49 PM

Today we have converted our H1 data center, including both Phase 1 and Phase 2, to full utility power. This means that we're operating in what we all consider "normal" mode, backed by our existing UPS and backup generators. There was no loss of power to the servers during the cut-over of either phase.

There is still what we consider "cosmetic" work to do on the interior of the data center; otherwise it's business as usual.

=======

Complete!

-- Harry
__________________
(See our Terms of Service)
-- Harry Engines Ignitions Videos Photos
Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

F o r u m Jump

Similar Threads Chosen at Random
Thread Thread Starter F o r u m Replies Last Post
Mega fire or Nova to fire Wico?? stugguls Magnetos, Ignition Coils and Spark Plugs 3 02-25-2008 10:25:59 PM
Jack Junior Electrical System Harry R. Hit & Miss Gas Engine Discussion 8 11-12-2006 06:47:53 PM
Jack Junior Electrical System Harry R. Hit & Miss Gas Engine Discussion 2 11-05-2006 08:11:58 PM
System News and Information Harry Antique Engine Archives 1 10-15-2002 12:35:47 AM


Use "Ctrl" mouse wheel to change screen size.
All times are GMT -4. The time now is 05:12:07 AM.

Smokstak and Enginads site search!


All use is subject to our TERMS OF SERVICE
SMOKSTAK® is a Registered Trade Mark - A Community of Antique Engine Enthusiasts
Copyright © 2000 - 2019 by Harry Matthews P.O. Box 5612 - Sarasota, FL 34277