5 posts tagged “internets”
Think you are having a bad day... These guys are having a much worse one...
UPDATE: 5:00 P.M., Wednesday, July 25, 2007
A complete investigation of the power incident continues with several specialists and 365 Main employees working around the clock to address the incident.
Generator/Electrical Design OverviewThe San Francisco facility has ten 2.1 MW back-up generators to be used in the event of a loss of utility. The electrical design is N+2, meaning 8 primary generators can successfully power the building (labeled 1-8), with 2 generators available on stand-by (labeled Back-up 1 and Back-up 2) in case there are any failures with the primary 8.
Each primary generator backs-up a corresponding colocation room, with generator 1 backing up colocation room 1, generator 2 backing up colocation room 2, and so on.
Series of Electrical EventsThe following is a description of the electrical events that took place in the San Francisco facility following the power surge on July 24, 2007:
- When the initial surge was detected at 1:47 p.m., the building’s electrical system attempted to roll all colocation rooms to diesel generator power.
- Generator 1 detected a problem in its start sequence and shut itself down within 8-10 seconds. The cause of the start-up failure is still under investigation though engineers have narrowed the list of suspected components to 2-3 items. We are testing each of these suspected components to determine if service or replacement is the best option. Generator 1 was started manually by on-site engineers and reestablished stable diesel power by 2:24 p.m.
- After initial failure, Generator 1 attempted to pass its 732 kW load to Back-up 1, which also detected a problem in its start sequence. The exact cause of the Back-up 1 start sequence failure is also under investigation.
- After Generator 1 and Back-up 1 failed to carry the 732 kW, the load was transferred to Back-up 2 which correctly accepted the load as designed.
- Generator 3 started up and ran for 30 seconds before it too detected a problem in the start sequence and passed an additional 780 kW to Back-up 2 as designed.
- Generator 4 started up and ran for 2 seconds before detecting a problem in the start sequence, passing its 900 kW load on to Back-up 2. This 900kW brought the total load on Back-up 2 to over 2.4 MW, ultimately overloading the 2.1 MW Back-up 2 unit, causing it to fail. Generator 4 was manually started and brought back into operations at 2:22 p.m. Generator 4 was switched to utility operations at 7:05 a.m. on 7/25 to address an exhaust leak but is operational and available in the event of another outage.
- Generators 2, 5, 6, 7 and 8 all operated as designed and carried their respective loads appropriately.
- By 1:30 p.m. on Wednesday, July 25, after assurance from PG&E officials that utility power had been stable for at least 18+ continuous hours, 365 Main placed diesel engines back in standby and switched generators 2,5,6,7, 8 to utility power.
Customers in colocation rooms 2, 4, 5, 6, 7 & 8 are once again powered by utility, and are backed up in an N+1 configuration with Back-up 2 generator available.
Generators that had failed during the start-up sequence but were performing normally after manual start (1 & 3) continue to operate on diesel and will not be switched back to utility until the root causes of their respective failures are corrected.
Other DiscoveriesIn addition to previously known affected colocation rooms 1, 3 and 4, we have discovered that several customers in colo room 7 were affected by a 490 millisecond outage caused when the dual power input PDUs in colo 7 experienced open circuits on both sources. A dedicated team of engineers is currently investigating the PDU issue.
Next Steps
- Determine exact cause of generator start-up failure and PDU issues through comprehensive testing methodology.
- Replacements for all suspected components have been ordered and are en route.
- Continue to run generators 1 & 3 on diesel power until automatic start-up failure root cause is corrected.
- Continue to update customers with details of the ongoing investigation.
Regards,
Marcy
Marcy Maxwell
Vice President, Security
365 Main Inc.
“The World’s Finest Data Centers”
Additional Reading:
1. Did you ever have a domain name?
I have several and still do.
2. When did you first start blogging?
My LiveJournal is about to have its 6th year anniversary on Feb 25th. However, it is a journal and not a blog, but the differentiation is weak these days in mainstream blogging.
3. What is your favorite website?
I can't answer this! I
guess I don't favor websites in general -- it's like having a favorite
dynamic page in an encyclopedia. There is no way I could choose. xkcd has been keeping me company lately and I try to give my Flickr account attention. Last.fm is a great idea and discogs, imdb, and wikipedia are indispensable.
4. What is your preferred search engine, yahoo, google, etc?
Google, mostly because it is in my search bar in both Safari and Firefox.
5. Link: your favorite site.
A link: xkcd. Isn't this the same as question 3?
WEB 2.0 THINKING GAME
comment excerpts
Mike Cherim said on October 17th, 2006 at 11:01 am:
Web 1.0: Spam emails
Web 2.0: Spam emails and spam commentsWeb 1.0: Post copyrighted content
Web 2.0: Download and redistribute copyrighted contentWeb 1.0: Rounded corners are new and kinda cool
Web 2.0: Rounded corners are confused with Web 2.0
Shashwat Nagpal said on October 17th, 2006 at 12:45 pm:
Very impresive write up!
Web 1.0: Poor standard support for Browsers
Web 2.0: Poor standard support for Browsers
Josh said on October 17th, 2006 at 9:31 pm:
...
Web 1.0 What’s the easiest way to build a site for cheap?
Web 2.0 How can I build an internet for free?
Andrew said on October 17th, 2006 at 2:59 pm:
Web 1.0: Content is everything
Web 2.0: Someone else’s content is everything
Charlie said on October 17th, 2006 at 7:32 am:
Web 1.0: Large expanses of default-white dotted with unformatted text
Web 2.0: Large expanses of white with a gradient at the top dotted with meticulously formatted and precisely positioned text
Matt said on October 17th, 2006 at 8:05 am:
Web 1.0: “Under Construction”
Web 2.0: “Beta”
Paul Edmonds said on October 17th, 2006 at 10:17 am:
Web 1.0 The bigger the bevel the more important the button
Web 2.0 The shinier the plastic the more important the button
Paul R. Redmond said on October 17th, 2006 at 11:38 am:
Web 1.0 - Meta Tags
Web 2.0 - SEOWeb 1.0 - Looking for an experienced designer. Must be able to design and implement websites using HTML, CSS. Knowledge of Javascript a bonus.
Web 2.0 - Looking for an experienced designer. The candidate must be affluent in XHTML, CSS, Javascript, DHTML, AJAXY GOODNESS, Ruby on Rails, PHP, JSP, SQL, MySql, ASP.net, XML, Actionscript (2.0 and 3.0), Adobe CS2 products, Dreamweaver, Coldfusion, Quark, eCommerce, SEO, linux, unix, IIS.Web 1.0 - Google Ninja
Web 2.0 - AJAX/Web Ninja*Web 1.0 - Actually design websites
Web 2.0 - Attend and host hundreds of conferences*I never got the whole freaking ninja thing? There are too many geeks on the web.
Jordan Ryan Moore said on October 17th, 2006 at 11:47 am:
Web 1.0:
<code><P><FONT COLOR=green SIZE=4>Header</P></FONT></code>
Web 2.0:
<code><h2 class=”sIFR-replaced”><embed style=”width:200px; height:40px;” class=”sIFR-flash” type=”application/x-shockwave-flash” sifr=”true” bgcolor=”#FFFFFF” wmode=”" flashvars=”txt=Header&textcolor=#080&w=200&h=40″ quality=”best” src=”/swf/cool-font.swf” height=”40″ width=”200″><span class=”sIFR-alternate”>Header</span></h2></code>
will chatham said on October 17th, 2006 at 1:14 pm:
Web 1.0: a,e,i,o,u
Web 2.0:
wow, awesome blast from the past. i was modifying my résumé for an online-oriented employer, and out of curiousity i wanted to see how long Prodigy had been around (or *P, for those who remember) to see if i had gotten my dates/years straight in my head. for those keeping track, i was on around 1991.
i ended up switching to AOL (version 1.6!) at around 1993 because they had ftp access without needing a shell account (which meant first web site!), it used the windows 3.1 UI so i didn't have to download a whole new gui every time i wanted to look at something else, and it also had archie/veronica searchable access so i could get guitar tab from sunet.se. or maybe that was a little later? i do remember the stigma of having been an aol.com email holder by internet "elitists" however (then mostly compuserve and university students). i think i had flashed my 2400 baud to 33.6 by then. does that sound right? or maybe it was 1200 to 2400.
so in 1995ish i got my first "real" dial-up service from netcom, who was subsequently purchased by mindspring, who later merged with earthlink. at least my email address didn't change through that time. enter blazing-fast 56k modem.
the rest is uneventful: earthlink DSL in 1998, interquest/ygnition T1 in 2002, and now a cable internet connection.
uh, yeah, that's enough about internet tubes for now.
* i had set up accounts for everyone in my family, but i was the only one who used the service. hence, the lowly "D" account.
huntin' us some errant IP
