The Robot Co-op

Today (August 25, 2008) we’ll be bringing the site down around midnight PST (aka Seattle time). Todd will add columns to one of our database tables, and the sites have to come down for this to occur. This change to the database will enable some new features on 43 Things. Those features won’t roll out just yet, but we’re making the change now so we can begin to push the features and test them before they go live.

The outage should last about 60 minutes … but leave us some wiggle room. Good luck and godspeed, Todd

UPDATE: Switched off the sites and initiated the database update around 12:05 AM PST. Now we just have to wait while MySQL chews through millions of rows to add the new columns. This should complete within the hour.

12:40: First half of the update just completed, so it may be bit later than 1:00 by the time we’re back online. If the second half runs at the same speed, we should be back up and running by 1:30 PST.

1:20: Okay, database is upgraded and the websites are back up.  G’night, folks!

We’re having some server issues at the moment which we hope to have solved soon. We’ll keep you posted on progress …

update: sites are back up. We’re tweaking boxes today but we should be able to keep the sites live while we do so. I’ll let you know if things go wonky, but I think we should be alright.

We’re transferring the DNS settings for All Consuming this morning and that means it’ll be down for a little bit. The back story on this one is that allconsuming.net was registered by Buster long before the Robot Co-op was around. He and I spent many hours over the last 3 years trying to transfer the domain from Gandi to Joker (where we have 43T, 43P, 43Peeps, LOB, etc. registered). This was not easy. In fact, it was a total pain in the ass (damn you Gandi!).

But the good news is the transfer has occurred and we’re yanking the band-aid off the DNS settings today. Soon (hopefully later today) All Consuming will be humming again.

 UPDATE (Aug 8): All Consuming is back up as of a couple days ago.

7/26 UPDATE: we’ve added more RAM to the slave database and are adding 4GB more RAM to the master database on Monday.  Are people feeling things are zippier?

UPDATE: sites are back up—and things are better than before. This is game of inches on the performance front but I think we gained about 2 inches with the last buffer move. Okay, back to goaling!

UPDATE: we’re taking the sites down for 5 minutes to increase some mysql buffers—that’s a fancy way of saying hold on for just a sec and things should get much better.

Are you getting a lot of Oops pages in the last few days? You are not alone. I could make up an elaborate explanation, or simply say: we’re working on it. More specifically, we should have two new boxes (aka webservers) up by the end of the week. That may help things hum along. In addition we’re looking into other database efficiencies to deliver us from Oopsland to the promise land.

In other news I was lucky enough to make it this weekend to the Bay Area 43T Meetup in San Francisco and met some great folks. It was rewarding as a user of 43T to meet up with a bunch of smart, insightful 43T’ers. I highly recommend attending and/or organizing a 43T meetup of your own. If you do, let us know how it goes.

P.S. If you’d like to lodge your frustration about Oops-ness, feel free to do so here in the comments. You’ll feel better if you get it off your chest.

Thanks for all the email reports regarding performance issues with Lists of Bests in the last couple days. We’re making some tweaks now that should stabilize performance—and we’ll continue to work on getting the site healthy again.

Sorry for the hassle but thanks for your detailed feedback. It’s helpful to us.

We’re working on 43 People, All Consuming & Lists of Bests right now—they may be down for a few minutes to an hour as we work on some database issues. Hope to have things up before lunch!

The sites may be down periodically on Sunday April 15th from 12am – 1am-ish (Central time). Here’s the official word from our hosting company, Rackspace:

The window for this maintenance will be from 12:01 AM to 1:00 AM Central time on Sunday morning, April 15, 2007.

Rackspace will be performing upgrades and routine maintenance in the data center networking infrastructure. Due to this maintenance, there may be several brief interruptions of service while traffic fails over from one device to another. No maintenance will be performed on your specific devices but applications utilizing persistent TCP connections like SSH and RDP may lose connectivity to the internet during the traffic migration.

Hopefully it’ll go without many hiccups. Thanks for your patience.