system down

Need extra help installing your Ooma Hub or Telo system? Let us know.
Post Reply
johnboy
Posts: 561
Joined: Tue Apr 29, 2008 4:43 pm

system down

Post by johnboy » Tue Sep 16, 2008 3:55 am

Hi Folks:

My system was down for 8-10 hours... from 9pm Eastern (9/15/08) through 7:45am on 9/16/08. Flashing red on the hub. I reset the hub several times with no success. No one to call at OOMA. No System messages posted.

a. What was going on?
b. Is there a 24 hr help line for something like this?

John

trim81
Posts: 262
Joined: Thu Jul 31, 2008 9:14 pm

Re: system down

Post by trim81 » Tue Sep 16, 2008 5:10 am

I am up here in Los Angeles county....never had ANY downtime with Ooma..

Did you try plugging the Oooma directly behind the modem?

User avatar
Bobby B
Ooma Moderator
Posts: 1457
Joined: Mon Feb 18, 2008 8:41 pm
Location: Palo Alto, CA
Contact:

Re: system down

Post by Bobby B » Tue Sep 16, 2008 11:37 am

We did have an outage related to the ooma Lounge and Activation last night. We're currently taking a look to see if any ooma Hubs were affected last night. I'll update the service update page with the details.

-Bobby
Bobby B

SivRami
Posts: 7
Joined: Wed Apr 30, 2008 5:29 pm

Re: system down

Post by SivRami » Tue Sep 16, 2008 1:08 pm

Bobby B wrote:We did have an outage related to the ooma Lounge and Activation last night. We're currently taking a look to see if any ooma Hubs were affected last night. I'll update the service update page with the details.

-Bobby
Ah, my ooma hub must have been one of those affected then!

trim81
Posts: 262
Joined: Thu Jul 31, 2008 9:14 pm

Re: system down

Post by trim81 » Tue Sep 16, 2008 4:11 pm

SivRami wrote:
Bobby B wrote:We did have an outage related to the ooma Lounge and Activation last night. We're currently taking a look to see if any ooma Hubs were affected last night. I'll update the service update page with the details.

-Bobby
Ah, my ooma hub must have been one of those affected then!

Those that were down: are you guys up and running now?

johnboy
Posts: 561
Joined: Tue Apr 29, 2008 4:43 pm

Re: system down

Post by johnboy » Tue Sep 16, 2008 4:45 pm

I was back up this morning. All is well. Would like to know what the problem was and that it was addressed. More importantly, not posts on line to let me know what was going on and when ETA of repair was expected. No tech support available. I have been "up" without incident for months and months. Not having someone to call and report the problem and having no internet report on time for repair is not good!!!

John

User avatar
Dennis P
Ooma Moderator
Posts: 1858
Joined: Fri Apr 25, 2008 9:09 am
Location: Palo Alto, CA
Contact:

Re: system down

Post by Dennis P » Tue Sep 16, 2008 6:01 pm

Hi John,

I apologize for your downtime. It is inexcusable for an outage like that to occur. As Bobby has indicated in his status post, the problem affected Hubs that were trying to register to the ooma network. This includes new installations and Hubs which were power-cycled. Unfortunately the mode of failure was a new one for us, so our automated testing and monitoring systems did not alert us to the problem and we did not discover the problem until this morning.

We did what was considered a minor upgrade of our database last night which triggered a bug in the open source database that we use. Because the bug was in the database itself, not the code that we deployed, the failure modes were different than the validation tests we ran after the upgrade (which tested our code).

We're doing a few things to avoid a repeat of such a scenario. First, we are upgrading our database to avoid the bug that tripped us up last night. Second, we are expanding our post-upgrade validation tests to encompass all common usage scenarios, regardless of whether the upgrade touches those functions. Third, we will expand our automated testing and monitoring systems to look for more modes of failure so that we can respond immediately.

Thank you for your patience and understanding. Hopefully I will not have to ask for forgiveness in the future. :)

Dennis

Post Reply