Need extra help installing your Ooma Hub or Telo system? Let us know.
#608 by johnboy
Tue Sep 16, 2008 3:55 am
Hi Folks:

My system was down for 8-10 hours... from 9pm Eastern (9/15/08) through 7:45am on 9/16/08. Flashing red on the hub. I reset the hub several times with no success. No one to call at OOMA. No System messages posted.

a. What was going on?
b. Is there a 24 hr help line for something like this?

John
#610 by trim81
Tue Sep 16, 2008 5:10 am
I am up here in Los Angeles county....never had ANY downtime with Ooma..

Did you try plugging the Oooma directly behind the modem?
#615 by Bobby B
Tue Sep 16, 2008 11:37 am
We did have an outage related to the ooma Lounge and Activation last night. We're currently taking a look to see if any ooma Hubs were affected last night. I'll update the service update page with the details.

-Bobby
#617 by SivRami
Tue Sep 16, 2008 1:08 pm
Bobby B wrote:We did have an outage related to the ooma Lounge and Activation last night. We're currently taking a look to see if any ooma Hubs were affected last night. I'll update the service update page with the details.

-Bobby


Ah, my ooma hub must have been one of those affected then!
#618 by trim81
Tue Sep 16, 2008 4:11 pm
SivRami wrote:
Bobby B wrote:We did have an outage related to the ooma Lounge and Activation last night. We're currently taking a look to see if any ooma Hubs were affected last night. I'll update the service update page with the details.

-Bobby


Ah, my ooma hub must have been one of those affected then!



Those that were down: are you guys up and running now?
#619 by johnboy
Tue Sep 16, 2008 4:45 pm
I was back up this morning. All is well. Would like to know what the problem was and that it was addressed. More importantly, not posts on line to let me know what was going on and when ETA of repair was expected. No tech support available. I have been "up" without incident for months and months. Not having someone to call and report the problem and having no internet report on time for repair is not good!!!

John
#621 by Dennis P
Tue Sep 16, 2008 6:01 pm
Hi John,

I apologize for your downtime. It is inexcusable for an outage like that to occur. As Bobby has indicated in his status post, the problem affected Hubs that were trying to register to the ooma network. This includes new installations and Hubs which were power-cycled. Unfortunately the mode of failure was a new one for us, so our automated testing and monitoring systems did not alert us to the problem and we did not discover the problem until this morning.

We did what was considered a minor upgrade of our database last night which triggered a bug in the open source database that we use. Because the bug was in the database itself, not the code that we deployed, the failure modes were different than the validation tests we ran after the upgrade (which tested our code).

We're doing a few things to avoid a repeat of such a scenario. First, we are upgrading our database to avoid the bug that tripped us up last night. Second, we are expanding our post-upgrade validation tests to encompass all common usage scenarios, regardless of whether the upgrade touches those functions. Third, we will expand our automated testing and monitoring systems to look for more modes of failure so that we can respond immediately.

Thank you for your patience and understanding. Hopefully I will not have to ask for forgiveness in the future. :)

Dennis

Who is online

Users browsing this forum: No registered users and 9 guests