A major outage of the ooma service began today (April 13th) at approximately 11AM (pacific time). This outage affected all ooma customers and was triggered by Internet connectivity issues at our upstream service provider. While we do not have all the details around the problems that affected the Internet connection at this time, we do know that these problems were not isolated to ooma, but did effectively cut off the ooma service from the outside world.
Between 2PM and 3PM, Internet connectivity was slowly being restored to our service. However, the flood of ooma Hubs coming back online created an immense amount of load on our provisioning systems. We rushed to add capacity to the system, but the nature of the network outage had interfered with the system’s ability to recover by itself. Beginning at 2:30PM, we began throttling Hubs from connecting to our servers. This allowed the system to settle down and work through the backlog of requests. As the systems stabilized, we slowly began opening up blocks of users to connect to the ooma service.
As of 5PM, most service has been fully restored. Some Hubs may not recover automatically in an expedient manner – if your ooma service is still down, please reboot your ooma Hub. Unplug power from the back of the device, wait one minute and then plug the power back in. The ooma Hub may need to download a software update, but it should go into service shortly thereafter.
Rest assured that we are taking this outage very seriously. Discussions have already started on how to make the service resilient to a similar event in the future. ooma currently has one data center located in west coast. We have planned to light up a second data center in the midwest or east coast this year, and this outage has served as a stark reminder for us to get moving on that. This has also served as a good opportunity for us to re-evaluate our contingency and business continuity plans.
We know that phone service is critical function that everyone relies on. We apologize for the outage this afternoon and thank you for your patience as we work through some growing pains.
Update: Please refer to our latest post to read about how we plan to mitigate and handle outages in the future.
Loading ...
Well, it looks like we can see how Ooma has learned from the April 13 outage since it looks like there’s an April 15 outage as well. I love Ooma’s service and am willing to show some patience, but two major outages in three days is putting that to the test.
Yeah, after getting my service backup for about 10 hours yesterday, I’m down again. But that is ok since it’s free and I’m saving lots of money, yahoo! Just like not having any food in the house to eat, yeah I’m saving money but I’m also starving! Rethinking the whole OOMA idea!
Place a status link that can be easily seen right on your home page (ooma.com).
Two days later, and another outage here in NJ. This is not looking good.
7:30am EST @ NJ
I am happy to know it wasn’t my equipment!
But, today I have No OOMA 4/15/09 7:35 am. I hope it is “your equipment” and not mine. But, as mentioned above, I would hope you would learn and post a notice this time. (If it is “your” equipment and not “mine”).
One other note – as of 7:37AM Eastern, the official status report in the ooma forum indicates no problem with the service, more than an hour after the first reports of outages showed up on the forum.
We’re still down in Michigan. Please get the Hubs back up and working
Ooma is becoming Un-reliable day by day..good thing ever since I became Ooma Customer, I kept my Vonage account active too and ever ported the number to Ooma.
If there is more breakdown in service, I am out of Ooma, If a company is charging $250 for Ooma unit, $99 for premier service and $40 for number porting, they need to ENSURE an un-interruptable service
What happens if someone has an emergency to call 911 and they do not have cell phone and Ooma is down?
I have today problem ??IF anybody has this problem again
An e-mail notice would be nice when there is an outage. I spent some time checking everything on my end.
On the other hand the phone company would not send out a message when the phones are down and would give me all kinds of grief to send out a tech to see if it is a customer problem or not.
If this is all I have to put up with from time to time I will be happy.
Before Ooma I had quest. The phones would quit working every time it rained. Fortunately I live in the desert so that was not too often. On the other hand, this areas phones are still wired with post war tech, (WW I) and the call quality was very poor.
06:30AM 2009.4.15 – OOMA service still out.
Is the Ooma service down again this am? At 5:15am ET my Ooma was not working again. It came back up after the outage on the 13th, but now appears to be down again. I unplugged and reset Ooma as well as the modem, and still have no service. I think if this continues the best is to return this to Best Buy, and request Ooma to refund the money for porting charges before the numbers are switched over. I still do not see any updates on the website regarding any outages today. Anyone else experiencing an outage of Ooma
Is service down again this morning?
My ooma phone service was inoperable yesterday, came back up late in the day, but is down again on Wednesday the 15.
My guess is you still have some problems…
Is the service still down? I still have no connection.
Still have the same problem like yesterday,it seems like nothing is fixed yet.
Here we go again. Ooma is down, no status message at the company’s site. Only way to verify is Twitter:-(
Once again Ooma is down and has been for a few hours now but it will be another 2 hours until you can reach someone at Ooma since their support doesn’t open until 8am PST.
Is there any system in place to monitor things after hours or are we looking at another extended outage (the 2nd in 2 days?)
Is the OOMA network down again today? The status light has gone red on my OOMA box this morning. Tough to carry on daily life without phone service. Would you please share an update? Not sure if it is just me or all again.
Thx.
I am very very upset about this. It took me some time to convince my housemate to allow me to hook up the ooma to the dsl which she pays for. I convinced her that it was perfectly safe.
Now a few months later, with her calls ringing only on my phone she is pissed. I tried pulling the wool over her eyes because she was out of town when it first happened but I don’t think she is buying it.
I had to turn off the ooma and remove it completely from the system this morning because it went down again. I want my money back.
Believe me, this has scared a lot of your potential customers away. My # will be completely ported sometime this week and I am already second guessing my decision on Ooma! I was (mostly) on the fence to begin with. Ooma, you just cannot afford something like this — period. These are the times when only the fittest will survive. And, mind you, that is not mutually exclusive with being lean and robust. I am just one such experience away from switching back to Vonage.
And yes, I completely agree with Drew’s comment on having a very prominent Status Page (rather than burying the status in some obscure forum which most customers don’t visit or are not savvy enough to navigate to). Think from a user’s perspective who is lay.
Look folks, I work in a data center and if this company has only one upstream provider they are asking for trouble.
One of the fundamentals of services like this is redundancy. Multiple routers, multiple switches, and certainly multiple bandwidth providers. To provide reliability of service, which is what you deserve, they need redundancy of power, access (bandwidth) and location.
I can’t believe they have all of their equipment and are serving the whole nation from one data center on the west coast. Calls from the east must have terrible latency.
If they don’t have such redundancy and are in only one data center it must be for one of two reasons… they are trying to save money…. or they are stupid.
Someone should be fired.
SEND AN EMAIL!!! Personally, I am very understanding of network outages. No one is immune to this sort of thing. Certainly there are things you can do increase your network redundancy, but failures will still occur. I do have a HUGE problem with the lack of communication. I haven’t read all these comments but I read a few and just have to add my .02. Send an email! You have email addresses for most of your customers. A simple, “Sorry, your Ooma service is temporarily down” message (and then another letting me know when it’s up) would be VERY proactive and alleviate a LOT of my concerns. Yes, post it on your homepage, but the most proactive thing to do is to notify US rather than the customer needing to seek you out by calling or going to your web site to find out there’s a problem.
Service is down again?
Day 3 and my phone still doesn’t work. I have had good service until now and it seems that the outage is not completely resolved. Would be nice if OOM could post more up to date information. Anyone else still having trouble?
I congratulate myself for signing up for ooma at least once a month when I’m paying my bills.
BEST THING SINCE SLICED BREAD.
I’m confident this will be a positive learning experience for ooma product development.
Happy Camper
I understand that technology at some point will have some issues so the first time it happened I let it go. Today it happened again. I was without service all morning. I want to know If this is gonna continue this way so I can take the equipment back to the seller. I want a phone service that is reliable.
sometimes small frustrations get blown out of perportion i guess. some of us remember when we were happy to have a phone with a dial. dialing gave you the chance to think about what you were going to say. so think about what you are saying before whining about this service with no phone bills.
The dialer…….
This outage was really not a big deal if you know about internet phone service, try Vonage, Sunrocket, or Verizon? Even then, I think everyone should give OOMA a break. I do feel that since they were able to send their customers an email about what happened after the fact, why not send it before or during the outage? Most everyone didn’t think or know to go to the forums and so were further frustrated by getting an email after they gave up trying to reach customer service, creating a backlog. I think the bottom line here are expectations from inexperienced customers. Try the above companies and you’ll realize OOMA deserves a break! Yes, the call center and communication can be better but I find myself saying the same thing about the guys above.
Overall though, the OOMA phone call quality is quite impressive and inexpensive (free) and I’ll take that any day over the other yahoo’s.
Service is down again…and I can’t log into the lounge. Frustrating.
I have to admit I’m really concerned about this situation with the outages. I just setup my system on 4/11 and requested my number to be ported and my ooma hub is constantly flashing red and OOS. Outages on 4/13 AND 4/15 in Indy – ooma has been out of service for me longer than it has actually worked.
Why is there no notification of the April 15th outage? Did you not learn anything from April 13th? There is nothing posted anywhere on this site about today’s situation and of course the customer service lines are a continuous busy signal.
YOU MUST NOTIFY CUSTOMERS OF OUTAGES!!
1) Immediately post a notice on the ooma website on the front page and the lounge login page and the support page as soon as you receive reports of outages.
2) Send out an email to all of your customers letting us know you are doing something about this, TODAY, APRIL 15th.
I will return my system next weekend if ooma does not respond to its customers by communicating issues to us. Isn’t it someone’s job at ooma to post notifications on the website or to email customers? Finally, how hard would it be to put a recording on your customer service phone line? If you do not have that technology, I definitely made a mistake.
This is a minor blip in the system that can be corrected. There really is no point in getting all bent out of shape over this.
However, having said that and just an FYI for omma, I had to reset my ooma system the morning of the 14th (around 2AM) and then again this morning (9AM) as the system was down twice. The base station came up fine yesterday, but this morning it took a very long time. It was rather frustrating and I was starting to wonder if the unit was busted. So, I would personally like to see some voice messages from the unit explaining why it is not coming up and to call support for more info or give us a web -page we can access to help us better understand what is going on. A few lights that flash and/or remain on is not very helpful.
Thanks ooma for providing an inexpensive alternative to the very expensive Corporate Phone service vendors!
Well, that’s it for me…I sent my ooma back today. So sorry I paid for porting my number. I’ve had nothing but problems. Going back to ATT.
Goodbye and Good luck everyone. Be warned…this is not everything it is cracked up to be.
I have no problem with ooma. every system goes down at times. get used to it. Just keep thinking how much you are saving, no more phone bills! I love it!!!
ooma is great ive had it for over a year and its been rock solid and i have saved over $500. Ooma keep up the good work!
i found out i did not have service this morning (wednesday) when i had several people call my cell phone wondering why they were being immediately transfered to voicemail on my ooma line. i had no service for the better part of the day and even when i went to the ooma website to find out if there was some problem, there was no information on the outage on the website. needless to say, i thought this problem was limited to just me. when i did get a dial tone later in the day, every call that came in kept on being disconnected after only a few minutes of the call. i had family calling me from overseas and their calls were disconnected. this has been a very unpleasant experience for me.
If you are not paying for the service, do not complain, but in case the system is down Ooma should forward the calls to an other number or send us a e-mail regarding the status of the voice message. I will suggest that charge us a small fee for the maintenance of the system, so you are accountable for the service. We paid a high price for the equipment so the service should be there. Please in the future let us know what is happing. Keep up the good work. I am not paying, so I have no complaints.
Communication is always the key. I think that those customers that are made would be less mad if ooma had sent out an email and a text message to all their customers. I would have appreciated that because I did spend some time wondering if the problem was with my equipment or system wide. I figured it was system wide when I could not even log in to my ooma lounge account.
I still like ooma, as long as the reliability does not have repeated letdowns I look forward to enjoying the good service and the savings for a long time to come.
Email notification(which can be solved by the forum subscription) and network unavailabitliy forwarding would solve this problem…mistakes happen…just make sure you learn from them
Ooma (soley)user 1 yr and loving it
From what I understand – this outage took EVERYBODY out.
There was no ATM service, no cell service, and no phone service for EVERYONE on the affected area. It didn’t matter who your provider is – you were in the dark.
Even Credit Card terminals in supermarkets were down!! Most people had to walk away from their shopping carts. No one – myself included – had enough cash. It was like a wacky Sci-Fi movie…
I had my ooma rigged in “fail over mode” so it could (theoretically) use a land line in a disaster. AT&T was down too.
This outage was a lot bigger than ooma and had everybody scratching their heads…
-Steve
Folks: To all out there upset about this, you need to remember two things:
1. You are getting a FREE VoIP service. This is the only company out there doing it and starting up something like Ooma is no small feat. A little slack is appreciated especially considering Ooma’s percentage down is a lot better than many other VoIP providers.
2. FACT OF LIFE: All VoIP goes down. A lot. Ooma has not in the time I have had it except for this date. To those complaining of no fail safe: EVERY SINGLE VOIP COMPANY STATES THAT IT IS RECOMMENDED YOU HAVE A REGULAR PHONE LINE FOR BACKUP USE DURING INTERNET INEVITABLE CONNECTIVITY OUTAGES and the Ooma is setup well for this condition. The very very large company I work for uses VoIP across it’s entire worldwide network and it goes down daily in at least one office somewhere around the world, and we pay dearly for the service. My Ooma at home is much more reliable and drastically cheaper.
Anyway, life goes on… Don’t hold Ooma to any more expectation than you would any other VoIP system. It is clearly stated in the manual that service cannot be provided by an internet outage no matter what the cause. YOU and I (the idiot end user) need to come up with a plan for when it’s out and enjoy the free phone service when it’s up, which is a huge percentage of the time. Most of the population has a cell phone plan capable of picking up the slack, and for those that don’t you can pay a telecom $13 a month for a basic line if you simply must have a backup line. The sense of entitlement the average person has in the US is thoroughly absurd. I’d hate to see the reaction of the complainers to a real problem.
I don’t hold it against OOMA – love OOMA – these things happen – you guys are doing a great job. Keep up the good work.
Look people, where can you find a company that is willing to listen to your suggestions, they are doing a great job, remember – don’t complain – make suggestions, thats what this is all about…you ask them to put a banner on the Home Page – they do it. Its called innovation.
I have been using Ooma since December and could not be happier. Sure it might have a drawback or 2 (at times). HOWEVER, the service is FREE and I like that. When the service went out….no big deal. I have a cell phone. My DSL with a big carrier starting with a V goes out from time to time. Its technology. Its not perfect. If you want home phone service that is 99.9% reliable then get it with MaBell and pay through the nose and don’t forget the taxes. I love you Ooma and so does my family. We support you 100%. We own 2 so far and I know more are in our near future.Thank you!!!!!
Get a clue people. Shit happens.
OOMA still rules and I appreciate the honesty.
I think the best way to prevent future outages is to create more data centers across the US in all of its time zones and geographical regions, and build it on a network basis. Ooma must strengthen its infrastructure for its current and future users.
Set my OOMA up on Apr.12,09 and received very good Tech Support.I did not have to wait on line a long time and I spoke to an english speaking person that I could understand.For me OOMA is a blessing.I am 70yrs.old and living on a fixed income with today’s economy I am trying to save every way I can.I did not notice the outage or problem Apr.13.But I am not expecting perfection from any company in this imperfect world.As long as the Company is working to improve I stand with them 100%.My OOMA has done everything that it was advertised to do.So I say to the company “Keep on keeping on”I love my OOMA.
Ooma rocks! I love this service. I was paying almost $50 for my long distance to Cox Telephone now I can save that money. Thanks Ooma!