If you’re already enjoying the convenience of receiving your ooma voicemail in your e-mail or on your mobile phone – get ready for more. We’ve recently rolled out a couple highly useful enhancements to the voicemail forwarding feature:

1. New e-mail client support. We’ve expanded support for mobile e-mail clients. Now you can listen to your ooma voicemail from your Blackberry, iPhone or other mobile smartphone.

2. MP3 encoding. Voicemail attachments are now encoded in MP3 format, reducing the file size by more than half without sacrificing any quality.

These two enhancements are ready for use now. If don’t have voicemail forwarding enabled – go ahead and take it for a spin!

Voicemail forwarding is only available with ooma Premier service. If you’re interested in learning more about ooma’s enhanced voicemail features, visit ooma.com/premier.

There have been a number of discussions internally about the two outages this week. Needless to say, we are very disappointed in our handling of the situation. Although there are some external factors we don’t control, there are many that we do, and we clearly have room to improve. We’ve listened to all the feedback that the community has been kind enough to share and I want to take a moment to go over some of the steps that we are taking to improve communication about major outages and how we plan to mitigate/avoid them in the future.

Communication

• We have closed a loophole that prevented the product marketing team from being immediately notified about major outages during off-hours. This prevented us from responding in the forums in a timely manner this morning.

• We are adding a mechanism for us to display outage notifications on our homepage and Lounge login page. The notification will be displayed at the top of the home page and link to our status in the forums. This should be live tomorrow.

• We have created an official ooma_status twitter feed. You can now find us at twitter.com/ooma_status. This gives you RSS and SMS notification options as well.

• We will send outage notification e-mails to customers who are subscribed to our marketing communication list. If you are not subscribed, you can do so here: www.ooma.com/contact/. Be sure to click the “I currently own an ooma system” checkbox. This is the only opt-in list we can use right now where we can safely send tens of thousands of e-mails without getting blacklisted. In the future, we will consider allowing separate opt-in’s for outage vs marketing notifications.

Mitigation/Prevention

• We are adding a second upstream Internet provider. We have hardware on order, hope to have it live next week.

  We are doubling the capacity of our provisioning systems so we can accelerate the rate of recovery. That should be live by the end of the week.

• We have found two issues that prevented the ooma Hub from automatically recovering during the last two outages. The fix is being tested as we speak and will remove the need for a manual power cycle.

• We are adjusting our back-off and retry mechanisms to prevent the Hubs from placing undue load on our system during outage recovery. That should be complete by the end of the week.

• Longer term, we are planning to add a second data center in the Midwest/East Coast.

No doubt there will always be room for improvement, and we will continue to look to the community for feedback.

A major outage of the ooma service began today (April 13th) at approximately 11AM (pacific time). This outage affected all ooma customers and was triggered by Internet connectivity issues at our upstream service provider. While we do not have all the details around the problems that affected the Internet connection at this time, we do know that these problems were not isolated to ooma, but did effectively cut off the ooma service from the outside world.

Between 2PM and 3PM, Internet connectivity was slowly being restored to our service. However, the flood of ooma Hubs coming back online created an immense amount of load on our provisioning systems. We rushed to add capacity to the system, but the nature of the network outage had interfered with the system’s ability to recover by itself. Beginning at 2:30PM, we began throttling Hubs from connecting to our servers. This allowed the system to settle down and work through the backlog of requests. As the systems stabilized, we slowly began opening up blocks of users to connect to the ooma service.

As of 5PM, most service has been fully restored. Some Hubs may not recover automatically in an expedient manner – if your ooma service is still down, please reboot your ooma Hub. Unplug power from the back of the device, wait one minute and then plug the power back in. The ooma Hub may need to download a software update, but it should go into service shortly thereafter.

Rest assured that we are taking this outage very seriously. Discussions have already started on how to make the service resilient to a similar event in the future. ooma currently has one data center located in west coast. We have planned to light up a second data center in the midwest or east coast this year, and this outage has served as a stark reminder for us to get moving on that. This has also served as a good opportunity for us to re-evaluate our contingency and business continuity plans.

We know that phone service is critical function that everyone relies on. We apologize for the outage this afternoon and thank you for your patience as we work through some growing pains.

Update: Please refer to our latest post to read about how we plan to mitigate and handle outages in the future.