I didn't try turning off and then on the modem, router, Ooma hub, no. My router has the Tomato firmware on it, and I don't think it has any of the firewall settings turned on by default. I don't see any check marks on the firewall settings that enable firewall restrictions, so I don't think any firewall settings are enabled right now. Nor do I see any access restrictions nor port blocking that may be enabled. And I generally don't have any problems with my router blocking or preventing any internet services from running.
My workaround for the problem is by checking my Ooma voicemail from my Uniden handset. Doing this will clear my voicemail waiting indicator lights on all 3: Ooma hub, Uniden handset, and Uniden base. So, it's not a problem with my router's firewall or settings blocking signals from getting through to the cordless handset and cordless base. But I find it rather annoying that if I check and clear my Ooma voicemail on the Ooma hub, rather than the handset, that the voicemail indicator lights won't turn off on the Uniden handset and base. Though it does turn off the voicemail light on the Ooma hub.
Other than this problem. The Ooma service works pretty well for me, with very good voice quality and reliability.