This is happening all of the time. Here are my results from http://www.whichvoip.com/voip/speed_test/ppspeed.html
VoIP test statistics
Jitter: you --> server: 1.5 ms
Jitter: server --> you: 5.6 ms
Packet loss: you --> server: 0.0 %
Packet loss: server --> you: 0.0 %
Packet discards: 0.0 %
Packets out of order: 0.0 %
Estimated MOS score: 4.0
Speed test statistics
Download speed: 1452520 bps
Upload speed: 351384 bps
Download quality of service: 84 %
Upload quality of service: 92 %
Download test type: socket
Upload test type: socket
Maximum TCP delay: 192 ms
Average download pause: 10 ms
Minimum round trip time to server: 91 ms
Average round trip time to server: 93 ms
Estimated download bandwidth: 16000000bps
Route concurrency: 11.015339
Download TCP forced idle: 65 %
Maximum route speed: 5761312bps
Or it goes silent and you can't hear the other person?
Sometimes with voip, there is NO noise between conversation, could this be your problem?
You might want to check with your provider to see if quality can be improved.
Also, power cycling (turn off, then on) your modem can make a difference if you haven't in awhile.
Cycling the power also does nothing.