Are we talking MPEG2 or MPEG4? Packet loss will give different visual artifacts, being worse with MPEG4. Usually with MPEG4, you go back to the base frame (is that an I frame? not sure anymore of the correct terminology :)), so visualy you have a lot of pixelisation, then an old frame and your back to the stream at the next I-frame.
Are the fast moving images of sport channel producing a higher bitrate stream? (are you re-encoding the streams?).
Check your buffer usage, and eventually try to adapt it on the required port if needed.