Give ScrollViewer a background. If the background is transparent, it will not catch any mouse events.
This is true for all controls. If you try to catch mouse events, the control must have a background or controls below, it will receive mouse events.