I suppose this refers to the fact that shadows can be easily (more or less) calculated from the vertices (corners) of a silhouette, while visibility requires checking all possible rays of light that can bounce off an object (and not just its edges).
Thus, the number of rays for shadows is much less than the number of rays for determining visibility.
It makes sense? I could paint a picture ...
source share