You did not say this, but I suspect that you are grepping through a directory tree.
This may not be the most elegant solution, but you can use the "find" output.
I often find myself recursively grepping a directory tree as follows:
grep <needle> `find . \( -name '*.cpp' -o -name '*.h' \) -print`
You could do something like:
grep <needle> `find . \! -name '.??*swp' -print`
source share