You can use the command find:
find . -name "*'*" -delete
As @Bryan noted in his comment, this will delete all files in the current directory and in all subdirectories . If you do not want to drop directories, use:
find . -name "*'*" -maxdepth 1 -delete
find .