There is a fairly reliable and efficient way to both reject the noisy points and determine the transformation between your points of interest. The algorithm that is commonly used to reject outliers is known as RANSAC (http://en.wikipedia.org/wiki/RANSAC), and the algorithm used to determine the conversion can take several forms, but the latest technology is known as the five-point algorithm and can be found here - you can find the MATLAB implementation. Note that you need to determine the transformation, even if you do not need the exact rotation between two images - this is how outliers are identified.
Unfortunately, I do not know about the mature implementation of both of them; you may have to do some work to implement RANSAC and integrate it with the five-point algorithm.
OpenCV implements an implementation that is redundant for your task (which means that it will work, but will take more time than necessary), but is ready to work out of the box. The function of interest is called cv :: findFundamentalMat (http://opencv.willowgarage.com/documentation/cpp/camera_calibration_and_3d_reconstruction.html#cv-findfundamentalmat)
source share