You have dynamic libraries installed, but do you have a development package? You will probably need a file or a symbolic link with the name:
libfftw3.so
You may need to install the fftw3-devel package (or fftw3-dev).
Also try removing the lib prefix and the .so.3 suffix:
FIND_LIBRARY(FFTW3_LIBRARIES NAMES fftw3 libfftw3)
If this does not work, try adding the PATHS argument:
FIND_LIBRARY(FFTW3_LIBRARIES NAMES fftw3 libfftw3 PATHS /usr/lib <other paths>)
Make sure that the CMAKE_FIND_ROOT_PATH variable is set correctly (apparently you are not cross-compiling, so it is probably empty and cmake will use reasonable places to search for libraries).
See the cmake man page for a detailed use of the find_library function.
Finally, look at this site for how to write a find_package script: http://www.cmake.org/Wiki/CMake:How_To_Find_Libraries
source share