Scipy FFT Frequency Very Noisy Signal Analysis

Question

I have noisy data for which I want to calculate the frequency and amplitude. Samples were collected every 1/100 sec. From the trend, I believe that the frequency is ~ 0.3

When I use the numpy fft module, I get a very high frequency (36.32 / sec), which is clearly wrong. I tried filtering the data using pandas rolling_mean to remove noise before fft, but that didn't work either.

 import pandas as pd from numpy import fft import numpy as np import matplotlib.pyplot as plt Moisture_mean_x = pd.read_excel("signal.xlsx", header = None) Moisture_mean_x = pd.rolling_mean(Moisture_mean_x, 10) # doesn't helps Moisture_mean_x = Moisture_mean_x.dropna() Moisture_mean_x = Moisture_mean_x -Moisture_mean_x.mean() frate = 100. #/sec Hn = fft.fft(Moisture_mean_x) freqs = fft.fftfreq(len(Hn), 1/frate) idx = np.argmax(np.abs(Hn)) freq_in_hertz = freqs[idx] 

Can anyone advise me how to fix this?

Answer 1

You are right, something is wrong. You need to ask pandas for the null column:

 Hn = np.fft.fft(Moisture_mean_x[0]) 

Otherwise, something wrong will happen, which can be seen due to the fact that the FFT result was not symmetrical, which should be the case with real input.

Answer 2

It seems that @tillsten has already answered your question, but here are a few additional confirmations. The first graph is your data (zero average value, and I changed it to csv). The second is the power spectral density, and you can see the fat mass with a peak at ~ 0.3 Hz. I enlarged the image in the third section to see if the second latent frequency was close to the fundamental frequency.

 import pandas as pd import numpy as np import matplotlib.pyplot as plt from scipy import signal x = pd.read_csv("signal.csv") x = np.array(x, dtype=float)[:,0] x = x - np.mean(x) fs = 1e2 f, Pxx = signal.welch(x, fs, nperseg=1024) f_res, Pxx_res = signal.welch(x, fs, nperseg=2048) plt.subplot(3,1,1) plt.plot(x) plt.subplot(3,1,2) plt.plot(f, Pxx) plt.xlim([0, 1]) plt.xlabel('frequency [Hz]') plt.ylabel('PSD') plt.subplot(3,1,3) plt.plot(f_res, Pxx_res) plt.xlim([0, 1]) plt.xlabel('frequency [Hz]') plt.ylabel('PSD') plt.show() Hn = fft.fft(x) freqs = fft.fftfreq(len(Hn), 1/fs) idx = np.argmax(np.abs(Hn)) freq_in_hertz = freqs[idx] print 'Main freq:', freq_in_hertz print 'RMS amp:', np.sqrt(Pxx.max()) 

Fingerprints:

 Main freq: 0.32012805122 RMS amp: 0.0556044913489 

Answer 3

FFT is a filter bank. Just find the peak value only in the expected frequency range as a result of the FFT (instead of the entire result vector), and most of the other spectrum will be essentially filtered out.

Answer 4

There is no need to pre-filter the signal, because the FFT is a filter. Just skip the parts of the FFT that correspond to the frequencies that you know to contain a lot of noise - zero them out or otherwise exclude them.

Answer 5

Hope this can help you.

http://wiki.scipy.org/Cookbook/ButterworthBandpass

You should only filter the range around the expected frequency and improve the signal noise ratio before applying the FFT.

Edit:

Mark Ransom gave a more reasonable answer, if you need to do an FFT, you can just cut off the noise after the transformation. This will not give a worse result than a filter.

Answer 6

You should use a low-pass filter, which should contain large periodic changes and first smooth out some of the materials with a higher frequency. After that, you can do FFT to get peaks. Here is the recipe for the FIR filter that is commonly used for this particular thing.