Window overlap in Pandas

Question

There are several data management methods in pandas in this window (for example, pd.rolling_mean or pd.rolling_std .) However, I would like to set the window overlap, which I think is a pretty standard requirement, For example, in the following image you can see the window, spanning 256 samples and spanning 128 samples.

How to do this using optimized methods included in Pandas or Numpy?

Answer 1

Using as_strided , you would do something like this:

 import numpy as np from numpy.lib.stride_tricks import as_strided def windowed_view(arr, window, overlap): arr = np.asarray(arr) window_step = window - overlap new_shape = arr.shape[:-1] + ((arr.shape[-1] - overlap) // window_step, window) new_strides = (arr.strides[:-1] + (window_step * arr.strides[-1],) + arr.strides[-1:]) return as_strided(arr, shape=new_shape, strides=new_strides) 

If you pass a 1D array to the aforementioned function, it will return a 2D representation to that array with the form (number_of_windows, window_size) so you can calculate, for example. window value means:

 win_avg = np.mean(windowed_view(arr, win_size, win_overlap), axis=-1) 

For instance:

 >>> a = np.arange(16) >>> windowed_view(a, 4, 2) array([[ 0, 1, 2, 3], [ 2, 3, 4, 5], [ 4, 5, 6, 7], [ 6, 7, 8, 9], [ 8, 9, 10, 11], [10, 11, 12, 13], [12, 13, 14, 15]]) >>> windowed_view(a, 4, 1) array([[ 0, 1, 2, 3], [ 3, 4, 5, 6], [ 6, 7, 8, 9], [ 9, 10, 11, 12], [12, 13, 14, 15]]) 

Answer 2

I am not familiar with pandas, but in numpy you would do something like this (untested):

 def overlapped_windows(x, nwin, noverlap = None): if noverlap is None: noverlap = nwin // 2 step = nwin - noverlap for i in range(0, len(x) - nwin + 1, step): window = x[i:i+nwin] #this is a view, not a copy y = window * hann(nwin) #your code here with y 

This is torn from some old code to calculate the average PSD, which you usually process with half-intersecting windows. Note that window is the “view” in the x array, which means that it does not copy any data (very fast, so probably good), and that if you change the window , you also change the x (so don't window = hann * window ).