Just to remind yourself how MATLAB stores frequency content for Y = fft(y,N) :
Y(1) - constant offsetY(2:N/2 + 1) - many positive frequenciesY(N/2 + 2:end) is a set of negative frequencies ... (usually we will build it to the left of the vertical axis)
To create a true low pass filter, we must keep both low positive frequencies and low negative frequencies.
Here is an example of this with a multiplier rectangle filter in the frequency domain, as you did:
% make our noisy function t = linspace(1,10,1024); x = -(t-5).^2 + 2; y = awgn(x,0.5); Y = fft(y,1024); r = 20; % range of frequencies we want to preserve rectangle = zeros(size(Y)); rectangle(1:r+1) = 1; % preserve low +ve frequencies y_half = ifft(Y.*rectangle,1024); % +ve low-pass filtered signal rectangle(end-r+1:end) = 1; % preserve low -ve frequencies y_rect = ifft(Y.*rectangle,1024); % full low-pass filtered signal hold on; plot(t,y,'g--'); plot(t,x,'k','LineWidth',2); plot(t,y_half,'b','LineWidth',2); plot(t,y_rect,'r','LineWidth',2); legend('noisy signal','true signal','+ve low-pass','full low-pass','Location','southwest')

A full low-frequency fitler does a better job, but you will notice that the reconstruction is a bit βwavyβ. This is because the multiplication with the rectangle function in the frequency domain coincides with the convolution with the sinc function in the temporary domain . The convolution with the sinc function replaces each point with a very uneven average value of its neighbors, therefore, the "wave" effect.
The Gaussian filter has more pleasant properties of the low-pass filter, because the Fourier transform of the Gaussian is Gaussian . Gaussian decays to zero, so distant neighbors are not included in the weighted average during convolution. Here is an example with a Gaussian filter preserving positive and negative frequencies:
gauss = zeros(size(Y)); sigma = 8; % just a guess for a range of ~20 gauss(1:r+1) = exp(-(1:r+1).^ 2 / (2 * sigma ^ 2)); % +ve frequencies gauss(end-r+1:end) = fliplr(gauss(2:r+1)); % -ve frequencies y_gauss = ifft(Y.*gauss,1024); hold on; plot(t,x,'k','LineWidth',2); plot(t,y_rect,'r','LineWidth',2); plot(t,y_gauss,'c','LineWidth',2); legend('true signal','full low-pass','gaussian','Location','southwest')

As you can see, the reconstruction is much better.