Check for duplicates in the terminal?

Question

The following code prints to me a long list of files with hashes and file names

md5sum *.java

I tried unsuccessfully to list the lines that contain identical hashes so that I can delete identical files.

How can you filter and delete identical files having the same content?

Answer 1

fdupes and less duplicate viewing

Use fdupeswhich is a command line program like

fdupes -r /home/masi/Documents/ > /tmp/1 
less -M +Gg /tmp/1

temp. less . fdupes Wikipedia . OSX apt-get Linux.

fdupes

Run

fdupes -rd /home/masi/Documents

, , , :

Set 4 of 2664, preserve files [1 - 2, all]: all

   [+] /home/masi/Documents/Exercise 10 - 1.4.2015/task.bib
   [+] /home/masi/Documents/Exercise 9 - 16.3.2015/task.bib

[1] /home/masi/Documents/Celiac_disease/jcom_jun02_celiac.pdf
[2] /home/masi/Documents/turnerWhite/jcom_jun02_celiac.pdf

Set 5 of 2664, preserve files [1 - 2, all]: 2

   [-] /home/masi/Documents/Celiac_disease/jcom_jun02_celiac.pdf
   [+] /home/masi/Documents/turnerWhite/jcom_jun02_celiac.pdf

, 2664 . , ; . , bib , , .

Answer 2

:

md5sum *.java | sort | uniq -d -w32

uniq 32 , md5, .

EDIT: -w , :

md5sum *.java | awk '{print $1}' | sort | uniq -d

, , ... , ,

md5sum *.java | grep 0bee89b07a248e27c83fc3d5951213c1

( - ). , script .

Answer 3

:

md5sum *.java | sort | uniq -d

.

Answer 4

This list lists all the files, placing an empty line between duplicates:

$ md5sum *.txt \ 
  | sort       \
  | perl -pe '($y)=split; print "\n" unless $y eq $x; $x=$y'

05aa3dad11b2d97568bc506a7080d4a3  b.txt
2a517c8a78f1e1582b4ce25e6a8e4953  n.txt
e1254aebddc54f1cbc9ed2eacce91f28  a.txt
e1254aebddc54f1cbc9ed2eacce91f28  k.txt
e1254aebddc54f1cbc9ed2eacce91f28  p.txt
$

To print only the 1st of each group:

$ md5sum *.txt | sort | perl -ne '($y,$f)=split; print "$f\n" unless $y eq $x; $x=$y'
b.txt
n.txt
a.txt
$ 

if you are brave, change the if if to, and then

$ rm `md5sum ...`

delete everything except the first from each group