I have a data file for fortune that contains many repeating states. I would like to remove them.
Fortune is outlined %, so an example of a good luck file might look like this:
%
This is sample fortune 1
%
This is
sample fortune 2
%
This fortune
is repeated
%
This is sample fortune 3
%
This fortune
is repeated
%
This fortune
is unique
%
As you can see, fate can span several lines, making decisions useless here .
What can I do to find and remove duplicate states? I was thinking of just finding ways to awkignore lines starting with %, but some states have the same lines but not the same overall (for example, the last two in my example), so this is not enough.
awk , .