How to remove "similar" but not identical content in a MySQL database

Question

Suppose I have this table:

ID | description
-------------------
5  | The bird flew over the tree.
2  | The birds, flew over the tree

These two lines have “similar” content. How to remove # 2?

• What algorithm should be used for "similar" text?
• How do I do this with Python?

Thanks!

Answer 1

What you could try was to strip out the necessary punctuation and execute each sentence with a stemmer (like Porter Stemmer ).

, . , , , (, 40 ).

, , - , .

Python Porter.

Answer 2

, "" ( , ... , ), .

Answer 3

LIKE.

DELETE FROM myTable WHERE description LIKE 'The bird%flew over the tree%';

Answer 4

, , . , , , .

http://en.wikipedia.org/wiki/Edit_distance

http://en.wikipedia.org/wiki/Levenshtein_distance

http://en.wikipedia.org/wiki/Damerau%E2%80%93Levenshtein_distance

, , , . 2 3 .

This may not be a good idea if your database is large.

In pseudo code, you mostly say

if editDist(stringA, stringB) > editThreshold
    remove stringA %or stringB
end