Preprocessing and clearing Java text

Question

Could you advise me on java libraries for text preprocessing and cleaning? Lib should perform the following tasks:

• convert all verbs to infinitive
• convert all nouns to a special form
• remove useless (for text) words

Answer 1

The conversion of words to canonical forms (for example, verbs to infinitives and nouns to singular ones) is called lemmatization . One Java-lemmatizer Standford CoreNLP .

For “useless words,” you probably want to “stop the words” - there is no standard list, but there are many floating around the Internet that function more or less the same, the only difference is how many words they include (usually from 100 to 1000). I already knew that before people used this list . When deleting stop words, remember to ignore the case when searching for matches.

Answer 2

Not sure if this is all you need, but check mrsqg.

http://code.google.com/p/mrsqg/