They differ in how they divide the analyzed text into tokens.
StandardTokenizer does this based on the following (taken from lucene javadoc):
- Separates words in punctuation characters by removing punctuation marks. However, a dot not followed by spaces is considered part of the token.
- Separates words in hyphens if there is no number in the token, in which case the entire token is interpreted as the product number and is not divided.
- Recognizes email addresses and Internet host names as one token.
WhitespaceTokenizer does this based on space characters:
A WhitespaceTokenizer is a tokenizer that divides text into spaces. Adjacent character sequences without spaces form tokens.
You must choose the tokenizer that best suits your application. In any case, you need to use the same analyzers / tokenizers for indexing and searching!