Anyone want to explain the "Tokenized Field" in terms of databases?

Question

I am reading about SOLR and indexing a MySQL database in SOLR.

What do they mean by tokenize and un-tokenize?

And what does it mean when the fields are "normalized"?

I know how and what it means to normalize a database, but a field? How can a normal field be normalized?

thank

Answer 1

What do they mean by “tokenization” and “un-tokenize”?

, .. , . Untokenized , . " ", " ", "".

, ""?

Unicode - , . U + 0060 - `( ), è (U + 00E8), (U + 0060 U + 0065). , , , , è.

Answer 2

, . , .. , (, , ..). , .

: " !" : 1) 2) 3)

"the" (), cat (), , .