What is text tokenization?

Question

Neural networks

Hello!
Can someone clearly and in simple language explain what text tokenization in machine learning is and how it is applied in practice

Answer 1

tokenization - splitting text into words (and non-words, those punctuation marks, paragraph boundaries, etc.). Its usefulness in machine learning is a direct message to the grid of the fact that a person (whose actions it needs to be taught to imitate) perceives the text as a stream of words, not a stream of letters.