How to find the most common combinations of words in several texts?

Question

PHP

There is a DB with set of the textual information. It is necessary to find the most common combinations of words in order to train some scripts in the future.
How would it be logically correct to organize the search so as not to miss anything?

Answer 1

I think to do it as follows:
We take the text, divide it into shingles with a length of 1-2-3-4-5 overlapping, remove duplicates.
Next, we check each shingle for presence in the database. If this shingle does not exist, then we add it to the database and increase the number of mentions by 1. If there is a shingle, then we simply increase the number. And so is every text. As a result, we get the desired set of shingles and sort them.

Answer 2

Here is information on recursive searching and on a search tree structure for storing text.

Answer 3

www.nltk.org/howto/collocations.html

Answer 4

Everything has already been thought of before us. It also works with different word forms for the Russian language: carrot2