Where to start writing a search engine?

D

dima_antonenko2015-03-27 17:30:12

Algorithms

dima_antonenko, 2015-03-27 17:30:12

It has always been interesting how search engines work in terms of algorithms. For the sake of an example, I want to create a simple project that will parse data from several sites with the ability to text search. Tell me what literature you should start with and in what direction to move on.
I apologize in advance for the clumsy wording, I'm a newbie

Reply

Answer the question

In order to leave comments, you need to log in

7 answer(s)

S

SeptiM, 2015-03-27
@dima_antonenko

logic.pdmi.ras.ru/~yura/internet.html
https://ru.wikipedia.org/wiki/%D0%9F%D0%BE%D0%B8%D...
https://ru.wikipedia .org/wiki/%D0%9F%D0%BE%D0%B8%D...

S

Saboteur, 2015-03-27
@saboteur_kiev

Apparently you should start with the literature on algorithms. Graphs and other mathematics.

3

386DX, 2015-03-27
@386DX

There are open source search engines.
yacy.net/en/index.html
is wget

Y

YV, 2015-03-27
@targetjump

Understand exactly how you will index content and choose effective tools for this.

A

Alexander Taratin, 2015-03-27
@Taraflex

sphinxsearch.com
https://www.google.ru/webhp?sourceid=chrome-instan...

S

sivabur, 2015-03-28
@sivabur

1. Thoroughly study PageRank.
2. Descriptions of the student work of the creators of Google (Google prototype) wseob.ru/seo/searchengine-anatomy
3. You need to start with the algorithms that will be taken into account in the search results (that is, determine the factors and how exactly they will affect the construction of the search results)
4. Then the technical the implementation here already other algorithms come into force. How to find it faster? how to iterate? how to store the page? etc.

R

RokkerRuslan, 2015-03-29
@RokkerRuslan

Read Chapter 4 of Programming the Collective Mind
Chapter 4 Search and Ranking
Describes the various components of a search engine, including the spider, indexer, query engine. We reviewed an algorithm for ranking pages based on their leading links, PageRank, and showed how to create a neural network that learns which keywords are associated with different results.
Programming language - Python.