I
I
im_dimas2019-07-02 11:10:03
Parsing
im_dimas, 2019-07-02 11:10:03

What characteristics of the server to choose for parsing?

Good afternoon, it is planned to rent a server for parsing information from the site.
I do not know what information is needed to determine the best characteristics of the desired server, but on a weak 4-core laptop with a processor frequency of 1.6 Hz, parsing is completed in about 20 minutes.
When parsing, about 900 pages of the site are loaded

Answer the question

In order to leave comments, you need to log in

1 answer(s)
V
Vladimir Korotenko, 2020-01-25
@firedragon

As already mentioned above, the problem is not in the processor.
Parsers are slow in nature, while the page loads you have time to parse it, so the processor is not loaded.
Memory can be estimated something like this: 20 - 100MB + page size * 12 ( DOM )
That is, if you do not parse a bunch of pages, but create a pool of workers, then the simplest hosting with 512 brains can handle it.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question