How to work with CREST api (EVE online)?

Question

PHP

There is such api . There is documentation, but if there is no experience, then in this case, it can do little to help.
According to the eve-market-data-relay.readthedocs.org/en/latest/us... documentation , I learned how to connect to the server using this code

/*
 * Example PHP EMDR client.
 */

$context = new ZMQContext();
$subscriber = $context->getSocket(ZMQ::SOCKET_SUB);

// Connect to the first publicly available relay.
$subscriber->connect("tcp://relay-us-central-1.eve-emdr.com:8050");
// Disable filtering.
$subscriber->setSockOpt(ZMQ::SOCKOPT_SUBSCRIBE, "");

while (true) {
    // Receive raw market JSON strings.
    $market_json = gzuncompress($subscriber->recv());
    // Un-serialize the JSON data to a named array.
    $market_data = json_decode($market_json);
    // Dump the market data to stdout. Or, you know, do more fun things here.
    var_dump($market_data);
}

At the moment, the main problem is that despite the speed of php7, I can not get all the information from the loop. Even if I remove the restriction on the duration of the script in php (the script worked for more than two hours, and so it did not finish working, then did not continue, this is not permissible for my purpose).
And here is the first request for help: how to work with such a large amount of data? The base must be up-to-date, since this is the base of the game server, its market, where information changes every minute.
Maybe you can get this base through other languages, for example, through c ++? But the thing is, I don't understand it. But, if someone explained in detail on this issue how to do this in conjunction with php, then maybe I would have figured it out faster.

Answer 1

OK. finally a valid question.
If you want to do something similar to a real-time price update, then attach some WS or Long-polling server thread to your script, for example socketo.me On the client, you can use autobahn.ws/js to listen and display data.
The game base dump can be found on the off. forum. Useful for converting IDs to titles, adding descriptions, etc. Although all this can also be done through CREST.
A lib for working with CREST can be downloaded from here https://github.com/EvELabs/oauth2-eveonline

Answer 2

As far as I know, EVE has a very... very, very huge amount of information.
And here is your first mistake, you want to copy it, I do not have the capacity to achieve the goal.
How this can be solved:
1. to delimit information into separate sections of the category and so on.
2. keep track of the desired values, for example, do not copy everything to yourself, but immediately pull up the necessary data at the user's request. With a sensible implementation of point 1, everything will work pretty quickly for you.
3. try to understand in small portions what kind of info comes in, do you need all of it, such a large-scale company should have an intelligent API and there most likely you need to make the request correctly, then the information can become many times less, processing is faster, etc.

Answer 3

And what was actually not understood in the eve-central API?
Of course, I agree with Dmitry Pechurkin that you don’t need the entire database at all from the word at all, especially since in JSON format you can pull real-time from evycentral.
The only thing you need is to keep a database of matching item names with their IDs,
if you are going to monitor for many systems, then the correspondence of system names with their ids.
Item base dumps, etc. once laid out on the off. wiki, now I don’t know, items can generally be taken from the dump of the eventral itself on their github https://raw.githubusercontent.com/theatrus/eve-cen...
And the central API is the simplest
JSON response with the status of the plex for the last 4 hours in life.
It is better to take data for 4 hours, at least 2 hours, you will not receive data with relevance less than 2 hours in most cases, although most likely you will not receive it at all.
So your idea of ​​getting data every minute is not realistic from any source.