How to process and store data received via JSON?

Question

JavaScript

Hello toster!

I started to master node.js, but not for the sake of “entering IT”, but for myself. Now I'm doing a little "Hello, world!" a project based on the API of a very popular MMO game to explore the power of node.js and json. And I ran into a problem, the answer to which is the experience gained, and not reading the documentation and "Google coding" (which is what I actually do).

The essence of the project: there is an API, when I request it, I get the following response from the server in json:

{
    "files": [{
        "url": "http://auction-api-eu.worldofwarcraft.com/auction-data/ef92b9868d1316c1066e9fb5e6d979a0/auctions.json",
        "lastModified": 1485460486000
    }]
}

The content of the url link (like itself) is constantly changing (with an interval of 15-30 minutes and the answer can weigh from 10Mb to 35Mb). Having received a similar response, the server compares the lastModified parameter and if it is different, then ..

And here the question is, what would be wiser to do next?
I don't have a strict need to store all the data received in auctions.json, I'm only interested in certain parameters ("auc", "item" "owner", "owner Realm" "bid" "buyout" "quantity" "timeleft", but when provided that the value of the "item" parameter is equal to some certain number or series of numbers). The auctions.json structure itself looks something like this:

{
"realms": [
  {"name":"Outland","slug":"outland"}],
"auctions": [
{"auc":1270238130,"item":82800,"owner":"Tapicka","ownerRealm":"Outland","bid":79999999,"buyout":79999999,"quantity":1,"timeLeft":"VERY_LONG","rand":0,"seed":577282176,"context":0,"modifiers":[{"type":3,"value":1333},{"type":4,"value":50331667},{"type":5,"value":1}],"petSpeciesId":1333,"petBreedId":19,"petLevel":1,"petQualityId":3}

What is the best way to store data here? Use DBMS? (and what? I have experience with SQL solutions, and even neo4j, but I don't know anything about Mongo.db, and is it suitable here at all?) The best thing is to download auctions.json from the server, parse it "in place" and then write the data of interest to me into the database, or is it more optimal / better to parse it directly about the url and then write to the database everything that meets the conditions above? Or would it be easier not to bother at all, and write the entire auctions.json to the database and then display what I need from the database?

I'm posting the question here, not on Stackoverflow, in the hope that someone will take the time to review it and help find the optimal data processing sequence. From myself I promise from now on, not to produce such stupid questions.

Answer 1

It can be stored in Redis, before entering into the database, you need to pass through the handler, and based on its results, do certain actions.