Data Organization / Do I need a database?

Question

Node.js

I started learning JAVASCRIPT with .user.js . I wanted to slightly automate the interface through which I work. Then I got interested in Node, I always wanted something server-based - some magic, but recently my senior friend suggested that I try to do something. Although I understood that this was clearly not my level, I took it on, since this is a great way to get bumps, gain experience, etc. And yes, there is no such responsibility.

The program must collect data from different sources, using the same keys, and then analyze for the presence of a second group of keys and vice versa.

I wanted to parse, but I found some API, undescribed and received data from one site using the same key, and now I need to somehow systematize this matter, taking into account the fact that I will have to collect data using these keys periodically and add new data.

The data is ideally

{
id:``,
name:``,
artist:[],
preview:``,
text:``,
date:``,
source:``
}

Perhaps instead of / along with the full text of the article there will be a pdf file

. Now I just collected everything in one JSON file. To be honest, this practice suits me so far (as part of my training, I invented the wheel. I tried to write my own website with a self-written CMS) But such a volume of data and, what is probably more important, their analysis, confuse me and I still have no idea how to approach this, and then I ask how best to collect and organize, so that there would be fewer problems later.
I also don't know how best to organize the ID and maybe there will be new fields like different statuses and keys

. Also, I don't even really want SQl. For the time being, I feel comfortable imagining it in a JSON-like way and somehow I don’t want to learn the principles of SQL at the same time. (I suspect I'll just score in this case) and the speed of the entire system is not so important

Answer 1

If the database is static and not large (it is loaded at the start of the application), then of course json is better.
If it is dynamic, then the problems are:
1. Speed ​​of work
2. Cycling - you will have to organize the data storage order yourself (if you understand how to do it so as not to lose changes, then you will understand the cost of the item 1)
3. Size
But have you ever looked at examples of working with sql on a node? Nevertheless, wrapped and rewrapped utterly, in some frameworks, working with a sql database is described easier than with a json file. And then believe me, cycling will take away your desire more than working even with raw sql