Choice of tools for data calculation?

Question

SQL

Hello. Theoretical question:
What tools are better to use?
1. At the moment, initially the data is received in text form (csv) (about 60 GB per month), it is also possible to import them directly into the mysql database.
2. In a separate excel file, data is filtered by PQ requests every 3 hours to fit into 1 million rows. respectively, in a daily file of 8 sheets of data.
3. Further, in the next excel file, every 3 hours, by means of PQ queries, the data rows are compared for price changes, as well as the change value is calculated.
4. In the next file, the data for the day is combined and rechecked.
As a result, there is a folder structure with excel and csv (initial data) structured by dates.
Ultimately, a product is selected from a separate structure and all data on price changes over the past 15 days are searched for by ID.
At the moment, everything is working, slowly and confidently (although there is a premonition that through one place). But there are ideas about expanding the number of data users (at the moment, use only locally). And there is an idea about the transition from the structure by dates to the structure by ID in view of a very long search.
1. Does it make sense to switch to the database format? (For example, Mysql, knowledge at the level of installing CMS on hosting and small changes in the structure of the database)
2. What options are there for processing (calculations, grouping) data. What software can be used for this? Should I learn any programming languages? Is it possible to perform calculations similar to PQ queries using sql queries?
3. Isn't it a perversion to store data in a database, import it into excel, process it using pq queries and export it back?

Thanks in advance.

Answer 1

>1. Whether there is a sense to pass to a DB format?
If you need multi-user work, then yes, I think this is the only right way.
> What software can be used for this?
For starters, you can use the same Excel, it can connect to a database and take data from there.
> Should I learn any programming languages?
There is no big difference between popular languages ​​for this task. If you do not know any, it will be better to hire professionals. :)
> Is it possible to make calculations similar to PQ queries using sql queries?
Yes, at least for the most part. I do not know all the features of Power Query, perhaps something will have to be implemented in the code.
>3. Isn't it a perversion to store data in a database, import it into excel, process it using pq queries and export it back?
No, not a perversion. If Power Query solves your problems, why not.
The truth about "export back" is not sure. In multi-user work, someone must ensure that users do not interfere with each other. Do not overwrite the result of someone else's work, for example.

Answer 2

1. It seems to me that with such a volume of data - yes.
But you need to look more specifically at the nature of the data. Maybe some specialized solution, like clickhouse, is better than mysql
2. You have to look at exactly what things are happening in Excel. If you really like Excel, then you can look at power bi.
3. It seems to me that doing db->excel->bd is a perversion.
It seems to me that by moving away from Excel, you can save a lot of time. You can email me and I can help you with more specific questions.