How to organize such a database?

Question

Database design

DBF arrives in the mail every month.
It is picked up by php, parsed in sql. As a result, from a database with 20 columns and from 10 to 100 rows (as when), only 3-4 columns are needed for our application needs.
If we connect 100-1000-10000 clients with the same task, it turns out that every month we pull in 100-1000-10000 such databases and for each client we scatter only 3-4 columns in his table. And so every month.
Question: what is the best way to organize it? Should I keep all the records in full, or take what I need, delete the rest, and put dbfks in the archive on the server?

Answer 1

If there are no problems with the place, I would keep everything.
You never know. Might come in handy for demolition.
And so - these volumes for modern DBMS are nonsense, seeds.

Answer 2

Usually for such tasks I organize a separate table for data import. I stuff everything that comes from the outside in there. Then I process in a separate thread (several threads): I write only the necessary data to the main tables. At the same time, I put a flag or an error message on the processed records of the import table. Then it's easier to figure out what worked and what didn't and why. And you don't have to re-upload external files.
When the process is debugged, I set a periodic task to clean up (or archive, if reconciliation is needed) the old data in the import table. Can be run in prom.