A
A
Alexey2018-08-03 15:34:18
Database
Alexey, 2018-08-03 15:34:18

How to properly process data from CSV and upload it to SQL?

Good afternoon!
The crux of the matter, if global:
suppliers issue a list of goods in csv format by URL. About 10 csv files.
All providers may have different column names, date format, one has dd/mm/YYYY, another has YYYY-mm-dd, and so on... in different ways.
You need to combine them all in one database (or in one CSV) so that you can search (filter) by product name, date and compare prices between different suppliers on the front.
A kind of synchronization of data from all suppliers, sorting, cleaning, bringing to the front with filters. File sizes up to 250 MB.
Please tell me the solution on which it is optimal to implement such work with CSV data, what script, parser or some cloud solutions allow you to work with data from CSV in this way? Who is in the subject? Who knows or can help?

Answer the question

In order to leave comments, you need to log in

2 answer(s)
E
Eugene, 2018-08-03
@klim76

BULK INSERT?

A
AVKor, 2018-08-03
@AVKor

A script in any language that allows you to parse CSV files normally. For example ruby ​​+ smarter_csv gem.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question