A
A
Alexander2016-05-12 20:14:50
Python
Alexander, 2016-05-12 20:14:50

Which technology stack to choose for data processing and import?

It is necessary to load files of different formats (xml, json, csv) from the disk, validate them according to the specified rules.
It is desirable that the validation rules are set in json.
Then, from all this, substitute the necessary values ​​​​in the required fields, that is, from file 1, the name field, substitute the data in the dict by index from the main file, which is then imported into the database.
There can be a lot of data in files of 10k-100k lines.
What stack do you recommend?

Answer the question

In order to leave comments, you need to log in

2 answer(s)
S
spotifi, 2016-05-13
@spotifi

A typical console utility. It is done in any language that you know better. Clever words like "stack" are not appropriate here at all, unless, of course, the task is to process 1 pentabyte of data in 10 minutes.

A
Acheron, 2016-05-25
@Acheron

Pandas, Numpy and your own validator.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question