Answer the question
In order to leave comments, you need to log in
What tools to use to process the history of relationships between entities?
Advise, please, approach or tool on which to look.
There are some set of entities conditionally described, for example, like this:
EntityA:
Aid, Cid, Aprop1, Aprop2, Aprop3, ...
EntityB:
Bid, Cid, Bprop1, Bprop2, Bprop3, ...
EntityC:
Cid, Aid, Bid
Yes both logically independent entities and entities that implement a connection between several others.
There is a set of text files (csv and xml) about these entities that describe the states of different entities "at the moment", that is, at the time of generating text files with a description. There are dozens of entity types, tens of millions of entities of each type, historical states (that is, files with a snapshot of certain entities) - a couple of thousand weighing 200GB.
I would like to slip all the files into one place in order to study the big picture for the quantity and quality of relationships between entities, how they change from cast to cast, and so on.
The easiest way is to stuff everything into SQL, but as long as there can be dozens of states about an entity with one id, it is not clear how long queries will spin without adequate indexes, which it is not clear how to build.
I have never come across tools for data analysis, I have no ideas. :)
Answer the question
In order to leave comments, you need to log in
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question