What are the efficient methods for handling large amounts of data?

Question

MySQL

Hello.
There is a large array of data in the database 500 thousand.
The data consists of the timestamp and duration of the event.
What effective methods exist for processing large data arrays and what can you recommend in this case? And what can be done with such data? In what visual form can it be presented?
Maybe there are statisticians?

Answer 1

1. 500 thousand what? Records? What is the recording volume? But generally speaking, data of such a volume cannot be considered "large" - these are quite moderate volumes for almost any modern tasks and, accordingly, modern tools for working with data.
2. Data of the specified type - refer to the so-called. time series. A classic of the genre, if timestamps set equal intervals, but otherwise - you can work with such data relatively easily - the main thing is to understand what you want to do with this information.
3. The form of visualization of time series - various kinds of graphics. X-axis - time, Y-axis - series values. However, there may be other representations, depending on the purpose of such a visualization.
4. Two classes of problems are usually solved with such data. Or, based on a set of values ​​of a series in the past, a prediction is made of the values ​​that the series will take at future points in time. Or the detection of a number of some anomalies in the past values. There can be two options - either identifying outliers, i.e. single points, whose behavior contradicts the series behavior model, or revealing the moment of change of the series behavior model itself.
5. Yes, "specialists in statistics" are here - probably to your surprise - there are.
6. “ I can’t formulate a task, because I don’t know the subject area of ​​statistics - and you formulate your task in your subject area, and statisticians will help you later.