S
S
Siegurd2019-01-08 13:39:47
Database
Siegurd, 2019-01-08 13:39:47

How to build a system for recording, storing and conveniently analyzing data from sensors?

Hello!
There is an object telemetry system that sends data to the server via TCP-IP via GPRS. Data from sensors (pressure, temperature, relays, etc.) are received several times per second. This is necessary to create a mathematical model of the object. To date, all data is recorded by the server in an Excel file. And every day a new Excel file is created with the name of the current date, in order to somehow distinguish the logs at least by day. The daily log in Excel weighs 300 megabytes (9 GB per month). The task of analyzing indicators for a month/week turns into hell, since you need to shovel 30 tables (each with 250 tons of rows), which cannot be done quickly, and 16 GB of RAM is barely enough. I would like to find some unified architecture, preferably Open source for Windows, into which all this data can be quickly written (several times per second, 100 values ​​of different sensors), and then it is convenient to view them, analyze and possibly make reports. The most convenient option would be to write to the database with Matlab, and the analysis is already in the software environment that is designed for this.
To summarize, we need an analogue of ThingSpeak as software for Windows, only without restrictions on the speed and number of records.

Answer the question

In order to leave comments, you need to log in

2 answer(s)
D
Dmitry Dart, 2019-01-08
@gobananas

Is it on Windows? And so you take the clickhouse database, write data to it with any demon, then select the necessary data with queries and that's it

R
Roman Mirilaczvili, 2019-01-08
@2ord

InfluxDB, Prometheus with their time series database and Grafana visual interface. Write a wrapper to integrate with them.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question