How to organize a server for polling and storing data from 10,000 sensors?

Question

linux

I’ll try to formulate a task, there is a certain system for controlling solar energy, I develop the brains of this system personally and from scratch, theoretically, there is absolutely any interface for transferring data to a computer in power ... While the prototype of the brains works autonomously, that is, without human participation (almost) , but that’s not the point .. I need to organize a server so that all the data from these brains is displayed in the web interface, I don’t want to use third-party hosting, because I’m mastering debian in parallel with development and I would like not to depend on anyone.
To begin with, the plans are to make such a system:
There are, for example, 1000 houses with such brains, all of them transmit data about the operation of the system to the host computer via the Internet. The data is the values ​​of various sensors such as temperature, voltage, current, illumination and much more ...
I think that I need a database in which the values ​​\u200b\u200bof each brain would be written and on demand, I could read everything about their state . Naturally through the web interface.
What requirements should iron have for such a task?
And how are such systems generally organized?
What to smoke to understand how it all works?

Answer 1

To collect such statistics, any iron is enough.
mysql+ dedicated ip on the server. Which, in principle, is also not necessary (you can also tie ddns)
For monitoring / display - any language in which you know how to write. The interface on bootstrap + php can be assembled in a few hours.
In general, the task is not difficult.

Answer 2

Write yourself a Web_API to work with the database. On the side of each house, implement a client that will use this API.
2) Write a web face to display data from the database

Answer 3

Or maybe not invent your own server, but use existing solutions, and direct your efforts to the iron part of the project?
openenergymonitor.org/emon
www.openhab.org
teamtreehouse.com/features
tech.yandex.ru/events/yac/2013/talks/1122
smartanthill.ikravets.com
pushingbox.com

Answer 4

Zabbix,
Nagios,
Munin,
Cacti,
OpenNMS,
Zenoss,
Ganglia,
Riemann,
Foglight NMS,
Pandora FMS,
NetXMS,
Xymon,
Opmantek NMIS,
Icinga,
Sensu,
MRTG,
Collectd,
Splunk

....
From personal experience, I advise you to take a closer look at the priority respectively to Zabbix, Nagios, Sensu, Cacti
Quite easy to learn, very diverse functionality, ease of collecting data and writing your modules.

Answer 5

Look at graphite.wikidot.com
There is a database and a simple API for adding, and a web interface and a bunch of different aggregate functions and scalable and used in a bunch of similar systems and applications, and it builds graphs, and for any values, and data can be entered en masse and for any time interval. Fairly easy to install and use. It can be used both in small installations to monitor dozens of values, and, for example, in Yandex, it is used to collect parameters from tens of thousands of objects.