D
D
delkov2017-07-27 08:45:16
linux
delkov, 2017-07-27 08:45:16

How to write remotely to db? TCP/FTP?

Good day!
There is a com-port from which the flow is ~< 3000 characters/second.
This stream must be written to a database located remotely.
In this regard, there are two solutions:
1) read the stream through cat, and then transmit over TCP, something like: cat /dev... | nc ..
On the receiving side, organize a TCP server, in Python, for example. from where to write to the database.
Or
2) write the stream to a file, every second, which is then transferred via FTP, followed by processing on the server after receiving.
In the first case, you will have to provide for situations when the database is suddenly offline, write to files, which are then sent.
However, there is no delay, the flow has arrived - we write right away.
What to prefer, maybe there are some problems with FTP transfer, less stable../etc?
Thanks in advance!
UPD:
Thank you all for the discussion!
I chose this bundle: with COM_port'a I pour data into files (nc> filename), I check on the same machine for validity (python). If oki -> I send via scp, if it comes - I delete it (scp .. && rm ..);

Answer the question

In order to leave comments, you need to log in

4 answer(s)
P
PrAw, 2017-07-27
@delkov

I would first locally pour it into a file so that there would be no problems in case of a connection failure.
Then you can locally parse the stream to a structural state, and then it’s more interesting.
Option 1: direct connection via ssh with port forwarding (then the database does not stick out) and pour requests into the database
Option 2: raise a small web REST service on the recipient, where to drop packets through wget, which will be processed and poured into the database.
the choice of option strongly depends on the computer to which the com port comes. If there are enough resources, parse locally and upload to a remote server. In this case, more control. If there is a network failure, it will simply accumulate locally and wait for sending.
The option with a web service is a professional deformation, in this case we give all the rough work to third-party software, it remains only to process the incoming data on the shoulders of the server application. You can also implement TCP, no problem.

T
tupen, 2017-07-27
@tupen

Depends ONLY on the task:
And it will be a very big run. Literally from 10 minutes to 10 days of work
Your question is that you ask us to evaluate your features of the task, which we cannot evaluate without knowing the requirements for the task.

A
aynur_safin, 2017-07-27
@aynur_safin

2) write a stream to a file, every second, which is then transferred via FTP, followed by processing on the server, after receiving.

And why write every second, write per-minute files,
if 1 character = 1 byte, then a minute is about 180Kb, a good size for FTP transfer.
Once every 5 minutes, for example, we start synchronization of the local directory from the FTP server ~ 1Mb.
And then - files, for example older than a day - are deleted.
Having found new files, the service on the server slowly uploads them to the database - it has 5 minutes.

Y
Yaroslav, 2017-07-27
@yaror

I would:
1. Parse directly on the machine that receives the stream from the COM port
2. Merge into the database directly from the same machine, via SQL: "insert into values ​​such and such". In the case of an exception that got out (no connection to the database, the database fell, etc.), the parsed values ​​could be logged locally

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question