Q
Q
q1we2016-07-23 13:52:04
Database
q1we, 2016-07-23 13:52:04

How to organize access to remote data (database, data processing)?

There is an application (written by me in C #) that downloads data from the Internet, processes it and saves it in a database (mssql). In view of the large amount of data, you need to split the work into several computers.
What is the best way to organize it?
I see two possible extensions. Write and install auxiliary clients on additional computers that will:
1) download data and transfer for processing to the application on the "main" computer (there are enough resources for processing). (The average total download speed for all connections that the main PC can make to the download site is approximately 300 kb. The maximum speed that it can accept is 1.2 mb. The site that gives information gives it slowly. The speed between the client computer and the main PC - 10 MB.)
2) download and process data and transfer the results to save in the database.
Please tell me which way to look to do this?
How to access a database that is stored on another computer?
How to organize data transfer between two PCs?
On computers, win7 is installed, each has a static ip.
I'm just learning C# myself. Please do not swear about the level of questions).

Answer the question

In order to leave comments, you need to log in

2 answer(s)
M
MrDywar Pichugin, 2016-07-23
@Dywar

1) Download and transfer. - it makes no sense, other PCs become just a proxy, another additional and unnecessary intermediate node, to drive traffic to an empty one.
2) Download and process - if processing takes a lot of time, then it makes sense, if not, then there is no point in it either. All the same, the main PC will not accept more traffic than it already downloads during operation.
If the result of the work is needed in 1 database, then you can do different parts of the same work on different PCs (from the very beginning to the end - download, process, save), and then put them in one basket.
Between themselves, they can only distribute work, the main PC will report that they say you are downloading from 1 to 99, you are from 100 to 199.
Finished? Download another portion here, or if the queue is over, then send the result of the work there.

Q
q1we, 2016-07-23
@q1we

There is a nuance regarding point 1) "Download and transfer" - the average total download speed for all connections that the main PC can create to the download site is approximately 300 kb. The maximum speed that it can accept is 1.2 MB. A site that gives information gives it away slowly. Therefore, proxies are needed to receive portions of data from them at a higher speed. The speed between possible proxy computers and the main PC is 10 MB.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question