How to increase the number of requests to the server in the minimum time?

Question

C++ / C#

Hello, I immediately apologize for my stupidity (I am just starting to learn c #), but it is very necessary to write this program. The problem is that I send a request to the server:

private string GET(string Url)
        {
            WebRequest req = WebRequest.Create(Url);
            WebResponse resp = req.GetResponse();
            Stream stream = resp.GetResponseStream();
            StreamReader sr = new StreamReader(stream);
            string Out = sr.ReadToEnd();
            sr.Close();
            return Out;
        }

But I don't like the speed of my program. In 10 seconds, I can only send a maximum of 20 requests to different pages of the site. I understand it happens like this:
1) The program sends a request.
2) We are waiting for some time until the server will answer us (here is the most difficult place, I think so)
3) Processes the response.
4) Go back one step.
I understand correctly, tell me please?
Is it possible to do this:
1) The program sends a request to 1 page.
2) The program sends a request to page 2.
3) We are waiting for some time until the server will answer us for 1 page.
4) We are waiting for some time until the server will answer us on page 2.
5) We process the answers.
There are suggestions that you need to use threads for this ... If it's not difficult with the code, please explain. Thank you very much in advance.

Answer 1

Try using HttpClient :

private async Task<string> Get(string url)
{
  using (var client = new HttpClient())
  {
    using (var r = await client.GetAsync(new Uri(url)))
    {
      return await r.Content.ReadAsStringAsync();
    }
  }
}

But you will have to call the Get method with await , and further up the chain, add async/await :
Alternatively, you can run queries asynchronously and pass the result of the query to a function. With your code, something like this:

private void Get(string url)
{
  Task.Run(async () =>
  {
    WebRequest req = WebRequest.Create(url);
    WebResponse resp = await req.GetResponseAsync();
    Stream stream = resp.GetResponseStream();
    StreamReader sr = new StreamReader(stream);

    string result = await sr.ReadToEndAsync();

    sr.Close();

    // передаем результат
    ResultCallback(url, result);
  });
}

private void ResultCallback(string url, string result)
{
  // выводим в консоль результат
  Console.WriteLine(url);
  Console.WriteLine(result);
}

Similar option with HttpClient

private void Get(string url)
{
  Task.Run(async () =>
  {
    using (var client = new HttpClient())
    {
      using (var r = await client.GetAsync(new Uri(url)))
      {
        string result = await r.Content.ReadAsStringAsync();

        ResultCallback(url, result);
      }
    }
  });
}

Wrapping the code specified in the question text with minimal changes

private void Get(string url)
{
  // выполняем запрос в отдельном потоке
  Task.Run(() =>
  {
    WebRequest req = WebRequest.Create(url);
    WebResponse resp = req.GetResponse();
    Stream stream = resp.GetResponseStream();
    StreamReader sr = new StreamReader(stream);

    string result = sr.ReadToEnd();

    sr.Close();

    // передаем результат в функцию обратного вызова
    ResultCallback(url, result);
  });
}

private void ResultCallback(string url, string result)
{
  Console.WriteLine(url);
  Console.WriteLine(result);
}

Thread wrapped option if using older version of .NET

private void Get(string url)
{
  // создаем поток
  var t = new Thread(() =>
  {
    WebRequest req = WebRequest.Create(url);
    WebResponse resp = req.GetResponse();
    Stream stream = resp.GetResponseStream();
    StreamReader sr = new StreamReader(stream);

    string result = sr.ReadToEnd();

    sr.Close();

    ResultCallback(url, result);
  });

  // запускаем
  t.Start();
}

private void ResultCallback(string url, string result)
{
  Console.WriteLine(url);
  Console.WriteLine(result);
}

Answer 2

I'll edit the first answer to your question.
await - synchronizes the request, and as I understand it, you want several requests in parallel.
maybe so

var task1= Get("https://toster.ru/q/470457");
var task2= Get("https://toster.ru/q/470458");
var task3= Get("https://toster.ru/q/470459");

var result1=task1.Result;
var result2=task2.Result;
var result3=task3.Result;

I think you can convert it to an array, but this is just an example