Why can't I get the page's source code?

Question

Parsing

Good afternoon.
I would like to know why this code is applicable to habr, yandex and many others, but does not want to work with asos, adidas

var url = "https://ya.ru/";
var HttpClient = new HttpClient();
var html = HttpClient.GetStringAsync(url);
Console.WriteLine(html.Result);

What should be applied or in which direction to dig?

Answer 1

Because on these sites the page is generated by javascript.
And in order to parse, you need to either analyze the site architecture and try to get exactly the data from the API (if any), or use browsers that can execute JS and get the page from them.

Answer 2

You can use cefsharp to inject into the page using c# and wait for all boot scripts to load and fire there. If anything, there is also cefsharp.offscreen (headless) if you want to run it as a service or in the console.