What do you need to know to solve the problem of visual analysis of the content of a web page?

Question

Machine learning

I want to write a system that automates actions on websites.
Something that is usually done with selenium.

But I want to do it at the level of visual analysis of the web page.
That is, the content of the screen is analyzed, the browser window is located, with a certain probability there are basic controls (back, update, line with url, etc.), the appearance of the web page itself is analyzed, controls are searched for on it (input fields, then, what can be a button (again, probabilistic)

In machine learning, I have only theoretical knowledge.In computer vision, I am zero.Is

there in the world something similar to what I want to do?
What image analysis technologies will help me in solving tasks?

Answer 1

Why take on a task that requires several complex technologies, none of which you own? Isn't it easier to start with the gradual development of these same technologies, and only then - if by that time you do not understand the weak motivation when creating the described application - move on to solving it?

Answer 2

imagesearch?