How to create a simple neural network in TF that can control the game through screen capture?

Question

Python

Gifts! I know that I'm a teapot, a moron, etc. and I ask you to write something useful, not this. In general, I wanted to create a neuron in python and decided that it would play geometry dash and peeped from one person who made a neuron for gta 5 part of the code responsible for capturing the screen using OpenCV and Pillow ( https://www.youtube.com/ watch?v=ks4MPfMq8aQ&list=P... )
Here is the code, if anyone is interested:

So I ran into a problem: I'm too much of a teapot and I don't know how to push the screen capture into the neuron, how to tell it that you can press the buttons and how to tell her what is required of her (desired result). Therefore, I turn to you and hope that you can help in some way. Thanks in advance!

Answer 1

Judging by the questions, I'm afraid you have set yourself an impossible task so far.
To begin with, you must manually determine which parts of the screen you need to press when, this is the reaction of your artificial intelligence, it’s good when, depending on the situation, the coordinates of these areas do not change (roughly speaking, the unit’s control buttons).
Then you have to code very well (with separate algorithms, opencv itself is well suited for this) to recognize objects on the screen (it’s stupid to use a neuron for this, or rather it’s EXPENSIVE), as a result you will have information about the game world in your view, convenient for use in your program. And only then start thinking about how to use the information received, whether it is worth starting everything on neural networks.
In a blunt version, neural networks allow you to very quickly provide a reaction based on the search for similar actions in the data collected during the training process, if you do not have a super computer available and a lot of money to pay for its work, then it is unlikely that you will go further than this. Those. your artificial intelligence will play as if it randomly and mindlessly presses the right combinations of buttons.
geometry dash is a regular platformer, if I understand correctly, there is only one action at all - a jump? This game does not require neurons from the word at all, all actions are calculated by a simple analysis of the game world (enemy movements are calculated for 3-4 frames in a short version, or you yourself determine in advance that this enemy is moving according to such rules) and then search through the 'maze' by enumeration or depth-first search.