How to solve a similar problem using async/await/asyncio?

Question

Python

I am developing bots for VKontakte and usually use the multiprocessing library, but today I came up with the idea to rewrite part of the code using asyncio, for some reason I thought that I would significantly gain performance.
I read the manual on it, then another one, then twenty more and did not understand anything. How simple and easy multiprocessing is, asyncio is just as incomprehensible. Either the manuals consist more than completely of shit and sticks, or asyncio itself.
I’ll probably help a little by describing how everything works for me now:
1. A new process is created for each bot

mp.Process(name=bot.name + 'bot', target=self.bot, args=(bot.name, bot.status)).start()

2. Further in this process, a variable with a queue is created, into which another process adds messages to be sent

def bot(self, bot_name, bot_status, MAX_DELAY=4.0, MIN_DELAY=1.6, STEP=0.2, MAX_MSG_LEN=4000, MAX_ATTS=10):
        q = mp.Queue()
        mp.Process(name=bot_name + 'bot_client', target=self.client, args=(q, bot_name, bot_status)).start()
        while True:
            MSG = q.get()
            if type(MSG) != dict: continue
        ... # далее идёт неинтересная отправка сообщения в диалог

def client(self, q, bot_name, bot_status, wait=45, lp_version=4, mode=2): # получение event'ов, обработка и добавление их в очередь на отправку
        while True:
            try:
                lps = self.bots[bot_status][bot_name].getLPS(lp_version)
                while True:
                    im = requests.get('https://' + lps['server'], params={'key': lps['key'], 'ts': lps['ts'], 'act': 'a_check', 'mode': mode, 'wait': wait, 'version': lp_version}).json()
                    if im.get('updates', None) != None:
                        lps['ts'] = im['ts']
                        if len(im['updates']) > 0:
                            for update in im['updates']:
                                if update[0] != 4: continue
                                u = self.bots[bot_status][bot_name].check_update(update, lp_version)
                                if u != None: mp.Process(target=lambda u: q.put_nowait(self.bots[bot_status][bot_name].f(u)), args=(u,)).start()
                    else:
                        print(bot_name, 'failed', im.get('failed', None))
                        if im.get('failed', None) == 1: lps['ts'] = im['ts']
                        else: lps = self.bots[bot_status][bot_name].getLPS(lp_version)
            except Exception as e: print(e)

How to solve a similar problem using async/await/asyncio?

Answer 1

you need to rebuild your brain a bit,
but the gain in memory devouring is worth it

Answer 2

As a first approximation, on asyncio, your bots will be coroutines and the queues must be replaced with those from the asyncio package. But you can’t get by with asyncio alone, since it’s too low-level, you will have to perform some kind of http requests, it’s possible to listen to requests from vk api, etc. Implementing this yourself on asyncio is pointless, it is better to take ready-made implementations like aiohttp as you have already been advised. Perhaps this synthetic example on bare asyncio will help you figure it out, here several coroutines receive messages from their queues and simply print them to the screen:

the code

import asyncio
import random


class Bot:
    def __init__(self, bot_name, queue):
        self._name = bot_name
        self._queue = queue

    async def start(self):
        try:
            while True:
                data = await self._queue.get()
                print('{}: {}'.format(self._name, data))
        except asyncio.CancelledError:
            print("{}: stoped".format(self._name))


async def message_generator(bots):
    i = 0
    while True:
        for bot in bots:
            i += 1
            await bot["queue"].put("msg {}".format(i))
            await asyncio.sleep(((random.random()/2.0)+0.1))


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    bots = []
    n = 4
    for i in range(n):
        q = asyncio.Queue()
        t = loop.create_task(Bot("bot{}".format(i), q).start())
        bot = {
            "queue": q,
            "task": t
        }
        bots.append(bot)
    try:
        loop.run_until_complete(message_generator(bots))
    except KeyboardInterrupt:
        for bot in bots:
            bot["task"].cancel()
            loop.run_until_complete(bot["task"])
        loop.stop()

Your bots will technically work in the same way, but instead of outputting to the terminal, they will execute some kind of their own logic.