How to deal with multithreading and asynchrony in a web application?

Question

Multithreading

A few months ago I started writing my first web application. I want to use it to analyze some independently (by a separate daemon) collected statistics from clients and display it in a clear and simple form. Since I am not a pro, the choice of technology stack was simpler: Flask as a simple and flexible framework, sqlite as a ready-made subd and matplotlib for drawing pictures. Over time, I wanted to make this application multi-user in order to monetize it in the future. I screwed the authorization with the Flask-login module, it was not difficult, but then there were problems.
1) I still don't get it: is Flask single-threaded or multi-threaded? Synchronous or asynchronous? When on one page I want to insert 2 graphs that are drawn on the fly according to data from the database, sqlite swears that "Recursive use of cursors not allowed". That is, it turns out that they are trying to access it for data from table 2 of the stream. Okay, the problem with the database can be solved by switching to a normal DBMS like postgres, or temporarily using the sqlite3worker library (to serialize queries to sqlite), as is done now. But here is problem number
2 2) Matplotlib also does not draw 2 graphs at the same time, the application simply closes without declaring an error (1 graph seems to have time to draw). Apparently, it also does not work in multi-threaded mode, on their website it says: "Matplotlib is not thread-safe".
3) I want to implement a job queue - that is, run a process that draws graphs separately (for example, also as a daemon or as RestAPI) and throw tasks into the queue from my application, and he will get them in turn, do and give them back. But so far I can’t figure out a bunch of options, and whether I’m looking at it in general.

a) I can’t block the creation of threads in Flask, because I don’t have multithreading in my code. He doesn't ask me.
b) If matplotlib is not thread-safe, can a separate process be launched from the application to create each plot? Has anyone done this? So generally do in serious applications?
c) The option with a queue for Celery or RQ is interesting, but so far the names run wide. Celery, RQ, some gevent, ForkingMixin, ThreadingMixing. I just need the graphs to build one by one. What is easier to do? Maybe even take naked Redis?
d) And the most interesting thing: before it's too late, does it make sense to develop a project on Flask, or should I take a multi-threaded framework like FastAPI, aiohttp, etc.? What do you advise? On the flask site, in the becoming big section ( https://flask.palletsprojects.com/en/1.1.x/becomingbig/ ) there are different tips, but they are for the pros, like subclass or read the source. In general, what are the criteria for big? I plan to develop in the future, but start small.

Of particular interest are the answers to the last two points - what is the best way to implement a queue and is it worth continuing the project on Flask?

Answer 1

1a) https://stackoverflow.com/a/38876915

more about threading mode

As of Flask 1.0, the WSGI server included with Flask is run in threaded mode by default.
Prior to 1.0, or if you disable threading, the server is run in single-threaded mode, and can only handle one request at a time. Any parallel requests will have to wait until they can be handled, which can lead to issues if you tried to contact your own server from a request.
With threaded=True requests are each handled in a new thread. How many threads your server can handle concurrently depends entirely on your OS and what limits it sets on the number of threads per process. The implementation uses the SocketServer.ThreadingMixIn class, which sets no limits to the number of threads it can spin up.
Note that the Flask server is designed for development only. It is not a production-ready server. Don't rely on it to run your site on the wider web. Use a proper WSGI server (like gunicorn or uWSGI) instead.

1b) About the limitations of Sqlite: try as mentioned in https://stackoverflow.com/a/26630550
2-3) use queues for plotting. Please note that RQ only works on a POSIX compliant OS (WSL on Windows). This is how queue workers work - on separate processes. From Flask they put it in the queue, and in the worker they take it from it. And so for each schedule or other task - just put the task in the queue and continue on. In the task message, you need to specify the identifier and parameters for plotting the graph, as well as everything you need to further enter the data into the database.

before it's too late, does it make sense to develop a project on Flask, or should I take a multi-threaded framework like FastAPI, aiohttp, etc.? What do you advise?

Try with queues first. You don't have thousands of users, it seems.

Answer 2

Flask is synchronous and single threaded. The competition in it is realized by launching several web application processes. Naturally, with this approach, the web application should be stateless, ideally 12-factor .

Answer 3

In what mode the Flask application works is determined by the wsgi server.
Recursive use of cursors not allowed
Means not what you wrote.
Judging by the rest of the questions, you've wised up the tin in your application, by the type of launching threads in views and other things
. Without code, it will be difficult to answer.