What is the best architecture to choose?

Question

MongoDB

There is a main application on Loopback. The application has about 25-30 models about changes in which the clients learn through streams (EventSource). There is one significant problem, users subscribed to streams may have different conditions for receiving updates. That is, after updating the model, you need to get the user himself, his company and coordinates from the database (this is all an example). At light loads, everything works fine, but after about 1000 connected users, delays begin to appear, which is understandable, since conditions for 1000 users are checked at the same moment, and in almost every check there are queries to the database. To optimize this whole miracle, I plan to create a separate micro-service that will process information, this should reduce the load on the main application. I don't think there should be any problems with this. The second thing I plan to implement is to cache everything as much as possible in order to reduce the load on the database. But there are already problems with this. How to properly (optimally) make a cache, use Redis for this, or store everything in the memory of the application itself, given that the safety of this cache is not important to me, I don’t know if it’s appropriate to use something extraneous. If you still use Redis, then do you need to allocate a separate mvshina for it or raise it next to the application? And lastly, and most not clear to me, at the same moment several identical requests can shoot into the database. Let's say there are 10 of them, but you need to make sure that only 1 goes to the database, and the rest from the cache, how to do this? use Redis for this or store everything in the memory of the application itself, given that the safety of this cache is not important to me, I don’t know if it’s appropriate to use something extraneous. If you still use Redis, then do you need to allocate a separate mvshina for it or raise it next to the application? And lastly, and most not clear to me, at the same moment several identical requests can shoot into the database. Let's say there are 10 of them, but you need to make sure that only 1 goes to the database, and the rest from the cache, how to do this? use Redis for this or store everything in the memory of the application itself, given that the safety of this cache is not important to me, I don’t know if it’s appropriate to use something extraneous. If you still use Redis, then do you need to allocate a separate mvshina for it or raise it next to the application? And lastly, and most not clear to me, at the same moment several identical requests can shoot into the database. Let's say there are 10 of them, but you need to make sure that only 1 goes to the database, and the rest from the cache, how to do this?

Answer 1

do you read english? maybe you can find it here
https://engineering.gosquared.com/making-dashboard...

Answer 2

you can’t cache the work of the application logic -> because you will encounter the fact that the cache does not correspond to the real data model, and this leads to errors in the service (google cache invalidation), as a result, in order to keep the cache correct, you will need to carry out even more load and check the correctness of the cache with the data in the database.
so forget about logic caching, I cache simple ready-made answers to the user, nothing more.
1. For gigantic acceleration (thousands of times), you need to transfer the model and logic to RAM, and for this, forget about all kinds of hueda nodes, and switch to statically typed compiled languages, and asynchronously save this data model in memory to the database , then everything will fly.
2. For easy and most importantly correctdeveloping a service with a large number of "event" dependencies - it is advisable to use "Functional Reactive Programming" (you can look for materials on Google why and how this is implemented)

Answer 3

Separate the subscription into a separate entity: it should keep a list of subscribed users for this event, while making a synchronous cache in memory (well, or just a cache) - i.e. here the user subscribes to event x in the category y - we write this in the user profile and add the user to the list of subscriptions of the category "y.event". As soon as the event occurs, we do not need to go through all user profiles, but simply go through all subscribed users. Or, you can optimize the work with the database and shift to its shoulders the check "whether user x is signed for a subscription y". How exactly to do it optimally depends on many factors: the easiest option is in the application memory,