S
S
Saboth2022-03-11 00:43:26
Web development
Saboth, 2022-03-11 00:43:26

What is the best technology stack for working with large databases?

Greetings. I want to make a reservation right away that I am not a developer but a customer, so I apologize in advance for the crookedly worded question. For a better understanding, I will try to comprehensively (but briefly) describe the situation and start with the background.
My goal is to make a website that, say, having a product catalog of 1000 items, will collect and process user data in the context of their preferences, based on data taken from their personal account, as well as customer reviews (there will be a fundamental division into positive and negative reviews in advance fixed criteria). That is, the system constantly collects statistics and, upon request, for example, for article No. 367, can provide the following information:
1. The percentage of buyers by gender / age (approx. 20% are women aged 25-35 years, 15% are men 45+, etc.
) scale 1-10 points about some aspect of the product) Example: 40% of positive reviews for product #367 contain marks on the “X” factor with a score of 7 or higher. The same goes for negative reviews.
3. Comparison of changes compared to previous months/quarters, etc. - Note: for the winter of 2022. The number/percentage of positive reviews decreased by _____ and the average score for factor “X” decreased by 0.7 points.
4. Ability to compare the performance of two products from the same category to each other.
In addition to collecting, storing and processing data, the system must also constantly update this information on the site for the attention of visitors.
Ultimately, if there is a large audience (and the project is planned for 100k+ visitors per month), the amount of data is considerable.

From conversations with developers (and I have already reached the 7th company in a row), they offer me two options - a full self-writing of 150k+ (I don’t pull it) or a site on Laravel + MySQL and then it varies. The first thing that confuses me is the three-fold spread in price tags, someone is ready to build for 20, and some even cut my requests in half for 60 and say: “either this way or 5-15+ k for each module being finalized.” The second point is the issue of load, some developers warn in advance that if the audience reaches the conditional 100 thousand per month, Lara will slowly start to suffocate from the load, others say that if I don’t need a second Facebook, everything will be ok. As a result, I get the impression that absolutely all offices hang noodles on my ears in one way or another, and simply promote those decisions on which they have a “full hand”.

Taking into account recent events, the situation is absolutely unpredictable, and for the time being I decided to put the project on pause and take the time to delve into the subject at least at a minimum level, I’m sitting here reading books, writing the first lines of code)) trying to understand the basic principles.
And finally, the question itself: based on the above, what technology stack would you choose as a developer to solve the above problem? So that the final price of development does not bite so much, but also so that it can withstand the load.
Thank you all in advance for your kind replies.

Answer the question

In order to leave comments, you need to log in

2 answer(s)
T
ThunderCat, 2022-03-11
@Saboth

Based on the above, what technology stack would you choose as a developer to solve the above problem?

To begin with, a little context - 2 important notes:
1) The described task is solved on any mentioned stack, + a dozen more possible options (Java / Spring, python, node, dotnet, golang, etc.).
2) The performance of the system depends on its slowest part, and in 95% of cases the slowest part is the database, simply due to the specifics of the work: a lot of data with complex relationships, selections with sorting and filtering from a large array, and that's it, at that time how the backend language basically performs fairly simple manipulations with ready-made data sets (with rare exceptions that are not in your case, or they are not described in the question).
In your case:
- the base can be said to be quite small, up to 1 million records the conditional muscle turns very quickly, if the hardware matches.
- the task of collecting metrics is quite trivial, you don’t need much intelligence in saving user actions, most of the work is outputting to beautiful graphs and writing quick queries with tricky selections / aggregations, playing around with formulas a little, it is possible to do some kind of summary selections with denormalized results for reports and other details. The work is not difficult, but tedious, and there is a lot of it.
About the cost and conditions:
full samopis 150k+ (I don’t pull it) or the site on Laravel + MySQL and then it varies.
full self-writing - either marketing or idiocy, no normal studio will write in plain language, in principle, in this case either a framework is used and the customer is presented as "made by hand by virgin programmers under the light of a silver moon", or developers are idiots who think they are cooler all (no). In the first case, this is the same as "a site on Laravel + MySQL" (the norm and standard for the average level of studios), in the second - a swindle for money from scratch and an attempt to sell shit in the "exclusiveness" wrapper.
For the price - today, if I'm not mistaken, 150k wooden ones is + - 1000ue. The price is acceptable for normal work, if you find it cheaper "on Laravel" - I think it will be generally ok. Again, the price is such a thing, someone will do it well for a penny, someone will slap something for expensive. The region also influences, it is clear that in Moscow time the price will be seriously higher than in conventional Chelyabinsk ...
An important point is that a normal TK simplifies both the search for an artist and the evaluation of work at times. If I were you, I would plunge into the topic of creating high-quality technical specifications, rather than into the jungle of modern technologies.
UPD:
the project is planned for 100 k+ visitors per month...
they warn that if the audience reaches the conditional 100 thousand per month, Lara will start to slowly suffocate from the load,
ok, let it be 200k for clarity.
On average, if this is a marketplace, then there will not be many regular visitors, I think about 10%, so they can be neglected. Conditionally - 200k / 30 days = 6600 per day, with active 10-12 hours of visiting, this is ~ 600 per hour. Let there be 5 page requests per visitor, that's 600*5=3000 per hour = 50 per minute or less than 1 request per second.
Even if you multiply this by 30 (assuming such a peak load), 30rps is generally "nothing". I don’t know who thought it was a problem for you, any modern mid-level iron draws out without tension.

V
Vladimir Korotenko, 2022-03-11
@firedragon

take a closer look at the ready-
made https://www.webmasters.by/articles/review-po/1137-...
in this case, the edits will be minimal, and the options on the lara are also self-written, just a different level of abstraction.
And in general, the price tag of 150k does not seem big to me. What are the salaries of programmers?

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question