M
M
Maxim2020-10-20 09:53:25
MongoDB
Maxim, 2020-10-20 09:53:25

How to optimize frequent write requests in mongodb?

Hi everybody. The site has a collection of minimal analytics - page views. With each page request, js sends a request to the api, where an entry of the form is placed in the mongodb collection:

{
    model: "book" (string),
    key: 42 (integer),
    date: "2020-10-21 09:37:00" (string),
}


The collection has indices:

{
   "v" : 2,
   "key" : {
      "_id" : 1
   },
   "name" : "_id_"
},
{
   "v" : 2,
   "key" : {
      "key" : 1,
      "model" : 1,
      "date" : 1
   },
   "name" : "key_1_model_1_date_1"
}


Model and key are needed to identify the page. Every hour this data is processed by the worker, view statistics are calculated for each desired page, but that's not the point.

A write request to mongo is completed in 0.008 seconds, but in general, the work of mongodb loads the processor several times more than requests for the pages themselves from users.

How can mongo be optimized? Will there be a performance boost if you write a timestamp value to date instead of a string? Or another option, to identify the page (model, key) when creating it, assign a unique int value to it (the content itself rarely changes), then model and key are replaced by page_id: (int).

Will these changes make any difference? What other options exist?

Mysql is used to store the site content. The site is on the same server, using apache-itk, nginx for statics.

Answer the question

In order to leave comments, you need to log in

1 answer(s)
P
Philipp, 2020-10-20
@maks15m

date should be changed to ISODate() or timestamp() depending on how convenient it is. For proper aggregations, ISODate is better.
If possible, do a batch insert, then the index will not be rebuilt for each document, but only after the group is inserted. If you have good attendance, then add up the array data and insert into the database in groups of 100 records. If attendance is not very high, insert a timer once a second.
Doing a composite index on all fields does not make sense. Figure out what exactly you need, build an index on it. I like the decision to use a numerical pageId. Such an index will be more compact.
Regarding the architecture - keep the site and statistics on different services (servers). For MongoDB, use an SSD.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question