S
S
Stanislav2018-07-11 04:11:51
MongoDB
Stanislav, 2018-07-11 04:11:51

How to reduce the execution time of a complex text query?

What I haven’t tried already =(
There is a collection with the following structure

var schema = new Schema({
  public: 			{ type: Number,	default: 1 },
  createdAt: 		{ type: Date,		default: Date.now },
  updatedAt: 		{ type: Date,		default: Date.now },
  indexAt: 			{ type: Date,		default: Date.now },
  owner: 			{ type: mongoose.Schema.Types.ObjectId, ref: 'Users', required: !0 },
  allow:			{ type: Number, 	default: 0 },
  stats: {
    like: 			{ type: Number, 	default: 0 },
    view: 		{ type: Number, 	default: 0 }
  },
  pixels: 			{ type: Array },
  exif:				{ type: Object },
  likes: 			[{ type: mongoose.Schema.Types.ObjectId, ref: 'Users'}], 

  comments: [{
    owner: 		{ type: mongoose.Schema.Types.ObjectId, ref: 'Users' },
    post: 		{ type: String, trim: !0 },
    public: 		{ type: Number, default: 0 },
    createdAt: 	{ type: Date, default: Date.now }
  }],

  catalog: 			{ type: mongoose.Schema.Types.ObjectId, ref: 'Catalogs', required: !0 },
  size: {
    width: 		{ type: Number, 	required: !0 },
    height: 		{ type: Number, 	required: !0 }
  },
  tags: 			{ type: Array, 		required: !0 },

  tagsRu:			{ type: Array, 		required: !1 },
  tagsEn:			{ type: Array, 		required: !1 },
  
  subscribe:		{ type: String },
  image: 			{ type: String, 	required: !0 },
  domain: 			{ type: String, 	default: config.domain.photos },
  title: 			{ type: String,		trim: !0 },
  url: 				{ type: String, 	trim: !0, required: !0, unique: !0 }
})

An important subscribe field, all tags are placed in it and documents are located on it.
An index has been created
schema.index({ subscribe: 'text', title: 'text', url: 'text' })

It is vital to spit out a document with a ready-made structure, and therefore it is necessary to form it during aggregate.
Here is such a request
return Documents
        .aggregate([
            { $match: { public: { $gte: 2 }, $text: { $search: 'сам запрос' } }},
            { $sort: { score: { $meta: "textScore" } } },
            { $skip: request.skip },
            { $limit: request.limit },
            {
                $lookup: {
                    from: 'users',
                    localField: 'owner',
                    foreignField: '_id',
                    as: 'owner'
                }
            }, {
                $project: {
                    _id: 1,
                    image: { $concat: ['$domain', 'o/', '$image', '?route=thumb&h=350'] },
                    large: { $concat: ['$domain', 'o/', '$image', '?route=mid&h=750'] },
                    href: { $concat: [request.CONST.href, 'photo/', '$url', '.html'] },
                    hrefClass: config.hrefAddClass,
                    tags: { $slice: ['$tags', 12] },
                    size: {
                        width: { $trunc: { $multiply: [ { $divide: ['$size.width', '$size.height'] }, 350] } },
                        height: { $trunc: 350 }
                    },
                    style: {
                        $concat: [
                            'background-color: rgb(',
                            { $toLower: { $arrayElemAt: [ '$pixels.rgb.r', 0 ] } }, ',',
                            { $toLower: { $arrayElemAt: [ '$pixels.rgb.g', 0 ] } }, ',',
                            { $toLower: { $arrayElemAt: [ '$pixels.rgb.b', 0 ] } }, ')'
                        ]
                    },
                    owner: { $arrayElemAt: [ '$owner', 0 ] },
                    likes: 1,
                    indexAt: 1,
                    title: 1
                }
                
            }, {
                $group: {
                    _id: "$_id",
                    image: { $first: '$image' },
                    large: { $first: '$large' },
                    href: { $first: '$href' },
                    hrefClass: { $first: '$hrefClass' },
                    tags: { $first: '$tags' },
                    size: { $first: '$size' },
                    style: { $first: '$style' },
                    owner: { $first: {
                        _id: '$owner._id',
                        name: '$owner.name',
                        avatar: { $concat: ['background-image:url(', '$owner.avatar', ')'] },
                        href: { $concat: [request.CONST.href, '$owner.username', '/'] }
                    } },
                    likes: { $first: '$likes' },
                    indexAt: { $first: '$indexAt' },
                    title: { $first: '$title' }
                }
            },
            { $sort: { score: { $meta: "textScore" } } } // Дожимаю результаты, так как после $group бывает сверху шлак вылазит
        ])
        .then()

So, in a collection with documents where the "Request" is rare (for example, there are 50-100 such documents), everything works just fine, but where there are thousands of results in which the "Request" is present, the selection works just hellishly long, 5-6 seconds , but I'm sure there are longer requests. In addition, all this greatly burdens the UPU.
If you make the same selection, but sort by the indexAt field, everything works fine here and there, but the results themselves leave much to be desired, because what a person is looking for can be at the very bottom, and at the top get similar records at his request.
What to do? Is there something wrong with the structure? Or maybe select a separate collection for requests (although I'm not even sure that something will change). Can split into two requests? Any help will be helpful! I've been suffering for half a year now.

Answer the question

In order to leave comments, you need to log in

1 answer(s)
S
Stanislav, 2018-07-11
@ms-dred

When splitting the request into two, it turned out to reduce the time, but not as much as we would like.
I took a request running in 4.9ms and managed to cut it down to 1.1ms

return Wallpapers
        .aggregate([
            { $match: { public: { $gte: 2 }, $text: { $search: 'сам запрос' } } },
            { $project: { _id: 1, score: { $meta: "textScore" } } },
            { $sort: request.sort },
            { $skip: request.skip },
            { $limit: request.limit },
            { $group: { _id: null, uid: { $push: '$_id' }}}
        ]).then(e => {
            return e.length ? Wallpapers.aggregate([
                { $match: { _id: { $in: e[0].uid }} },
                {
                    $lookup: {
                        from: 'users',
                        localField: 'owner',
                        foreignField: '_id',
                        as: 'owner'
                    }
                },
                { $project: {
                    _id: 1,
                    image: { $concat: ['$domain', 'o/', '$image', '?route=thumb&h=350'] },
                    large: { $concat: ['$domain', 'o/', '$image', '?route=mid&h=750'] },
                    href: { $concat: [request.CONST.href, 'wallpaper/', '$url', '.html'] },
                    hrefClass: config.hrefAddClass,
                    tags: { $slice: ['$tags', 12] },
                    size: {
                        width: { $trunc: { $multiply: [ { $divide: ['$size.width', '$size.height'] }, 350] } },
                        height: { $trunc: 350 }
                    },
                    style: {
                        $concat: [
                            'background-color: rgb(',
                            { $toLower: { $arrayElemAt: [ '$pixels.rgb.r', 0 ] } }, ',',
                            { $toLower: { $arrayElemAt: [ '$pixels.rgb.g', 0 ] } }, ',',
                            { $toLower: { $arrayElemAt: [ '$pixels.rgb.b', 0 ] } }, ')'
                        ]
                    },
                    owner: { $arrayElemAt: [ '$owner', 0 ] },
                    likes: 1,
                    indexAt: 1,
                    title: 1,
                }}, {
                    $group: {
                        _id: "$_id",
                        image: { $first: '$image' },
                        large: { $first: '$large' },
                        href: { $first: '$href' },
                        hrefClass: { $first: '$hrefClass' },
                        tags: { $first: '$tags' },
                        size: { $first: '$size' },
                        style: { $first: '$style' },
                        owner: { $first: {
                            _id: '$owner._id',
                            name: '$owner.name',
                            avatar: { $concat: ['background-image:url(', '$owner.avatar', ')'] },
                            href: { $concat: [request.CONST.href, '$owner.username', '/'] }
                        } },
                        likes: { $first: '$likes' },
                        indexAt: { $first: '$indexAt' },
                        title: { $first: '$title' }
                    }
                }
            ]).then() : []
        })

If anyone has other options, please write =)

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question