Is there a more convenient way to build an aggregation pipeline for filtering?

Question

MongoDB

Our CRM system has the ability to sort students by various parameters, such as: age, gender, first name, last name, patronymic, teachers, groups.

And at the moment, all this has the following implementation: the client sends filtering parameters to the server in the format of such a DTO

export class FilterDTO {
    @IsOptional()
    @IsString({ each: true })
    names?: string[];

    @IsOptional()
    @IsString({ each: true })
    surnames?: string[];

    @IsOptional()
    @IsString({ each: true })
    midnames?: string[];

    @IsOptional()
    @IsArray()
    ages?: number[];

    @IsOptional()
    @IsString({ each: true })
    gender?: string[];

    @IsOptional()
    @IsString({ each: true })
    groups?: string[];

    @IsOptional()
    @IsString({ each: true })
    tutors?: string[];

    @IsOptional()
    @IsObject()
    balance?: { $gte?: number; $lte?: number; $lt?: number };

    @IsOptional()
    @IsBoolean()
    emptyAge?: boolean;

    @IsOptional()
    @IsString({ each: true })
    @IsMongoId({ each: true })
    statuses?: string[];
}

And when all these parameters arrive at the server, a separate huge function builds an aggregation pipeline

private createFilterPipeline(filters: FilterDTO): any[] {
        if (!filters) return;

        const pipeline = [];

        if (filters.names) {
            const nameFilter = {
                $match: {
                    $or: []
                }
            };

            filters.names.forEach(name => {
                nameFilter.$match.$or.push({
                    name: { $regex: new RegExp(`${name}`, 'i') }
                });
            });

            pipeline.push(nameFilter);
        }

        if (filters.surnames) {
            const surnameFilter = {
                $match: {
                    $or: []
                }
            };

            filters.surnames.forEach(surname => {
                surnameFilter.$match.$or.push({
                    surname: { $regex: new RegExp(`${surname}`, 'i') }
                });
            });

            pipeline.push(surnameFilter);
        }

        if (filters.midnames) {
            const midnameFilter = {
                $match: {
                    $or: []
                }
            };

            filters.midnames.forEach(midname => {
                midnameFilter.$match.$or.push({
                    midname: { $regex: new RegExp(`${midname}`, 'i') }
                });
            });

            pipeline.push(midnameFilter);
        }

        if (filters.balance) {
            if (filters.balance.$lte) {
                pipeline.push({
                    $match: {
                        $and: [
                            {
                                balance: {
                                    $gte: filters.balance.$gte
                                }
                            },
                            {
                                balance: {
                                    $lte: filters.balance.$lte
                                }
                            }
                        ]
                    }
                });
            } else {
                pipeline.push({
                    $match: {
                        $or: [
                            {
                                balance: {
                                    $gte: filters.balance.$gte
                                }
                            },
                            {
                                balance: {
                                    $lt: filters.balance.$lt
                                }
                            }
                        ]
                    }
                });
            }
        }

        if (filters.gender) {
            pipeline.push({
                $match: {
                    gender: {
                        $in: filters.gender
                    }
                }
            });
        }

        if (filters.groups) {
            pipeline.push({
                $match: {
                    groups: {
                        $all: filters.groups
                    }
                }
            });
        }

        if (filters.ages) {
            const agesFilter = {
                $match: {
                    $or: []
                }
            };

            if (filters.emptyAge === true) {
                agesFilter.$match.$or.push({ dateOfBirth: null });
            }

            filters?.ages?.forEach(age => {
                agesFilter.$match.$or.push({
                    dateOfBirth: {
                        $gte: new Date(
                            moment()
                                .subtract(age + 1, 'years')
                                .add(1, 'day')
                                .toISOString()
                        ),
                        $lt: new Date(
                            moment().subtract(age, 'years').toISOString()
                        )
                    }
                });
            });

            pipeline.push(agesFilter);
        } else if (!filters.ages && filters.emptyAge === true) {
            pipeline.push({ $match: { dateOfBirth: null } });
        }

        if (filters.tutors) {
            pipeline.push({
                $match: {
                    'tutors.tutor': { $in: filters.tutors }
                }
            });
        }

        if (filters.statuses) {
            pipeline.push({
                $match: {
                    statuses: {
                        $all: filters.statuses.map(status =>
                            Types.ObjectId(status)
                        )
                    }
                }
            });
        }

        return pipeline;
    }
}

This approach is extremely inconvenient, but at the time of writing this method, I did not come up with / did not find anything better.

The project is currently in a state of refactoring, and I really want to remove this feature and replace it with something more concise. What options for the correct implementation of such behavior exist? Maybe there are some design patterns? Here, if I understand correctly, the builder suggests itself, but how then to organize the reception of data for sorting from the client?

Answer 1

Well, we can generalize all this a bit - let's remove the darkness of if's
. we have a set of filters with arbitrary parameters (each filter has its own)
And at the output we go to get the final query to the database
. We can describe the filter processing rules as an object, where the key is the name of the filter, and the value is a callback that returns an element of this pipeline.
Something like

const filters = {
  groups: (groups) => ({
                $match: {
                    groups: {
                        $all: groups
                    }
                }
            })
...
}

Next, we loop through all the specified filters, successively applying callbacks from the object.