MongoDB crashes due to lack of memory, how to overcome?

Question

MongoDB

About a year ago, there were similar problems with MongoDB, when I switched to version 4.0, then after the next update, everything somehow resolved itself and monga stopped falling.
At the moment, MongoDB v4.2.8 is installed.
Now the problem is the following, it seems that due to the fact that I updated the npm packages MongoDB 3.6.0 and Mongoose 5.10.0, everything started again, maybe this is not the case, I can’t say for sure.
The logs contain the following information:

/var/log/mongodb/...

2020-08-16T09:46:46.491+0300 F  -        [conn101305] out of memory.
 0x557226fc67a1 0x557226fc5ce7 0x557226fbb151 0x557226f940f9 0x557226f94199 0x5572267bc6bf 0x557225ea92c9 0x557225ea94e2 0x557225e8d7e8 0x557225ea75f0 0x557225e8d7e8 0x557225e8d7e8 0x557225e75d39 0x557225e8d7e8$
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"557224708000","o":"28BE7A1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"557224708000","o":"28BDCE7","s":"_ZN5mongo29reportOutOfMemoryErrorAndExitEv"},{"b":"557224708000","o":"28B3151","s":"_$
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x557226fc67a1]
 mongod(_ZN5mongo29reportOutOfMemoryErrorAndExitEv+0x87) [0x557226fc5ce7]
 mongod(_ZN5mongo11mongoMallocEm+0x21) [0x557226fbb151]
 mongod(_ZNK5mongo7BSONObj4copyEv+0x29) [0x557226f940f9]
 mongod(_ZNK5mongo7BSONObj8getOwnedEv+0x59) [0x557226f94199]
 mongod(_ZN5mongo16WorkingSetMember20makeObjOwnedIfNeededEv+0x4F) [0x5572267bc6bf]
 mongod(_ZN5mongo11TextOrStage7addTermEmPm+0x489) [0x557225ea92c9]
 mongod(_ZN5mongo11TextOrStage16readFromChildrenEPm+0x72) [0x557225ea94e2]
 mongod(_ZN5mongo9PlanStage4workEPm+0x68) [0x557225e8d7e8]
 mongod(_ZN5mongo14TextMatchStage6doWorkEPm+0x70) [0x557225ea75f0]
 mongod(_ZN5mongo9PlanStage4workEPm+0x68) [0x557225e8d7e8]
 mongod(_ZN5mongo9PlanStage4workEPm+0x68) [0x557225e8d7e8]
 mongod(_ZN5mongo10FetchStage6doWorkEPm+0x59) [0x557225e75d39]
 mongod(_ZN5mongo9PlanStage4workEPm+0x68) [0x557225e8d7e8]
 mongod(_ZN5mongo15ProjectionStage6doWorkEPm+0x47) [0x557225e8db67]
 mongod(_ZN5mongo9PlanStage4workEPm+0x68) [0x557225e8d7e8]
 mongod(_ZN5mongo16PlanExecutorImpl12_getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0x230) [0x557225ed4d10]
 mongod(_ZN5mongo16PlanExecutorImpl7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x4D) [0x557225ed546d]
 mongod(_ZN5mongo20DocumentSourceCursor9loadBatchEv+0x2E6) [0x557225eb5ed6]
 mongod(_ZN5mongo20DocumentSourceCursor7getNextEv+0xB4) [0x557225eb62b4]
 mongod(_ZN5mongo42DocumentSourceSingleDocumentTransformation7getNextEv+0x38) [0x5572268f7228]
 mongod(_ZN5mongo18DocumentSourceSort8populateEv+0x2C) [0x5572268fb9fc]
 mongod(_ZN5mongo18DocumentSourceSort7getNextEv+0xDC) [0x5572268fbb9c]
 mongod(_ZN5mongo18DocumentSourceSkip7getNextEv+0x57) [0x5572268f77b7]
 mongod(_ZN5mongo19DocumentSourceGroup10initializeEv+0x5F) [0x5572268ae1df]
 mongod(_ZN5mongo19DocumentSourceGroup7getNextEv+0xBC) [0x5572268af37c]
 mongod(_ZN5mongo8Pipeline7getNextEv+0x3D) [0x557226908c4d]
 mongod(_ZN5mongo18PipelineProxyStage11getNextBsonEv+0x35) [0x557225e8d235]
 mongod(_ZN5mongo18PipelineProxyStage6doWorkEPm+0x46) [0x557225e8d0a6]
 mongod(_ZN5mongo9PlanStage4workEPm+0x68) [0x557225e8d7e8]
 mongod(_ZN5mongo16PlanExecutorImpl12_getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0x230) [0x557225ed4d10]
 mongod(_ZN5mongo16PlanExecutorImpl7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x4D) [0x557225ed546d]
 mongod(+0x14B97FC) [0x557225bc17fc]
 mongod(_ZN5mongo12runAggregateEPNS_16OperationContextERKNS_15NamespaceStringERKNS_18AggregationRequestERKNS_7BSONObjERKSt6vectorINS_9PrivilegeESaISC_EEPNS_3rpc21ReplyBuilderInterfaceE+0x275F) [0x557225bc669f]
 mongod(+0x14B177F) [0x557225bb977f]
 mongod(+0x118C568) [0x557225894568]
 mongod(+0x118E7D4) [0x5572258967d4]
 mongod(_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE+0x41A) [0x55722589756a]
 mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x3C) [0x5572258850ec]
 mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xEC) [0x557225890e7c]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x17F) [0x55722588e15f]
 mongod(+0x1187DEC) [0x55722588fdec]
mongod(_ZN5mongo12runAggregateEPNS_16OperationContextERKNS_15NamespaceStringERKNS_18AggregationRequestERKNS_7BSONObjERKSt6vectorINS_9PrivilegeESaISC_EEPNS_3rpc21ReplyBuilderInterfaceE+0x275F) [0x557225bc669f]
 mongod(+0x14B177F) [0x557225bb977f]
 mongod(+0x118C568) [0x557225894568]
 mongod(+0x118E7D4) [0x5572258967d4]
 mongod(_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE+0x41A) [0x55722589756a]
 mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x3C) [0x5572258850ec]
 mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xEC) [0x557225890e7c]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x17F) [0x55722588e15f]
 mongod(+0x1187DEC) [0x55722588fdec]
 mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x182) [0x5572266ebe62]
 mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x10D) [0x55722588b02d]
 mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0x753) [0x55722588c8a3]
 mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x316) [0x55722588d456]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0xDB) [0x55722588e0bb]
 mongod(+0x1187DEC) [0x55722588fdec]
 mongod(+0x1FE42CB) [0x5572266ec2cb]
 mongod(+0x2639B75) [0x557226d41b75]
 mongod(+0x2639BD4) [0x557226d41bd4]
 libpthread.so.0(+0x74A4) [0x7f7faa6df4a4]
 libc.so.6(clone+0x3F) [0x7f7faa421d0f]
-----  END BACKTRACE  -----

As I understand it, the process is killed due to lack of memory.
Judging by mongostat, some miracles are also happening, there are 16GB of RAM on the server, and exorbitant values ​​are used according to mongostat

Screenshot

Default config settings

storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# network interfaces
net:
  port: 27017
  bindIp: 127.0.0.1

It is clear that you can rollback npm versions and try with older ones, with those that were earlier, but still I would like to figure out what the problem is and maybe I'm not alone =)

Answer 1

Actually, the problem was solved by rolling back to previous versions of npm MongoDB and Mongoose packages. I didn’t figure out which of them worked crookedly, I think it was Mongoose ...

Answer 2

add storage.wiredTiger.engineConfig.cacheSizeGB mongs to the config with a reasonable value (the smaller the cache, the more CPU load). But you don't need much. Try 1 and increase if necessary.