How to limit process memory allocated via malloc on Mac os?

Question

macOS

I need to limit the memory to the process so that malloc would return zero when trying to allocate.
unlimit - tried - does not work.
I found on the Internet that it seems possible using launchd , maybe someone knows how to do this using launchd or some other way?