How to distribute the load and what to use in scripts?

Question

PHP

Hey!
For a long time I have been writing a script that I can’t run in production because of very strong lags in spades. The essence of the script is to parse many pages and compare them with each other. The first option I tried to do everything in one php script. The amount of data grew and it took a lot of time to compare them in one cycle (1 script took only one processor core), in addition, there was a need to access intermediate data. I decided to rework a little and began to use a database, or rather MariaDB. I smashed everything according to the tables - everything became functionally correct. Only now MySQL also loads the processor core and lags in peaks.
1. To implement the functionality of the service, I need to store intermediate data, so I can’t completely abandon the database. Maybe there are some alternatives? By the way, the RAM is empty and there are no problems with its expansion. And there are other cores to which you can transfer the load.
2. How to correctly implement the work of demons (PHP scripts in the background) so that they are restored when disconnected? Currently using start-stop-daemon and bash cron script for recovery. I would like to somehow get rid of this crutch.
3. Pages are currently captured using curl. It takes a few seconds to complete this part, which also doesn't suit me. I tried to transfer everything to a separate hosting, which would be closer, but there was no result. Maybe you can somehow facilitate curl or maybe some alternatives? Pages on sites are heavy and post requests are used.

Answer 1

pancake well how it is possible to use file system for storage of intermediate options?
only RAM, that's what it's there for.
need performance? stop building programs on scripts from ready-made solutions, start writing normal code, in compiled languages.

Answer 2

Cache all the data we receive through requests.
With pages, you are unlikely to do anything.