What is the best way to scale an application?

Question

Software design

There are 100 data sections. There is an application which processes them results writes in SQLite. The problem is that:
- the application is single-threaded
- there are no 100 cores on the computer to run 100 copies, each of which would process its own section, and then I merged the databases.

Didn't use anything other than VPS. Are there any cloud services where you can get a lot of weak cores? You don't need a lot of memory.

I just see the only option somewhere to find a 100-core system and run 100 copies of the application there. Are there similar services?

Answer 1

I will probably be as simple as possible in the recommendations:

• AWS Lambda + AWS Step Functions
  
• GCP Cloud Functions + GCP Dataflow
  
• Azure Functions + Azure Logic Apps
  

in fact, I listed the horizontal computing power + orchestrator for each of them.
SQLite will have to be abandoned

Answer 2

You can create 100 threads or 100 coroutines and send data to the queue, and from there to the database.