How to organize duplication of Spark driver?

Question

Docker

There is Kafka, where events fall, they need to be filtered and entered into different databases.
Right now everything is built on Apche Spark (PySpark).
Each docker container ran its own local SparkContext, with its own writeStream.foreachBatch.
But this is very memory-consuming, so now I moved everything into one container, in which writeStream is connected to one context.

In any case, I would like to have redundancy so that two containers are launched on different machines with the same task. How can this be done?
I assume that a separate Spark cluster with two masters and N masters is required.
And how to duplicate tasks (spark driver)?

Answer 1

Can i know what are spark drivers?