How to fasten multi-threaded processing without memory problems?

Question

.NET

Hello, there is a function that generates many many lines in a separate thread. (a lot, storing them in a List is not an option because it crashes)
How can I create separate threads so that they take the result of the work and process it?
Tried doing something like

private void Generator()
{
// Тут он её генерирует
string value = "Строка которую он сгенерировал в чуть выше";
                    Thread a = new Thread(new ParameterizedThreadStart(Get_Obrabotka));
                    a.Start(value);
}

Crashes System.OutOfMemoryException"."

Answer 1

The generation of the strings themselves should be formatted as IEnumerable , and their processing should be done using Task .

Answer 2

If there is not enough memory, then it doesn't matter how many threads you use. Multithreading is needed either for asynchrony or for parallelism. Adding memory streams will not save, on the contrary, it will increase its consumption and even sometimes does not add speed.
If you have a list that many threads work with, then consider using thread-safe collections . Listnot thread safe. If you have a large amount of data that does not fit into memory, then only load the data in parts, if you want to add parallel processing of this data in different threads, then Parallel.ForEach
may suit you .
And yes, there is no need to add 100 threads, as already said, such a number will not fit into the number of cores and this will not add performance, but even vice versa, because one core will divide time into a part of dedicated threads and will switch context between them. It is not a fact, of course, that even two threads will be divided into cores, the OS is responsible for this, but 100 threads, this seems like overkill. You can, of course, experiment with the number and look at the performance at the same time.

Answer 3

The supplier-consumer pattern

Answer 4

Without looking at the code, all the advice is generally pointless.
Most likely generation of lines goes faster than their processing.
And most likely there is an incorrect (not optimal in terms of memory and performance)
processing, perhaps in general everything can work much faster and without any threads.

Answer 5

Don't create many threads.
If you want to get confused, you can monitor the state of the process memory and adjust their number. If you don't bother, then the performance gain from parallel execution is limited by the number of processor cores.
What you described fits the producer-consumer pattern.
Implementation example