How to optimize parsing of large csv files?

Question

PHP

There is a 300mb csv file. It has 200,000 lines of the form:

текст текст; "текст в двойных кавычках; текст в двойных кавычках; текст в двойных кавычках"; текст; текст
текст текст; "текст в двойных кавычках; текст в двойных кавычках; текст в двойных кавычках"; текст; текст; текст;  текст
текст текст; текст; текст;  "текст в двойных кавычках;текст в двойных кавычках; текст в двойных кавычках"; текст; текст

Delimiter ; , but since it also occurs inside the cells; I have to make a replacement using a regular expression, I found the solution here: 373729
Next, I use the following code to process the file:

$handle = fopen($file_save . $name_csv, "r");
      $data_import = array();
      if(empty($handle) === false) {
          while(($data = fgetcsv($handle, 0, ";")) !== FALSE) {
          array_push($data_import, $data);
          }
          fclose($handle);
      }

The problem is that the script takes 1GB of RAM for itself. The question is, maybe I'm missing some more correct way to parse a csv file of this size? The regular parsing and the parsing itself are separated separately, and the amount of memory consumed for parsing the file is indicated without the work of replacing the delimiter.

Answer 1

considering that then you push it into the database - push it immediately into the database and then do the logic in SQL. will be orders of magnitude faster and many times less memory-hungry

Answer 2

Read about generators Why do we need Generators (yield) in php? (I gave the link as an example, I just got the first one and there, just the same person showed csv processing)