How to write large log files in mssql in python?

Question

Python

There are log files of 150 - 750 MB in size.
Separator by ; and line break.
You need to read them and write them to sql.
At the moment I am doing line-by-line reading and writing, everything is fine, even the status bar is screwed up, but it lasts about 6 hours.
I tried to take all the records and shove it with 1 query, while sql just blocks, and a large amount of memory is used.
I read about pandas but I'm not entirely sure that this is what you need.
Can someone with experience tell me how to do it right?

Answer 1

MySQL has a LOAD DATA FROM LOCAL INFILE command, postgres has a similar one. If there is a correct CSV format, then it will probably be possible to upload it quite efficiently. Although, of course, the logs in the database are not ice.