How to increase the speed of the "parser" in python?

Question

Python

Good afternoon.
There is a python script that parses a csv file with 5 columns and 25-30 thousand lines.
Therefore, the script iterates through the file, performs some actions and writes to the database.
The script parses such a file in 12 minutes.
Intel i7
CPython 3.3
How can I improve the speed?

Answer 1

The parser has nothing to do with it. Insert not one record at a time, but 500 or more.

Answer 2

Wang what ... without source codes it doesn’t van.
Most likely, it's not about parsing CSV, but about writing to the database.

Answer 3

def unpack_csv(line):
    csv = line.split(";")
    csv_1 = csv[0]
    csv_2 = csv[1]
    ...
    return csv_1, csv_2, csv_3, csv_4, csv_5

db = pymysql.connect(host='', user='', passwd='', db='', charset='utf8')
cursor = db.cursor()
file = open('test.csv', 'r')

for line in file:
    number, date, fio, email, about = unpack_csv(line)
    sql = """INSERT INTO mydb(number, date, fio, email, about)
             VALUES ('%s','%s', '%s', '%s', '%s')
             """, (number, date, fio, email, about)
    cursor.execute(sql)
    db.commit()
db.close()
file.close()

Answer 4

Well, insert one record at a time,
enter a variable like how many records to insert at a time keys=10000,
well, as long as the cycle counter is less, then put all the data into the buffer
as soon as it is equal to or more, then you do insert with the data,
reset the counter and again read the lines further
, and at the end you need do not forget to throw the latest data into the database

Answer 5

If you write not one record at a time, but 10000 each
Result:
CSV file weighing 1.3 mb (20 thousand lines)
real 8.26
user 5.25
sys 0.93
Very cool) Thank you all!